Movatterモバイル変換

[0]ホーム
URL:

Skip to main content

Advertisement

Springer Nature Link
Log in

Image Captioning for Automated Grading and Understanding of Ulcerative Colitis

  • Conference paper
  • First Online:

Abstract

Ulcerative colitis (UC) is a chronic inflammatory disease of the large bowel characterised by quisent periodes and relapses. Endoscopic grading of the severity of UC is done by using a widely accepted scoring system known as the “Mayo Endoscopic Scoring” (MES). The MES score is largely based on the recognition of phenotypic features of the mucosal wall, and thus the subjectivity in clinical scoring is unavoidable. An automated grading and characterisation can certainly help to minimise the inter-observer variability and help trainees to get useful insights. For the first time, we a system capable of not only providing an automated MES scoring system, but also of generating a description of visible MES phenotypic mucosal representations in these endoscopic images through captions. Our aim is to combine the visual features together with word sequence embeddings that are learnt jointly through a recurrent neural network to predict such scene descriptions. In this work, we explore various recurrent neural network architectures together with other backbone architectures for visual feature representations. Our experiments on held-out test samples demonstrate high similarity between the reference and the predicted captions.

This is a preview of subscription content,log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Similar content being viewed by others

References

  1. Ali, S.: Where do we stand in AI for endoscopic image analysis? deciphering gaps and future directions. npj Dig. Med.5(1), 184 (2022)

    Google Scholar 

  2. Bhambhvani, H.P., Zamora, A.: Deep learning enabled classification of mayo endoscopic subscore in patients with ulcerative colitis. Eur. J. Gastroenterol. Hepatol.33(5), 645–649 (2021)

    Article  Google Scholar 

  3. Borgli, H., et al.: Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data7(1), 283 (2020)

    Article  Google Scholar 

  4. Click, B., Ramos Rivers, C., Koutroubakis, I.E., Babichenko, D., Anderson, A.M., et al.: Demographic and clinical predictors of high healthcare use in patients with inflammatory bowel disease. Inflamm. Bowel Dis.22(6), 1442–1449 (2016)

    Article  Google Scholar 

  5. Di Ruscio, M., et al.: Role of ulcerative colitis endoscopic index of severity (uceis) versus mayo endoscopic subscore (mes) in predicting patients response to biological therapy and the need for colectomy. Digestion102(4), 534–545 (2021)

    Article  Google Scholar 

  6. Fan, Y., et al.: A novel deep learning-based computer-aided diagnosis system for predicting inflammatory activity in ulcerative colitis. Gastrointest. Endosc. (2022)

    Google Scholar 

  7. Gong, L., et al.: Automatic captioning of early gastric cancer using magnification endoscopy with narrow-band imaging. Gastrointest. Endosc.96(6), 929–942 (2022)

    Article  Google Scholar 

  8. Ozawa, T., et al.: Novel computer-assisted diagnosis system for endoscopic disease activity in patients with ulcerative colitis. Gastrointest. Endosc.89(2), 416–421 (2019)

    Article  Google Scholar 

  9. Pagnini, C., Di Paolo, M.C., Mariani, B.M., Urgesi, R., Pallotta, L., et al.: Mayo endoscopic score and ulcerative colitis endoscopic index are equally effective for endoscopic activity evaluation in ulcerative colitis patients in a real life setting. Gastroenterol. Insights12(2), 217–224 (2021)

    Article  Google Scholar 

  10. Rabbenou, W., Ullman, T.A.: Risk of colon cancer and recommended surveillance strategies in patients with ulcerative colitis. Gastroenterol. Clin.49(4), 791–807 (2020)

    Article  Google Scholar 

  11. Schroeder, K.W., Tremaine, W.J., Ilstrup, D.M.: Coated oral 5-aminosalicylic acid therapy for mildly to moderately active ulcerative colitis. New Engl. J. Med.317(26), 1625–1629 (1987)

    Article  Google Scholar 

  12. Stidham, R.W., Liu, W., Bishu, S., Rice, M.D., Higgins, P.D., et al.: Performance of a deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis. JAMA Netw. Open2(5), 1–10 (2019)

    Article  Google Scholar 

  13. Sutton, R.T., Zaiane, O.R., Goebel, R., Baumgart, D.C.: Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images. Sci. Rep.12(1), 1–10 (2022)

    Article  Google Scholar 

  14. Taku, K., Britta, S., Chen, W.S., Ferrante, M., Shen, B., et al.: Ulcerative colitis (primer). Nat. Rev. Dis. Primers6(1) (2020)

    Google Scholar 

  15. Travis, S.P., Schnell, D., Krzeski, P., Abreu, M.T., Altman, D.G., et al.: Developing an instrument to assess the endoscopic severity of ulcerative colitis: the ulcerative colitis endoscopic index of severity (uceis). Gut61(4), 535–542 (2012)

    Article  Google Scholar 

  16. Ungaro, R., Mehandru, S., Allen, P., Peyrin-Biroulet, L., Colombel, J.: Colitis ulcerosa. Lancet389(10080), 1756–1770 (2017)

    Article  Google Scholar 

  17. Vashist, N.M., et al.: Endoscopic scoring indices for evaluation of disease activity in ulcerative colitis. Cochrane Datab. Syst. Rev. (1) (2018)

    Google Scholar 

  18. Xu, Z., Ali, S., East, J., Rittscher, J.: Additive angular margin loss and model scaling network for optimised colitis scoring. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE (2022)

    Google Scholar 

  19. Xu, Z., Ali, S., Gupta, S., Leedham, S., East, J.E., Rittscher, J.: Patch-level instance-group discrimination with pretext-invariant learning for colitis scoring. In: Machine Learning in Medical Imaging, pp. 101–110 (2022)

    Google Scholar 

Download references

Acknowledgements

The authors wish to acknowledge the Mexican Council for Science and Technology (CONACYT) for the support in terms of postgraduate scholarships in this project, and the Data Science Hub at Tecnologico de Monterrey for their support on this project. This work has been supported by Azure Sponsorship credits granted by Microsoft’s AI for Good Research Lab through the AI for Health program.

Author information

Authors and Affiliations

  1. School of Engineering and Sciences, Tecnologico de Monterrey, Monterrey, Mexico

    Flor Helena Valencia, Daniel Flores-Araiza & Gilberto Ochoa-Ruiz

  2. CINVESTAV Unidad Guadalajara, Zapopan, Mexico

    Obed Cerda

  3. Leeds Teaching Hospitals NHS Trust, Leeds, UK

    Venkataraman Subramanian

  4. Sahlgrenska University Hospital, Gothenburg, Sweden

    Thomas de Lange

  5. School of Computing, University of Leeds, Leeds, UK

    Sharib Ali

Authors
  1. Flor Helena Valencia

    You can also search for this author inPubMed Google Scholar

  2. Daniel Flores-Araiza

    You can also search for this author inPubMed Google Scholar

  3. Obed Cerda

    You can also search for this author inPubMed Google Scholar

  4. Venkataraman Subramanian

    You can also search for this author inPubMed Google Scholar

  5. Thomas de Lange

    You can also search for this author inPubMed Google Scholar

  6. Gilberto Ochoa-Ruiz

    You can also search for this author inPubMed Google Scholar

  7. Sharib Ali

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toSharib Ali.

Editor information

Editors and Affiliations

  1. University of Leeds, Leeds, UK

    Sharib Ali

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Fons van der Sommen

  3. Eindhoven University of Technology, Eindhoven, The Netherlands

    Maureen van Eijnatten

  4. University of Oxford, Oxford, UK

    Bartłomiej W. Papież

  5. National University of Singapore, Singapore, Singapore

    Yueming Jin

  6. Eindhoven University of Technology, Eindhoven, The Netherlands

    Iris Kolenbrander

Rights and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Valencia, F.H.et al. (2023). Image Captioning for Automated Grading and Understanding of Ulcerative Colitis. In: Ali, S., van der Sommen, F., van Eijnatten, M., Papież, B.W., Jin, Y., Kolenbrander, I. (eds) Cancer Prevention Through Early Detection. CaPTion 2023. Lecture Notes in Computer Science, vol 14295. Springer, Cham. https://doi.org/10.1007/978-3-031-45350-2_4

Download citation

Publish with us

Societies and partnerships

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

[8]ページ先頭
©2009-2025 Movatter.jp