Movatterモバイル変換

[0]ホーム
URL:

Skip to main content

Advertisement

Springer Nature Link
Log in

Towards I/O analysis of HPC systems and a generic architecture to collect access patterns

  • Special Issue Paper
  • Published:
Computer Science - Research and Development

Abstract

In high-performance computing applications, a high-level I/O call will trigger activities on a multitude of hardware components. These are massively parallel systems supported by huge storage systems and internal software layers. Their complex interplay currently makes it impossible to identify the causes for and the locations of I/O bottlenecks. Existing tools indicate when a bottleneck occurs but provide little guidance in identifying the cause or improving the situation.

We have thus initiatedScalable I/O for Extreme Performance to find solutions for this problem. To achieve this goal inSIOX, we will build a system to record access information on all layers and components, to recognize access patterns, and to characterize the I/O system. The system will ultimately be able to recognize the causes of the I/O bottlenecks and propose optimizations for the I/O middleware that can improve I/O performance, such as throughput rate and latency. Furthermore, the SIOX system will be able to support decision making while planning new I/O systems.

In this paper, we introduce the SIOX system and describe its current status: We first outline our approach for collecting the required access information. We then provide the architectural concept, the methods for reconstructing the I/O path and an excerpt of the interface for data collection. This paper focuses especially on the architecture, which collects and combines the relevant access information along the I/O path, and which is responsible for the efficient transfer of this information. An abstract modelling approach allows us to better understand the complexity of the analysis of the I/O activities on parallel computing systems, and an abstract interface allows us to adapt the SIOX system to various HPC file systems.

This is a preview of subscription content,log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

References

  1. Babu S, Borisov N, Uttamchandani S, Routray R, Singh A (2009) DIADS: addressing the “My-problem-or-yours” syndrome with integrated SAN and database diagnosis. In: FAST’09: proceedings of the 7th conference on file and storage technologies. USENIX Association, Berkeley, pp 57–70

    Google Scholar 

  2. Barham P, Donnelly ARI, Mortier R (2004) Using magpie for request extraction and workload modelling. Microsoft Research

  3. Chaarawi M, Gabriel E, Keller R, Graham RL, Dongarra JJ (2011) OMPIO: a modular software architecture for MPI I/O. Springer, Berlin/Heidelberg

    Google Scholar 

  4. Geimer M, Wolf F, Wylie BJN, Becker D, Böhme D, Frings W, Hermanns MA, Mohr B, Szebenyi Z (2009) Recent developments in the scalasca toolset. In: Tools for high performance computing, proceedings of the 3rd international workshop on parallel tools. Springer, Berlin

    Google Scholar 

  5. Hermanns MA, Geimer M, Wolf F, Wylie BJN (2009) Verifying causality between distant performance phenomena in large-scale MPI applications. In: Proceedings of the 17th Euromicro international conference on parallel, distributed, and network-based processing (PDP), Weimar, Germany. IEEE Computer Society Press, Los Alamitos, pp 78–84

    Google Scholar 

  6. Knüpfer A, Nagel WE (2006) Compressible memory data structures for event-based trace analysis. Future Gener Comput Syst 22:359–368

    Article  Google Scholar 

  7. Knüpfer A, Brunst H, Doleschal J, Jurenz M, Lieber M, Mickler H, Müller MS, Nagel WE (2008) The Vampir performance analysis tool-set. In: Tools for high performance computing, proceedings of the 2nd international workshop on parallel tools. Springer, Berlin, pp 139–155

    Chapter  Google Scholar 

  8. Kunkel J (2011) HDTrace—a tracing and simulation environment of application and system interaction. Tech. Rep. 2, Deutsches Klimarechenzentrum GmbH, Bundesstraße 45a, 20146, Hamburg

  9. Kunkel J, Ludwig T (2011) IOPm—modeling the I/O path with a functional representation of parallel file system and hardware architecture, to be published

  10. Lofstead J, Zheng F, Klasky S, Schwan K (2009) Adaptable, metadata rich IO methods for portable high performance IO. IEEE Computer Society, Washington

    Google Scholar 

  11. Minartz T, Molka D, Kunkel J, Knobloch M, Kuhn M, Ludwig T (2012) Handbook of energy-aware and green computing. Chapman and Hall/CRC Press Taylor and Francis Group LLC, Boca Raton, p 600

    Google Scholar 

  12. Noeth M, Ratn P, Mueller F, Schulz M, de Supinski BR (2009) ScalaTrace: scalable compression and replay of communication traces for high performance computing. J Parallel Distrib Comput 69:696–710

    Article  Google Scholar 

  13. Shende SS, Malony AD (2006) The TAU parallel performance system. Int J High Perform Comput Appl 20(2):287–311

    Article  Google Scholar 

  14. Thakur R, Gropp W, Lusk E (1999) On implementing MPI-IO portably and with high performance. ACM Press, New York

    Google Scholar 

  15. Thereska E, Salmon B, Salmon O, Strunk J, Wachs M, Abd-el-Malek M, Lopez J, Ganger GR (2006) Stardust: tracking activity in a distributed storage system. In: ACM SIGMETRICS conference on measurement and modeling of computer systems. ACM Press, New York, pp 3–14

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Bundesstraße 45a, 20146, Hamburg, Germany

    Marc C. Wiedemann

  2. Universität Hamburg—Deutsches Klimarechenzentrum GmbH, Hamburg, Germany

    Marc C. Wiedemann, Julian M. Kunkel, Michaela Zimmer & Thomas Ludwig

  3. High Performance Computing Center Stuttgart (HLRS), Universität Stuttgart, Stuttgart, Germany

    Michael Resch, Thomas Bönisch, Xuan Wang & Andriy Chut

  4. Zentrum für Informationsdienste und Hochleistungsrechnen, Technische Universität Dresden, Dresden, Germany

    Alvaro Aguilera, Wolfgang E. Nagel, Michael Kluge & Holger Mickler

Authors
  1. Marc C. Wiedemann

    You can also search for this author inPubMed Google Scholar

  2. Julian M. Kunkel

    You can also search for this author inPubMed Google Scholar

  3. Michaela Zimmer

    You can also search for this author inPubMed Google Scholar

  4. Thomas Ludwig

    You can also search for this author inPubMed Google Scholar

  5. Michael Resch

    You can also search for this author inPubMed Google Scholar

  6. Thomas Bönisch

    You can also search for this author inPubMed Google Scholar

  7. Xuan Wang

    You can also search for this author inPubMed Google Scholar

  8. Andriy Chut

    You can also search for this author inPubMed Google Scholar

  9. Alvaro Aguilera

    You can also search for this author inPubMed Google Scholar

  10. Wolfgang E. Nagel

    You can also search for this author inPubMed Google Scholar

  11. Michael Kluge

    You can also search for this author inPubMed Google Scholar

  12. Holger Mickler

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toMarc C. Wiedemann.

Additional information

We want to express our gratitude to the “Deutsches Zentrum für Luft- und Raumfahrt e.V.” as responsible project agency and to the “Bundesministerium für Bildung und Forschung” for the financial support under grant01 IH 11008 A-C.

Rights and permissions

About this article

Cite this article

Wiedemann, M.C., Kunkel, J.M., Zimmer, M.et al. Towards I/O analysis of HPC systems and a generic architecture to collect access patterns.Comput Sci Res Dev28, 241–251 (2013). https://doi.org/10.1007/s00450-012-0221-5

Download citation

Keywords

Access this article

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Advertisement

[8]ページ先頭
©2009-2025 Movatter.jp