- Notifications
You must be signed in to change notification settings - Fork6
Lemon is an MPI parallel I/O library that is intended to allow for efficient parallel I/O of both binary and metadata on massively parallel architectures. Data is stored in the SciDAC Lattice QCD Interchange Message Encapsulation format, that allows for storing large blocks of binary data and corresponding metadata in the same file.
License
NotificationsYou must be signed in to change notification settings
etmc/lemon
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
A longer description of this library, including a detailed overview of all the functions that are part of its API, has been published as A. Deuzeman, S.F. Reker and C. Urbach, for the ETM Collaboration,"Lemon: an MPI parallel I/O library for data encapsulation using LIME"and is currently available fromhttp://arxiv.org/abs/1106.4177This README is intended to help new users get Lemon up and running quickly. Upondownloading and extracting the Lemon tarball, one should have a obtained a directory containing, amongst this README file, a configure script and several subdirectories. These subdirectories include all the headers associated with thelibrary (in the directory include), the source code itself (in the directorysrc) and some binaries that do not themselves form part of Lemon (in thedirectory check).The GNU build system is used for Lemon and the usual configure script isprovided. Configuring Lemon should be straightforward, as only the commonarguments are taken into consideration. Of relevance is mainly the --prefixargument, that will set the install directory. It is crucial, however, to eitheruse an MPI wrapper around the compiler, or set the include and linker pathssuch that MPI can be found. This can be done by setting environment variables,e.g. CC=mpiccOnce the library has been configured, make will compile both the library itselfand two binaries that can be found in the test subdirectory (The system willcall aclocal-1.9, automake-1.9 and autoconf. This will result in an error ifonly newer versions are available, but one can simply invoke aclocal, automakeand autoconf manually and call make again to continue). These two binaries notonly provide short, self-contained samples of Lemon usage, but are in factpotentially useful in a production environment. The first, lemon_contents, is adirect port of the C-LIME lime_contents program. It displays a short overview ofall the records available within a particular LIME file. If the contents of arecord data block are both non-binary and short enough, lemon_contents will sendit to standard output. Potential uses for lemon_contents include checking if aLIME file is well-formed and displaying the metadata associated with aparticular file. The second program that is compiled by default islemon_benchmark. This executable will generate an artificial lattice of randomdata, the topology of which will be generated automatically taking into accountthe number of MPI processes. This lattice is written out and subsequentlyread in, both using Lemon's parallel I/O routines. The I/O speed is calculatedfrom the timing of both operations and reported on standard output. Two arguments should be provided. The first is the length L of the spatial dimensions of the lattice, which determines the global size of the lattice as L^3 x 2 L. The secondis the number of iterations of the write-read cycle. To check the correct operation of the library, an MD5 hash is calculated for the data before and after the I/O operations and any discrepancies will be reported. The MD5 implementation included here was written by L. Peter Deutsch and is available freely fromhttp://sourceforge.net/projects/libmd5-rfc/files/. This code is intended to allow the user to obtain basic information on the I/O performance on his or her particularsystem and to give some indication of the scaling. Of course, it also functions asa basic tool for detecting major problems in the writing or retrieving of data. After running make, the user can run make install to install the library, its headers and the two binaries described above to the directory specified by the --prefix argument to configure (or the default location if that argument has been omitted). Additional binaries can still be compiled by calling make check. These willinclude the writing and reading of an artifical metadata record, both usingblocking and non-blocking I/O (xlf and xlf_non_blocking}), that can be displayedusing lemon_contents. Also provided are two programs that write a small amountof data using the mapped parallel I/O routines, again in a blocking andnon-blocking version (parallel and parallel_non_blocking). The data written willbe a set of characters identifying a particular MPI process, such that theoutput file demonstrates the linearisation of the data. The last sample programthat has been included is canonical, a skeleton implementation of the library.None of these binaries will be installed, as they serve no particular functionother than being examples.As an example, an MPI run of lemon_benchmark on 4 MPI processes, performing 3measurements of the IO timings for an 8 x 8 x 8 x 16 lattice should provide thefollowing result (your reading and writing speeds and times may vary):mpirun -np 4 lemon_benchmark 8 3Benchmark on a block of data 4.72 MB in size,representing a 8 x 8 x 8 x 16 lattice,distributed over 4 MPI processesfor a local 4 x 4 x 8 x 16 lattice.Measurement 1 of 3.Time spent writing was 0.15 s.Time spent reading was 0.016 s.All nodes report that MD5 hash matches.Measurement 2 of 3.Time spent writing was 0.2 s.Time spent reading was 0.017 s.All nodes report that MD5 hash matches.Measurement 3 of 3.Time spent writing was 0.21 s.Time spent reading was 0.018 s.All nodes report that MD5 hash matches.Average time spent writing was 1.88e-01 s, with a standard deviation of 2.50e-02 s.Average time spent reading was 1.68e-02 s, with a standard deviation of 1.05e-03 s.Average writing speed was 25.1 MB/sAverage reading speed was 281 MB/sAll data hashed correctly.
About
Lemon is an MPI parallel I/O library that is intended to allow for efficient parallel I/O of both binary and metadata on massively parallel architectures. Data is stored in the SciDAC Lattice QCD Interchange Message Encapsulation format, that allows for storing large blocks of binary data and corresponding metadata in the same file.
Resources
License
Stars
Watchers
Forks
Packages0
No packages published