Repository files navigation

This software benchmarks the performance ofPnetCDF and MPI-IO methods for the I/Opattern used by the NASA's NAS Parallel Benchmarks (NPB) suite(http://www.nas.nasa.gov/publications/npb.html).The evaluation method isstrong scaling.

BTIO presents a block-tridiagonal partitioning pattern on a three-dimensionalarray across a square number of MPI processes. Each process is responsible formultiple Cartesian subsets of the entire data set, whose number increases withthe square root of the number of processes participating in the computation. Asingle global array with anunlimited dimension is created as a netCDF recordvariable in the output file. The array is of five dimensions and onlypartitioned among processes along the middle three dimensions. Eachrecordis a subarray of the least significant four dimensions. The number of recordsto write and read is user adjustable. All records are consecutively written toa shared file in parallel by appending one record after another. The arrayvariable is stored in the file in a canonical, row-major order. To measure theread performance, the global variable is later read back, using the same datapartitioning pattern. The size of global array can also be adjusted in theinput parameter file 'inputbt.data'. Figure 1 below shows an illustration ofdata partitioning pattern for the case when the number of MPI processes is 9.For more detailed description, please refer to:

• Wei-keng Liao. "Design and Evaluation of MPI File Domain PartitioningMethods under Extent-Based File Locking Protocol", in the IEEE Transactionson Parallel and Distributed Systems, 22(2):260-272, February 2011.

Figure 1. BTIO data partitioning pattern, The 4D subarray in each processis mapped to the global array in a block-tridiagonal fashion. This example uses9 processes and highlights the mapping for process P6.

Edit./Makefile and change the following 3 variables.

        MPIF90        -- MPI Fortran compiler        FCFLAGS       -- compile flag        PNETCDF_DIR   -- path of PnetCDF library (1.4.0 and higher is required)

For example:

        MPIF90      = /usr/bin/mpif90        FCFLAGS     = -O2        PNETCDF_DIR = ${HOME}/PnetCDF

Run commandmake to build the executable, namedbtio, in the current folder.

The input parameter file named 'inputbt.data' is required to run the benchmark.An example is provided in the current folder and shown below. Users can adjustthe parameters in the file.

        w                  # IO mode: w for write, r for read        3                  # IO method: 0 for MPI collective IO, 1 for MPI independent IO, 2 for PnetCDF blocking I/O, 3 for PnetCDF nonblocking I/O        40                 # number of writes/reads        512 512 512        # grid_points(1), grid_points(2), grid_points(3)        /scratch2/scratchdirs/wkliao/FS_1M_128

which set

• (first line) w is to perform write operations only
• (second line) 3 is the I/O method to use PnetCDF nonblocking APIs
• (third line) 40 is the number of global arrays to write to the file
• (fourth line) 512 512 512 are the 3D global array sizes
• (fifth line) the input/output directory name

Note thatbtio creates a file namedbtio.nc in the output directorycontaining a 5D array variable namedvar of sizeNUM_DUMPS x Z x Y x X x FIVE_DBL. The variable's data type is an 8-bytedouble precision. Theunlimited dimensionNUM_DUMPS corresponds to the number of record writes (orreads for read case). The dimension sizes of global array areZ,Y, andXcorrespond togrid_points(3),grid_points(2),grid_points(1),respectively.FIVE_DBL is the fifth dimension of size 5, which is notpartitioned among processes.

In the above example, NUM_DUMPS = 40, Z = 512, Y = 512, X = 512, and FIVE_DBL =5. When I/O method is MPI collective I/O, there arencells number ofcollective I/O calls per global variable and ncells = square root of number ofMPI processes. When I/O method is MPI independent I/O, there are alsoncellsnumber of independent I/O calls per global variable made by each MPI process.When I/O method is PnetCDF blocking I/O, there arencells number ofcollective PnetCDF I/O calls per global variable. In this case, there arencells MPI collective I/O calls underneath PnetCDF per global variable. WhenI/O method is PnetCDF nonblocking I.O, there arencells number of nonblockingPnetCDF I/O calls posted first, followed by annfmpi_waitall() call to flushthe requests. In this case, there is only one MPI collective I/O callsunderneath PnetCDF per global variable.

Example command to run an MPI job:

        mpiexec -n 1024 ./btio

or

        mpiexec -n 1024 ./btio inputbt.data

The only optional command-line argument is the input parameter file name. Inthis example, it isinputbt.data. This argument allows to use a differentinput file name besides the defaultinputbt.data. If this argument is notprovided, the default is./inputbt.data under the current directory.

    -- BT-IO Benchmark (write operation only) --    Number of MPI processes  :      1024    Global array size X      :       512    Global array size Y      :       512    Global array size Z      :       512    Number of I/O iterations :        40    Total I/O amount         :    204800.00 MiB    Time in sec              :        41.95    I/O bandwidth                   4882.00 MiB/s    ------------------------------------------    Using Parallel netCDF non-blocking I/O method    output file path         : /scratch2/scratchdirs/wkliao/FS_1M_128    file striping count      :       128    file striping size       :   1048576 bytes

A single netCDF file namedbtio.nc will be created once the run completes.To show the file header, use the netCDF utility programncdump.

% ncdump -h ./btio.ncnetcdf btio {dimensions:        FIVE_DBL = 5 ;        X = 512 ;        Y = 512 ;        Z = 512 ;        NUM_DUMPS = UNLIMITED ; // (40 currently)variables:        double var(NUM_DUMPS, Z, Y, X, FIVE_DBL) ;}

email:wkliao@eecs.northwestern.edu

Copyright (C) 2013, Northwestern University.

See COPYRIGHT notice in top-level directory.