MPI: Return of the MPI Datatypes

Lower Bounds, Upper Bounds, and Extents

The vehicle example introduced one of the most confusing parts of datatypes: bounds and extents. Every datatype has a lower bound, upper bound, and extent. The lower bound is the offset from the start of the user buffer to the start of the first datatype entry for the buffer. In the vehicle example above, if the fuel entry was first instead of last, MPI would need to know that it should skip over the fuel entry to find the destination entry. In this case, the lower bound would be sizeof(double). MPI_ADDRESS can be used to compute the lower bound, similar to how offsets between datatype entries are found. The lower bound can either be adjusted using MPI_TYPE_CREATE_RESIZED or MPI_TYPE_LB.

The upper bound is end of the last element in a datatype, plus any required padding. If there were an array of a given datatype, the start of the next entry would be directly after the upper bound of the current entry. The extent is the size of the datatype, or the upper bound minus the lower bound. Although the datatype's upper bound can be set using MPI_TYPE_UB, it is often much easier and less error prone to set the extent using MPI_TYPE_CREATE_RESIZED.

One-off Datatypes

In each of the datatype examples presented thus far, an instance of a structure is used to determine addresses of each element in the datatype. The addresses are then used to determine offsets to use in the datatype. The resulting datatype can be used to describe any instance of the same structure. However, there are some instances where a "one-off" datatype is created to describe a structure that will only exist once. In these cases, determining addresses to find offsets, only to use the offsets to recompute the addresses is wasteful.

MPI provides the constant MPI_BOTTOM to for instances where computing offsets will be wasteful.

Listing 4: Simple use of MPI_BOTTOM
MPI_Send(MPI_BOTTOM, 5, custom_type,...)

The MPI will still have to do some offset math in order to find the elements in the entire array. MPI_BOTTOM can be tempting, as it saves a couple lines of code. However, MPI_BOTTOM should generally be avoided. One of the advantages of datatypes is that they can be reused to avoid errors in user applications. If absolute addresses are used with MPI_BOTTOM, it is not possible to reuse the datatype in a generic way.

Common Pitfalls and Misconceptions

One common misconception with MPI datatypes is that they are slow. Early in the life of MPI, using MPI datatypes to pack messages was often slower than packing the data by hand. Datatype performance has been and continues to be an active area of research, allowing datatype implementations to achieve much higher performance. Some MPI implementations are even capable of doing scatter/gather sends and receives, completely eliminating the need to pack messages for transfer. In short, poor datatype performance is generally a thing of the past and getting better every day.

MPI provides huge, often overwhelming, number of options when working with datatypes. Although it is often tempting to use the predefined datatypes and avoid complexity, proper use of datatypes can reduce errors and improve performance. Using a complex datatype removes the problem of ensuring the correct order of sends and receives to move a structure piecemeal.

Where to Go From Here?

This column provides a number of examples of using datatypes to their full potential. The resources listed in the side bar present even more examples of utilizing datatypes to simplify applications. Next month, we will move on to any implementor's favorite subject: common mistakes in using MPI and how to avoid them. {mosgoogle right}

Resources
MPI Forum (MPI-1 and MPI-2 specifications documents) http://www.mpi-forum.org
MPI - The Complete Reference: Volume 1, The MPI Core (2nd ed) (The MIT Press) By Marc Snir, Steve Otto, Steven Huss-Lederman, David Walker, and Jack Dongarra. ISBN 0-262-69215-5
MPI - The Complete Reference: Volume 2, The MPI Extensions (The MIT Press) By William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, William Saphir, and Marc Snir. ISBN 0-262-57123-4.
NCSA MPI tutorial http://webct.ncsa.uiuc.edu:8900/public/MPI/

This article was originally published in ClusterWorld Magazine. It has been updated and formated for the web. If you want to read more about HPC clusters and Linux, you may wish to visit Linux Magazine.

Brian Barrett is a Ph.D. Candidate at Indiana University, and is one of the core developers of Open MPI.

    Search

    Feedburner

    Login Form

    Share The Bananas


    Creative Commons License
    ©2005-2012 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.