D R A F T

Document for a Standard Message-Passing Interface

Message Passing Interface Forum

November 2, 1993

This work was supported in part by ARPA and NSF under grant ASC-9310330, the

National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615, and by the Commission of the European Community

through Esprit project P6643.

The Message Passing Interface Forum (MPIF), with participation from over 40 organizations, has been meeting since January 1993 to discuss and define a set of library interface standards for message passing. MPIF is not sanctioned or supported by any official standards organization.

This is a draft of what will become the Final Report, Version 1.0, of the Message Passing Interface Forum. This document contains all the technical features proposed for the interface. This copy of the draft was processed by LaTEX on November 2, 1993.

MPIF invites comments on the technical content of MPI, as well as on the editorial presentation in the document. Comments received before January 15, 1994 will be considered in producing the final draft of Version 1.0 of the Message Passing Interface Specification.

The goal of the Message Passing Interface, simply stated, is to develop a widely used standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing.

\begin{matrix}c \\ ◯ \end{matrix}1993 University of Tennessee, Knoxville, Tennessee. Permission to copy without fee all or part of this material is granted, provided the University of Tennessee copyright notice and the title of this document appear, and notice is given that copying is by permission of the University of Tennessee.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Contents

Acknowledgments vii

1 Introduction to MPI 1 1.1 Overview and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Who Should Use This Standard? . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 What Platforms Are Targets For Implementation? . . . . . . . . . . . . . . 3 1.4 What Is Included In The Standard? . . . . . . . . . . . . . . . . . . . . . . 3 1.5 What Is Not Included In The Standard? . . . . . . . . . . . . . . . . . . . . 3 1.6 Organization of this Document . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 MPI Terms and Conventions 6 2.1 Procedure Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Semantic Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3.1 Opaque objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3.2 Array arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.3 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.4 Named constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.5 Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.6 Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4 Language Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4.1 Data Types and Naming Conventions . . . . . . . . . . . . . . . . . 9 2.4.2 Fortran 77 Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4.3 C Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.5 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.6 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.7 Implementation issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.7.1 Independence of basic runtime routines . . . . . . . . . . . . . . . . 12 2.7.2 Interaction with signals in POSIX . . . . . . . . . . . . . . . . . . . 13

3 Point to Point Communication 14 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2 Basic send operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2.1 Message data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2.2 Message envelope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3 Basic receive operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.1 Return status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.4 Semantics of point-to-point communication . . . . . . . . . . . . . . . . . . 19

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

3.5 Data Type Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.6 Data conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.7 Communication Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.8 Nonblocking communication . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.8.1 Communication Objects . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.8.2 Communication initiation . . . . . . . . . . . . . . . . . . . . . . . . 27 3.8.3 Communication Completion . . . . . . . . . . . . . . . . . . . . . . . 30 3.8.4 Semantics of Nonblocking Communications . . . . . . . . . . . . . . 31 3.8.5 Multiple Completions . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.9 Probe and Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.10 Persistent communication objects . . . . . . . . . . . . . . . . . . . . . . . . 38 3.11 Send-receive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.12 Null processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.13 Derived datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.13.1 Datatype constructors . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.13.2 Address and extent functions . . . . . . . . . . . . . . . . . . . . . . 52 3.13.3 Lower-bound and upper-bound markers . . . . . . . . . . . . . . . . 53 3.13.4 Commit and free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.13.5 Use of general datatypes in communication . . . . . . . . . . . . . . 55 3.13.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.13.7 Correct use of addresses . . . . . . . . . . . . . . . . . . . . . . . . . 65

3.14 Universal communication functions . . . . . . . . . . . . . . . . . . . . . . . 66 3.14.1 Persistent communication objects . . . . . . . . . . . . . . . . . . . . 67 3.14.2 Nonblocking communication initiation . . . . . . . . . . . . . . . . . 68 3.14.3 Communication completion . . . . . . . . . . . . . . . . . . . . . . . 68 3.14.4 Blocking communication . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.14.5 Probe and cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.14.6 Return status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.14.7 send-receive and exchange . . . . . . . . . . . . . . . . . . . . . . . . 69 3.14.8 Derived datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4 Collective Communication 70 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2 Communication Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.3 Barrier synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.4 Data move functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.4.1 Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.4.2 Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.4.3 Examples of Usage of MPI GATHER, MPI GATHERV . . . . . . . . . 75 4.4.4 Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4.5 Examples of Usage of MPI SCATTER, MPI SCATTERV . . . . . . . . 84 4.4.6 Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.4.7 All-to-All Scatter/Gather . . . . . . . . . . . . . . . . . . . . . . . . 88

4.5 Global Compute Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.5.1 Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.5.2 User-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.5.3 All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 4.5.4 Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

4.5.5 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.6 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

4.6.1 Synchronization Side-Effects . . . . . . . . . . . . . . . . . . . . . . . 100 4.6.2 Multiple Calls to Collective . . . . . . . . . . . . . . . . . . . . . . . 101

5 Groups, Contexts, Communicators, and Caching 104

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 5.1.1 Why we need libraries . . . . . . . . . . . . . . . . . . . . . . . . . . 104 5.1.2 Why existing systems do not support libraries . . . . . . . . . . . . . 104 5.1.3 What features are needed to support libraries . . . . . . . . . . . . . 105 5.1.4 What these features are called in MPI . . . . . . . . . . . . . . . . . 105

5.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.2.1 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.2.2 Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.2.3 Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.2.4 Predefined Communicators . . . . . . . . . . . . . . . . . . . . . . . 107

5.3 Group Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.3.1 Group Accessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.3.2 Group Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.3.3 Group Destructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

5.4 Communicator Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.4.1 Communicator Accessors . . . . . . . . . . . . . . . . . . . . . . . . 112 5.4.2 Communicator Constructors . . . . . . . . . . . . . . . . . . . . . . . 113 5.4.3 Communicator Destructors . . . . . . . . . . . . . . . . . . . . . . . 115

5.5 Inter-Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.5.1 Communicator Accessors . . . . . . . . . . . . . . . . . . . . . . . . 117 5.5.2 Intercommunicator Constructors and Destructors . . . . . . . . . . . 117 5.5.3 Name Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

5.6 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

5.6.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.7 Formalizing the Loosely Synchronous Model (Usage, Safety) . . . . . . . . . 125

5.7.1 Basic Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.7.2 Models of Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

5.8 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.8.1 Current Practice #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.8.2 Current Practice #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.8.3 (Approximate) Current Practice #3 . . . . . . . . . . . . . . . . . . 128 5.8.4 Example #4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 5.8.5 Library Example #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5.8.6 Library Example #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 5.8.7 Inter-Communication Examples . . . . . . . . . . . . . . . . . . . . . 133

6 Process Topologies 139 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.2 Virtual Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.3 Embedding in MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.4 Overview of the proposed MPI functions . . . . . . . . . . . . . . . . . . . . 140 6.5 Topology Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

6.5.1 Topology inquiry functions . . . . . . . . . . . . . . . . . . . . . . . 144 6.5.2 Cartesian shift coordinates . . . . . . . . . . . . . . . . . . . . . . . 148 6.5.3 Partitioning of Cartesian structures . . . . . . . . . . . . . . . . . . 149 6.5.4 Low–level topology functions . . . . . . . . . . . . . . . . . . . . . . 150

6.6 A simple example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

7 MPI Environmental Management 153

7.1 Implementation information . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 7.1.1 Environmental Inquiries . . . . . . . . . . . . . . . . . . . . . . . . . 153 7.1.2 Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

7.2 Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 7.3 Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

8 Profiling Interface 162 8.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 8.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 8.3 Logic of the design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 8.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

8.4.1 Profiler implementation . . . . . . . . . . . . . . . . . . . . . . . . . 163 8.4.2 MPI library implementation . . . . . . . . . . . . . . . . . . . . . . . 164

8.5 Multiple levels of interception . . . . . . . . . . . . . . . . . . . . . . . . . . 165

9 Initial Implementation Subset 166 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 9.2 Criteria and Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 9.3 Subset Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

9.3.1 Point to Point Functionality . . . . . . . . . . . . . . . . . . . . . . . 167 9.3.2 Collective Communication Functionality . . . . . . . . . . . . . . . . 169 9.3.3 Groups, Contexts, Communicators, and Cacheing . . . . . . . . . . . 169 9.3.4 Topology Functionality . . . . . . . . . . . . . . . . . . . . . . . . . 170 9.3.5 Language Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 9.3.6 MPI Environmental Management . . . . . . . . . . . . . . . . . . . . 171 9.3.7 Profiling Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . 171

9.4 Subset Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

Bibliography 173

A Language Binding 175 A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 A.2 Defined Constants for C and Fortran . . . . . . . . . . . . . . . . . . . . . . 175 A.3 C bindings for Point-to-Point Communication . . . . . . . . . . . . . . . . . 177 A.4 C Bindings for Collective Communication . . . . . . . . . . . . . . . . . . . 180 A.5 C Bindings for Groups, Contexts, and Communicators . . . . . . . . . . . . 181 A.6 C Bindings for Process Topologies . . . . . . . . . . . . . . . . . . . . . . . 182 A.7 C bindings for Environmental Inquiry . . . . . . . . . . . . . . . . . . . . . 183 A.8 Fortran Bindings for Point-to-Point Communication . . . . . . . . . . . . . 183 A.9 Fortran Bindings for Collective Communication . . . . . . . . . . . . . . . . 186 A.10 Fortran Bindings for Groups, Contexts, and Communicators . . . . . . . . . 189 A.11 Fortran Bindings for Process Topologies . . . . . . . . . . . . . . . . . . . . 190

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

A.12 Fortran Bindings for Environmental Inquiry . . . . . . . . . . . . . . . . . . 191

MPI Function Index 193

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Acknowledgments

The technical development was carried out by subgroups, whose work was reviewed by the full committee. During the period of development of the Message Passing Interface (MPI), many people served in positions of responsibility:

• Jack Dongarra, David Walker, Conveners and Meeting Chairs

• Ewing Lusk, Bob Knighten, Minutes

• Marc Snir, William Gropp, Ewing Lusk, Point to Point Communications

• Al Geist, Marc Snir, Steve Otto, Collective Communications

• Rolf Hempel, Process Topologies

• Ewing Lusk, Language Binding

• William Gropp, Environmental Management and Inquiry

• James Cownie, Profiling

• Tony Skjellum, Lyndon Clarke, Marc Snir, Richard Littlefield, Mark Sears, Groups, Contexts, and Communicators

• Steven Huss-Lederman, Initial Implementation Subset

• Steve Otto, Editor

The following list includes some of the active participants in the MPI process not mentioned above:

Ed Anderson  Joe Baron  Eric Barszcz  Scott Berryman
Rob Bjornson  Anne Elster  Jim Feeney  Vince Fernando
Sam Fineberg  Jon Flower  Daniel Frye  Ian Glendinning
Adam Greenberg Robert Harrison Leslie Hart  Tom Haupt
Don Heller  Tom Henderson Alex Ho  C.T. Howard Ho
John Kapenga  Bob Leary  Arthur Maccabe Peter Madams
Alan Mainwaring Oliver McBryan Phil McKinley  Charles Mosher
Dan Nessett  Peter Pacheco  Howard Palmer  Paul Pierce
Sanjay Ranka  Peter Rigsbee  Arch Robison  Erich Schikuta
Ambuj Singh  Alan Sussman  Robert Tomlinson Robert G. Voigt
Dennis Weeks  Stephen Wheat Steve Zenith

Oak Ridge National Laboratory made the draft available by anonymous FTP mail servers and was instrumental in distributing the document.

MPI operated on a very tight budget (in reality, it had no budget when the first meeting was announced). DARPA and NSF have supported research at various institutions that have made a contribution towards travel for the U.S. academics. Support for several European participants was provided by ESPRIT.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 1

Introduction to MPI

1.1 Overview and Goals

Message passing is a paradigm used widely on certain classes of parallel machines; especially those with distributed memory. Although there are many variations, the basic concept of processes communicating through messages is well understood. Over the last ten years, substantial progress has been made in casting significant applications in this paradigm. Each vendor has implemented its own variant. More recently, several systems have demonstrated that a message passing system can be efficiently and portably implemented. It is thus an appropriate time to try to define both the syntax and semantics of a core of library routines that will be useful to a wide range of users and efficiently implementable on a wide range of computers.

In designing MPI we have sought to make use of the most attractive features of a number of existing message passing systems, rather than selecting one of them and adopting it as the standard. Thus, MPI has been strongly influenced by work at the IBM T. J. Watson Research Center [1, 2], Intel’s NX/2 [18], Express [17], nCUBE’s Vertex [15], P4 [5], and PARMACS [4, 6]. Other important contributions have come from Zipcode [19, 20], Chimp [11, 12], PVM [3, 9], and PICL [14].

The MPI standardization effort involved about 60 people from 40 organizations mainly from the United States and Europe. Most of the major vendors of concurrent computers were involved in MPI, along with researchers from universities, government laboratories, and industry. The standardization process began with the Workshop on Standards for Message Passing in a Distributed Memory Environment, sponsored by the Center for Research on Parallel Computing, held April 29-30, 1992, in Williamsburg, Virginia [21]. At this workshop the basic features essential to a standard message passing interface were discussed, and a working group established to continue the standardization process.

A preliminary draft proposal, known as MPI1, was put forward by Dongarra, Hempel, Hey, and Walker in November 1992, and a revised version was completed in February 1993 [10]. MPI1 embodied the main features that were identified at the Williamsburg workshop as being necessary in a message passing standard. This proposal was intended to initiate discussion of standardization issues within the distributed memory concurrent computing community. Since MPI1 was primarily intended to promote discussion and “get the ball rolling,” it focused mainly on point-to-point communications. MPI1 did not include any collective communication routines. MPI1 brought to the forefront a number of important standardization issues; however, its major deficiency was that the management of resources

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

is not thread-safe.

In November 1992, a meeting of the MPI working group was held in Minneapolis, at which it was decided to place the standardization process on a more formal footing, and to generally adopt the procedures and organization of the High Performance Fortran Forum. Subcommittees were formed for the major component areas of the standard, and an email discussion service established for each. In addition, the goal of producing a draft MPI standard by the Fall of 1993 was set. To achieve this goal the MPI working group met every 6 weeks for two days throughout the first 9 months of 1993, and presented the draft MPI standard at the Supercomputing 93 conference in November 1993. These meetings and the email discussion together constituted the MPI Forum, membership of which has been open to all members of the high performance computing community.

The main advantages of establishing a message passing standard are portability and ease-of-use. In a distributed memory communication environment in which the higher level routines and/or abstractions are build upon lower level message passing routines the benefits of standardization are particularly apparent. Furthermore, the definition of a message passing standard, such as that proposed here, provides vendors with a clearly defined base set of routines that they can implement efficiently, or in some cases provide hardware support for, thereby enhancing scalability.

The goal of the Message Passing Interface simply stated is to develop a widely used standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing.

A complete list of goals follows.

• Design an application programming interface (not necessarily for compilers or a system implementation library).

• Allow efficient communication: Avoid memory to memory copying and allow overlap of computation and communication and offload to communication co-processor, where available.

• Allow for implementation that can be used in a heterogeneous environment.

• Allow convenient C and Fortran 77 bindings for the interface.

• Assume a reliable communication interface: User need not cope with communication failures. Such failures are dealt with by the underlying communication subsystem.

• Define an interface that is not too different from current practice, such as PVM, Express, P4, etc and provides extension that allow greater flexibility.

• Define an interface that can be implemented on many vendor’s platforms, with no significant changes in the underlying communication and system software.

• Semantics of the interface should be language independent.

• The interface should be designed to allow for thread-safety.

1.2 Who Should Use This Standard?

This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and C. This includes individual application programmers, developers

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

of software designed to run on parallel machines, and creators of environments and tools. In order to be attractive to this wide audience, the standard must provide a simple, easyto-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines.

1.3 What Platforms Are Targets For Implementation?

The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way may run on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those “machines” consisting of collections of other machines, parallel or not, connected by a communication network.

The interface is suitable for use by fully general MIMD programs, as well as those written in the more restricted style of SPMD. Although no explicit support for threads is provided, the interface has been designed so as not to prejudice their use. With this version of MPI no support is provided for dynamic spawning of tasks.

1.4 What Is Included In The Standard?

The standard includes:

• Point-to-point communication

• Collective operations

• Process groups

• Communication contexts

• Process topology

• Bindings for Fortran 77 and C

• Environmental Management and inquiry

• Profiling interface

• Subset specification

1.5 What Is Not Included In The Standard?

The standard does not specify:

• Explicit shared-memory operations

• Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

• Program construction tools

• Debugging facilities

• Auxiliary functions such as timers

• Explicit support for threads

• Support for task management

• I/O functions

There are many features that have been considered and not included in this standard. This happened for a number of reasons, one of which is the time constraint that was self imposed in finishing the standard. Features that are not included can always be offered as extensions by specific implementations. Perhaps future versions of MPI will address some of these issues.

1.6 Organization of this Document

The following is a list of the chapters in this document, along with a brief description of each.

• Chapter 2, MPI Terms and Conventions, explains notational terms and conventions used throughout the MPI document.

• Chapter 3, Point to Point Communication, defines the basic, pairwise communication subset of MPI. send and receive are found here, along with many associated functions designed to make basic communication powerful and efficient.

• Chapter 4, Collective Communications, defines process-group collective communication operations. Well known examples of this are barrier and broadcast over a group of processes (not necessarily all the processes).

• Chapter 5, Groups, Contexts, and Communicators, shows how groups of processes are formed and manipulated, how unique communication contexts are obtained, and how the two are bound together into a communicator.

• Chapter 6, Process Topologies, explains a set of utility functions meant to assist in the mapping of process groups (a linearly ordered set) to richer topological structures such as multi-dimensional grids.

• Chapter 7, MPI Environmental Management, explains how the programmer can manage and make inquiries of the current MPI environment. These functions are needed for the writing of correct, robust programs, and are especially important for the construction of highly-portable message-passing programs.

• Chapter 8, Profiling Interface, explains a simple name shifting convention that any MPI implementation must support. One motivation for this is the ability to put performance profiling calls into MPI without the need for access to the MPI source code. The name shift is merely an interface, it says nothing about how the actual profiling should be done and in fact, the name shift can usefully be used for other purposes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

• Chapter 9, Initial Implementation Subset, suggests to MPI implementors a “core” subset of MPI that is useful, internally consistent, and should appear first in the evolution of an MPI implementation. The subset is defined so that consistent implementations can appear rapidly, and portable parallel programming with MPI can start in a timely manner.

• Annex A, Language Bindings, gives specific syntax in Fortran 77 and C, for all MPI functions, constants, and types.

• The MPI Function Index is a simple index showing the location of the languageindependent definition of each MPI function.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 2

MPI Terms and Conventions

This chapter explains notational terms and conventions used throughout the MPI document, some of the choices that have been made, and the rationale behind those choices.

2.1 Procedure Specification

MPI procedures are specified using a language independent notation. The arguments of procedure calls are marked as IN, OUT or INOUT. The meanings of these are:

• the call uses but does not update an argument marked IN,

• the call may update an argument marked OUT,

• the call both uses and updates an argument marked INOUT.

There is one special case — if an argument is a handle to an opaque object, and the object is updated by the procedure call, then the argument is marked OUT. This is done even though the handle itself is not modified — we use the OUT attribute to denote that what the handle references is updated.

The definition of MPI tries to avoid, to the largest possible extent, the use of INOUT arguments, because such use is error-prone, especially for scalar arguments.

A common occurrence for MPI functions is an argument that is used as IN by some processes and OUT by other processes. Such arguments are, syntactically, an INOUT argument, although, semantically, they are not used in one call both for input and for output.

Another frequent situation arises when an argument value is needed only by a subset of the processes. When an argument is not significant at a process then an arbitrary value can be passed as argument.

2.2 Semantic Terms

When discussing point-to-point and collective communications the following terms are used:

nonblocking if the procedure may return before the related communication operation completes, and before the user is allowed to re-use resources (such as buffers) specified in the call;

locally-blocking if return from the procedure indicates the user is allowed to re-use re-

sources specified in the call;

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

globally-blocking if return from the procedure indicates the communication operation has completed.

When discussing the behavior of any MPI function, not necessarily a communication function, the following terms are used.

local An operation is local if its completion depends only on the local executing process. Such an operation does not require communication with another user process.

non-local An operation is non-local if its completion may depend on communication occurring with another user process.

collective An operation is collective if all processes in a process group need to perform the operation.

2.3 Data Types

2.3.1 Opaque objects

MPI procedures use opaque objects. These hide the internal representation used, thus allowing similar calls in C and Fortran; the avoidance of conflicts with the typing rules in these languages; and future extension of functionality. The mechanism for opaque objects used here loosely follows the POSIX Fortran binding standard.

An opaque object can be allocated in user storage or in system (i.e. MPI managed) storage. We call the former a user object and the latter a system object. User objects are allocated and deallocated explicitly by the user as a regular variable. The type definitions for such objects are provided by MPI. The objects can be accessed and modified only via MPI functions that take such objects as arguments.

System objects are accessed only via handles. In C, a different type of handle is defined by MPI for each type of opaque system object. In Fortran, all handles have type INTEGER. The allocation and deallocation of such an object, and any access or update to it, are done only via MPI calls that accept as arguments handles to such objects.

An opaque object, and the handle to such an object, are significant only at the process where the object was created, and cannot be transferred to another process.

Opaque system objects are allocated and deallocated by calls that are specific to each object type. These are listed in the sections where the objects are described. A call to deallocate an opaque system object does not necessarily deallocate the object immediately, but may only mark it as candidate for deallocation. The object will be deallocated as soon as there is no pending operation that is using this object; the handle then becomes null. For example, if a communication object is freed after the communication started but before it completed, then the pending operation is not affected; one can use the handle in calls to MPI WAIT or MPI STATUS. In fact, one must complete the communication with such a call. However, once the operation completes, it is erroneous to re-use the freed handle for a new communication.

Implementation note: Some objects, such as datatype objects (see section 3.13), can be used by several, distinct, concurrent communication operations. Thus, to find out whether such an object can be freed, one needs a reference count of pending operations.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI does not keep count of references to an opaque MPI object that are created by assigning to one handle the value of another, or are lost when a handle variable is deallocated (e.g., at procedure exit). In the first case, a call to free the object may leave a dangling reference; in the second case an opaque MPI object may be left with no handle to it, and no possibility of deallocating it. It is the user’s responsibility to avoid these situations. MPI keeps track only of changes in the number of active references to the MPI object that are a result of explicit MPI calls.

MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed.

2.3.2 Array arguments

An MPI call may need an argument that is an array of opaque objects, or an array of handles. The array-of-handles is a regular array with entries that are handles to objects of the same type in consecutive locations in the array. Whenever such an array is used, an additional len argument is required to indicate the number of valid entries (unless this number can be derived otherwise). The valid entries are at the begining of the array; len indicates how many of them there are, and need not be the entire size of the array. The same approach is followed for other array arguments.

2.3.3 State

MPI procedures use at various places arguments with state types. The values of such data type are all identified by names, and no operation is defined on them. For example, the MPI SET ERRORMODE routine has a state type argument with values MPI ERRORSFATAL,

MPI ERRORSRETURN, etc.

2.3.4 Named constants

MPI procedures sometimes assign a special meaning to a special value of a basic type argument; e.g. tag is an integer valued argument of point-to-point communication operations, with a special wild-card value, MPI ANY TAG. Such arguments will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as MPI ANY TAG) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environmental inquiry or environmental management functions (Chapter 7).

Implementation note: Implementors should detect, whenever possible, illegal uses of “special values”. Thus, the use of the MPI ANY TAG value to tag a message sent should be flagged as an error.

2.3.5 Choice

MPI functions sometimes use arguments with a choice (or union) data type. Distinct calls to the same routine may pass by reference actual arguments of different types. The mechanism for providing such arguments will differ from language to language.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

2.3.6 Addresses

Some MPI procedures use address arguments that represent an address in the calling program. The datatype of such an argument is an integer of the size needed to hold an address in the execution environment.

2.4 Language Binding

This section defines the rules for MPI language binding in general and for Fortran 77 and ANSI C in particular. Defined here are various object representations, as well as the naming conventions used for expressing this standard. The actual calling sequences are defined elsewhere.

It is expected that any Fortran 90 and C++ implementations use the Fortran 77 and ANSI C bindings, respectively. Although we consider it premature to define other bindings to Fortran 90 and C++, the current bindings are designed to encourage, rather than discourage, experimentation with better bindings which might be adopted later.

Since the word PARAMETER is a keyword in the Fortran language, we use the word “argument” to denote the arguments to a subroutine. These are normally referred to as parameters in C, however we expect that C programmers will understand the word arguments (which has no specific meaning in C), thus allowing us to avoid unnecessary confusion for Fortran programmers.

There are several important language binding issues not addressed by this standard. It is beyond the scope of this standard to discuss the interoperability of message passing between languages. Although it would be convenient to be able to guarantee that messages sent between languages be properly interpreted, the matching of types across languages prevents the inclusion of such a guarantee in this standard. It is fully expected that many implementations will have such features, and that such features are a sign of the quality of the implementation.

2.4.1 Data Types and Naming Conventions

Handles are represented as integers in Fortran and as pointers to structures in C. A special value MPI NULL is used to represent null handles in Fortran; in C, the constant NULL is used as usual.

Typed communication routines support character, double precision, real, and integer data in Fortran. int, double, real, short, char, and long are supported in C, along with unsigned int, unsigned long, unsigned short, and unsigned char. In addition, a special byte type is supported in both languages. This type allows communication to be performed with no type conversion at all. In C, the definitions of named constants, function prototypes, and aggregate type information must be supplied in an include file mpi.h. In Fortran, the corresponding include file for named constants (PARAMETERs) will be called

mpif.h.

2.4.2 Fortran 77 Binding Issues

All MPI names have an MPI prefix, and all characters are capitals. All MPI Fortran subroutines have a return code in the last argument. A few MPI operations are functions, which do not have the return code argument. The return code value for successful completion is

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

         double precision a
         integer b
         call MPI_send(a,...)
             .
             .
         call MPI_send(b,...)

Figure 2.1: This is an example of calling a routine with mismatched formal and actual arguments.

MPI SUCCESS; error codes are implementation dependent. Unless explicitly stated, the MPI F77 binding is consistent with ANSI standard Fortran 77. There are several points where this standard diverges from the ANSI Fortran 77 standard. These exceptions are consistent with common practice in the Fortran community. In particular:

• MPI Identifiers are limited to thirty, not six, significant characters.

• MPI Identifiers may contain underscores after the first character.

• An MPI subroutine with a choice argument may be called with different argument types. An example is shown in Figure 2.1. This violates the letter of the Fortran standard, but such a violation is common practice. The alternative would be to have a separate version of MPI SEND, for example, for each data type.

• Although not required, it is strongly suggested that named MPI constants be provided in an include file, called mpif.h. On systems that do not support include files, the implementation should specify the values of named constants.

• Vendors are encouraged to provide type declarations in the mpif.h file on Fortran systems that support user-defined types. One should define, if possible, the type MPI ADDR, which is an INTEGER of the size needed to hold an address in the execution environment. On systems where type definition is not supported, it is up to the user to use an INTEGER of the right kind to represent addresses (i.e., INTEGER*4 on a 32 bit machine, INTEGER*8 on a 64 bit machine, etc.).

2.4.3 C Binding Issues

We use the ANSI C declaration format. All MPI names have an MPI prefix, defined constants are in all capital letters, and defined types and functions have one capital letter after the prefix.

Almost all C functions return an error code. The successful return code will be MPI SUCCESS, but failure return codes are implementation dependent. A few C functions will return values, so that they can be implemented as macros. Logical flags are integers with values 1 and 0 meaning “true” and “false” respectively. Choice arguments are pointers of type void*.

The type MPI Aint is defined to be an int of the size needed to hold an address on the target architecture.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

2.5 Processes

An MPI program consists of autonomous processes, executing their own code, in an MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each process executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified.

MPI does not specify the execution model for each process. A process can be sequential, or can be multi-threaded, with threads possibly executing concurrently. Care has been taken to make MPI “thread-safe,” by avoiding the use of implicit state.

MPI does not specify either the interaction of processes with signals, in a UNIX environment, or with other events that do not relate to MPI communication. That is, signals are not significant from the view point of MPI, and implementors should attempt to implement MPI so that signals are transparent: an MPI call suspended by a signal should resume and complete after the signal is handled.

MPI does not provide mechanisms to specify the initial allocation of processes to an MPI computation and their binding to physical processors. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow the specification of the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution (the total number of processes is fixed), although it is intended to be consistent with such extensions. Finally, the current proposal does not specify a naming scheme for processes. We propose always to identify processes according to their relative rank in a group, so that, effectively, processes are identified by consecutive integers in the range 0..groupsize-1.

2.6 Error Handling

MPI provides the user with reliable message transmission. A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. If the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementor of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as global program failures. Similarly, MPI itself provides no mechanisms for handling node failures. (A mechanism that allows the user to restrict the scope of an unrecoverable error is presented in section 7.2.)

Of course, MPI programs may still be erroneous. A program error can occur when an MPI call is called with an incorrect argument (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a resource error may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. A high quality implementation will provide generous limits on some of the resources so as to alleviate the portability problem this represents. A program is safe if it is not subject to resource errors, unsafe, otherwise.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

All MPI calls return a code that indicates successful completion of the operation, or the error condition that occurred, otherwise. However, MPI does not require the user to check for error values. It is recommended that an implementation provide an environment for MPI execution that can be set so that any error encountered in an MPI call will cause a program to fail with a suitable error message returned to the user. The environment for an MPI execution can also be set so that (non-fatal) errors do not cause a program to fail, and the user can provide his or her own error recovery code. A mechanism that achieves this in a POSIX environment is described in section 7.2.

MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error argument associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error value can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). Such an error must be treated as fatal, since information cannot be returned for the user to recover from it.

The behavior of erroneous programs is not defined by this document. The desired behavior is that a relevant error code be returned, if no acceptable behavior can be defined. However, efficiency considerations may dictate that the system does not detect certain types of errors. Also, implementations may go beyond this document in supporting in a meanningful manners codes that are defined here to be erroneous.

MPI, like any other similar message-passing library, allows users to write non-deterministic programs, where race conditions exist and different timing of communication may lead to different results. There is no requirement that repeated executions of the same such program will yield the same result, even when repeated with the same physical configuration — neither user nor MPI can be expected to have full control of the execution environment. However, repeatability is a desirable quality-of-implementation goal. In particular, it is expected that MPI implementations will avoid introducing additional non-determinism. For example, while MPI can take advantage of associativity and commutativity in reordering the execution of a global reduce operation, it is desired (but not required) that the same operations order be used in repeated executions.

2.7 Implementation issues

There are a number of areas where an MPI implementation may interact with the operating environment and system. While MPI does not mandate that any services (such as I/O or signal handling) be provided, it does strongly suggest the behavior to be provided if those services are available. This is an important point in achieving portability across platforms that provide the same set of services.

2.7.1 Independence of basic runtime routines

MPI programs require that library routines that are part of the basic language environment (such as date and write in Fortran and printf and malloc in ANSI C) and are executed after MPI INIT and before MPI FINALIZE operate independently and that their completion is independent of the action of other processes in an MPI program.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Note that this in no way prevents the creation of library routines that provide parallel services whose operation is collective. However, the following program is expected to complete in an ANSI C environment regardless of the size of MPI COMM WORLD:

int rank;
MPI_Init( argc, argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
if (rank == 0) printf( "Starting program\n" );
MPI_Finalize();

The corresponding Fortran 77 program is also expected to complete.

An example of what is not required is any particular ordering of the action of these routines when called by several tasks. For example, MPI makes neither requirements or recommendations for the output from

MPI_Comm_rank( MPI_COMM_WORLD, &rank );
printf( "Output from task rank %d\n", rank );

In addition, calls that fail because of resource exhaustion or other error are not considered a violation of the requirements here (however, they are required to complete, just not to complete successfully).

2.7.2 Interaction with signals in POSIX

The intent of MPI is to be thread and signal safe. This has a number of subtle effects. For example, on Unix systems, a catchable signal such as SIGALRM (an alarm signal) must not cause an MPI routine to behave differently than it would have in the absence of the signal. Of course, if the signal handler issues MPI calls or changes the environment in which the MPI routine is operating (for example, consuming all available memory space), the MPI routine should behave as appropriate for that situation (in particular, in this case, the behavior should be the same as for a multithreaded MPI implementation).

A second effect is that a signal handler that performs MPI calls must not interfere with the operation of MPI. For example, an MPI receive of any type that occurs within a signal handler must not cause erroneous behavior by the MPI implementation. Note that an implementation is permitted to prohibit the use of MPI calls from within a signal handler, and is not required to detect such use.

It is highly desirable that MPI not use SIGALRM, SIGFPE, or SIGIO. An implementation is required to clearly document all of the signals that the MPI implementation uses; a good place for this information is a Unix ‘man’ page on MPI.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 3

Point to Point Communication

3.1 Introduction

Sending and receiving of messages by processes is the basic MPI communication mechanism. All other communication functions can be defined and implemented on top of this basic communication layer, using few auxiliary data management routines, and using the group and communicator manipulation functions.

The basic point to point communication operations are send and receive. A send operation creates and sends a message. The operation specifies a send buffer in the sender memory from which the message data is taken. In addition, the send operation associates an envelope with the message. This envelope specifies the message destination and contains distinguishing information that can be used by the receive operation to select a particular message.

A receive operation consumes a message. The message to be received is selected according to the value on its envelope, and the message data is put into the receive buffer.

The next sections describe the basic (blocking) send and receive operations. We discuss send, receive, basic communication semantics, type matching requirements, type conversion in heterogeneous environments, and more general communication modes. Nonblocking communication is addressed next, followed by channel-like constructs and send-receive operations. We then consider general datatypes that allow one to transfer heterogeneous and noncontiguous data, and conclude with a description of an implementation of MPI point to point communication using a small number of primitives.

3.2 Basic send operation

The syntax of the simplest send operation is given below.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI SEND(buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (nonnegative inte-

ger)

IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Send(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
MPI SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

3.2.1 Message data

The send buffer specified by the MPI SEND operation consists of count successive entries of the type indicated by datatype, starting with the entry at address buf. Note that we specify the message length in terms of number of elements, not number of bytes. The former is machine independent and closer to the application level.

The data part of the message consists of a sequence of count values, each of the type indicated by datatype. count may be zero, in which case the data part of the message is empty. The basic datatypes that can be specified for message data values correspond to the basic datatypes of the host language. The possible values of this argument for Fortran and the corresponding Fortran types are listed below

MPI datatype  Fortran datatype
MPI INTEGER  INTEGER
MPI REAL  REAL
MPI DOUBLE PRECISION DOUBLE PRECISION
MPI COMPLEX  COMPLEX
MPI LOGICAL  LOGICAL

                  MPI CHARACTER  CHARACTER(1)

MPI BYTE

The possible values for this argument for C and the corresponding C types are listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI datatype  C datatype

                 MPI CHAR  signed char
                 MPI SHORT  signed short
                 MPI INT  signed int
                 MPI LONG  signed long int

MPI UNSIGNED CHAR unsigned char

                 MPI UNSIGNED SHORT unsigned short int
                 MPI UNSIGNED  unsigned int
                 MPI UNSIGNED LONG unsigned long int

MPI FLOAT  float
MPI DOUBLE  double
MPI LONG DOUBLE  long double

MPI BYTE

The datatype MPI BYTE does not correspond to a Fortran or C datatype. A value of type MPI BYTE consists of 8 binary digits. A byte is uninterpreted and is different from a character. Different machines may have different representations for characters, or may use more than one byte to represent characters. On the other hand, a byte has the same binary value on all machines.

MPI supports the basic datatypes of Fortran 77 and ANSI C. Additional MPI datatypes should be provided if the host language has additional data types: For example, MPI LONG LONG INT, for 64 bit integers in C; MPI DOUBLE COMPLEX for double precision complex in Fortran; MPI REAL2, MPI REAL4 and MPI REAL8 for 2,4 or 8 byte reals, in Fortran; MPI INTEGER1 MPI INTEGER2 and MPI INTEGER4 for 1, 2 or 4 byte integers in Fortran; etc.

3.2.2 Message envelope

In addition to the data part, messages contain information that can be used to distinguish messages and selectively receive them. This information is contained in a fixed number of fixed-format fields, which we collectively call the message envelope. These fields are

source

destination

tag

context

The message source is implicitly determined by the identity of the message sender. The other fields are specified by arguments in the send operation.

The integer-valued message tag is specified by the tag argument. This integer can be used by the program to distinguish different types of messages. The range of valid tag values is implementation dependent and can be queried using the MPI GET VALID TAG RANGE environmental inquiry function, as described in Chapter 7.

The context of the message sent is specified by the comm argument. The message carries the context associated with this communicator (see Chapter 5).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The message destination is specified by the dest argument as a rank within the process group associated with that same communicator (remote process group, for intercommunicators). The range of valid values is 0, ... , n-1, where n is the number of processes in this group. Thus, point-to-point communications do not use absolute addresses, but only relative ranks within a group. This provides important modularity.

The message envelope would normally be encoded by a fixed-length message header. However, the actual mechanism used to associate an envelope with a message is implementation dependent: some of the information (e.g., source or destination) may be implicit, and need not be explicitly carried by messages; processes may be identified by relative ranks, or absolute ids; etc.

3.3 Basic receive operation

The syntax of the simplest receive operation is given below.

MPI RECV (buf, count, datatype, source, tag, comm, status)

OUT  buf  initial address of receive buffer (choice)
IN  count  number of elements in receive buffer (integer)
IN  datatype  datatype of each receive buffer element (handle)
IN  source  rank of source (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Recv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Status *status)
MPI RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE),
    IERROR

The receive buffer consists of the storage containing count consecutive elements of the type specified by datatype, starting at address buf. The length of the received message must be less than or equal to the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. The MPI PROBE function described in section 3.9 can be used to receive messages of unknown length.

The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an envelope pattern; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the source, tag and context fields of the message envelope. The receiver may specify a wildcard MPI ANY SOURCE value for source, and/or a wildcard MPI ANY TAG value for tag, indicating that any source and/or tag are acceptable. It cannot specify a wildcard value for context. Thus, a message can be received by a receive operation only if it is addressed to the receiving process, has a matching context, has matching

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

source unless source=MPI ANY SOURCE in the pattern, and has a matching tag unless tag= MPI ANY TAG in the pattern.

The message tag is specified by the tag argument of the receive operation. The message context is the context attached with the communicator specified by the argument comm. The message source, if different from MPI ANY SOURCE, is specified as a rank within the process group associated with that same communicator (remote process group, for intercommunicators). Thus, the range of valid values for the source argument is { 0, ... , n-1 }∪{ MPI ANY SOURCE }, where n is the number of processes in this group.

Note the asymmetry between send and receive operations: A receive operation may accept messages from an arbitrary sender; on the other hand, a send operation must specify a unique receiver. This matches a “push” communication mechanism, where data transfer is effected by the sender (rather than a “pull” mechanism, where data transfer is effected by the receiver).

Source = destination is allowed: a process can send a message to itself.

3.3.1 Return status

The source or the tag of a received message may not be known if wildcard values were used in the receive operation. Also, the actual length of the message received may not be known. Thus, this information needs to be returned by the receive operation. The information is returned by the status argument of the MPI RECV function. This is an argument of a special MPI-defined type. Status variables need to be explicitly allocated by the user – they are not system objects.

The status variable can be “decoded” to retrieve the count, source and tag fields, using the query functions listed below. The information returned by these query functions is the information last stored in the status variable by a receive function. It is erroneous to call these query functions if the status variable was never set by a receive.

MPI GET SOURCE(status, source)

IN  status  return status of receive operation (Status)
OUT  source  source rank (integer)

int MPI Get source(MPI Status status, int *source)
MPI GET SOURCE(STATUS, SOURCE, IERROR)
    INTEGER STATUS(MPI STATUS SIZE), SOURCE, IERROR

Returns the rank of the message source in the group associated with the communicator that was used to receive (remote group, for intercommunicators).

MPI GET TAG(status, tag)

IN  status  return status of receive operation (Status)
OUT  tag  message tag (integer)

int MPI Get tag(MPI Status status, int *tag)
MPI GET TAG(STATUS, TAG, IERROR)
    INTEGER STATUS(MPI STATUS SIZE), TAG, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Returns the tag of received message.

MPI GET COUNT(status, datatype, count)

IN  status  return status of receive operation (Status)
IN  datatype  datatype of each receive buffer element (handle)
OUT  count  number of received elements (integer)

int MPI Get count(MPI Status status, MPI Datatype datatype, int *count)
MPI GET COUNT(STATUS, DATATYPE, COUNT, IERROR)
    INTEGER STATUS(MPI STATUS SIZE), DATATYPE, COUNT, IERROR

Returns the number of elements received. (Here, again, we count elements, not bytes.) The datatype argument should match the argument provided by the receive call that set the status variable.

Note that it is not mandatory to query the return status after a receive. The receiver will use these calls, or some of them, only when the information they return is needed.

Implementation note:

One expects decode functions to be in-lined in many implementations.

All send and receive operations use the buf, count, datatype, source, dest, tag, comm and status arguments in the same way as the basic MPI SEND and MPI RECV operations described in this section.

3.4 Semantics of point-to-point communication

A valid MPI implementation guarantees certain general properties of point-to-point communication, which are described in this section.

Order Messages are non-overtaking, within each context: if two messages are successively sent from the same source, to the same destination, with the same context, then the messages are received in the order they were sent. I.e., if the receiver posts two successive receives that both match either message, then the first receive will receive the first message, and the second receive will receive the second message. This requirement facilitates matching of sends to receives. It guarantees that message-passing code is deterministic, if processes are single-threaded and wildcard MPI ANY SOURCE is not used in receives.

If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multi-threaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads: the operations are logically concurrent, even though one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. Note that the receiver is not required to receive messages in the order they were sent, if they have distinct envelopes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Progress If a pair of matching send and receives have been initiated on two processes, then at least one of these two operations will complete, independently of other actions in the system: the send operation will complete, unless the receive is satisfied by another message, and completes; the receive operation will complete, unless the message sent is consumed by another matching receive that was posted at the same destination process.

Fairness MPI makes no guarantee of fairness in the handling of communication. Suppose that a send was posted. Then it is possible that the destination process repeatedly posts a receive that matches this send, yet the message is never received, because it is each time overtaken by another message, sent from another source. Similarly, suppose that a receive was posted by a multi-threaded process. Then it is possible that messages that match this receive are repeatedly received, yet the receive is never satisfied, because it is overtaken by other receives posted at this node (by other executing threads). It is the programmer’s responsibility to prevent starvation in such situations.

Resource limitations The current practice for many commercial message-passing libraries is that (short) messages are buffered by the system, thus allowing blocking send operations to complete ahead of the matching receives. It is expected that many MPI implementations will follow this practice, and provide the same level of buffering that is available on the libraries they replace. However, message buffering is not a universal practice. Even on systems where buffering occur, the amount of buffer space available and the way it is allocated is bound to be implementation dependent.

Therefore, message buffering is not mandated by MPI. A valid MPI implementation of MPI SEND is to block the sender until a matching receive has been initiated. In general, the programmer can make no assumptions on the availability of buffer space, and how this space is allocated. Thus, portable (safe) MPI code should work under the assumption that an arbitrary subset of the send operations are going to return before a matching receive is posted, and the rest will block until a matching receive is posted.

An MPI implementation will provide information and control on the amount of available buffer space and on the buffering policy via the environmental enquiries described in Chapter 7.

Examples (involving two processes with ranks 0 and 1) The following program is safe, and should always succeed.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0)
  THEN
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
  ELSE    ! rank.EQ.1
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
END IF

The following program is erroneous, and should always fail.

CALL MPI_COMM_RANK(comm, rank, ierr)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

IF (rank.EQ.0)
  THEN
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
  ELSE    ! rank.EQ.1
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
END IF

The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock.

The following program is unsafe, and may succeed or fail, depending on implementation.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0)
  THEN
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
  ELSE    ! rank.EQ.1
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
END IF

The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered. Thus, this program can succeed only if the MPI implementation buffers messages, and the communication system has sufficient buffer space to buffer count words of data.

3.5 Data Type Matching

One can think of message transmission as consisting of three phases:

1. Data is pulled out of the send buffer and a message is assembled

2. A message is transferred from sender to receiver

3. Data is pulled from the incoming message and disassembled into the receive buffer

Type matching has to be observed at each of these three phases: The type of each variable in the sender buffer has to match the type specified for that entry by the send operation; the type specified by the send operation has to match the type specified by the receive operation; and the type of each variable in the receive buffer has to match the type specified for that entry by the receive operation. A program that fails to observe these three rules is erroneous.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

To define type matching more precisely, we need to deal with two issues: matching of types of the host language with types specified in communication operations; and matching of types at sender and receiver.

A type specified for an entry by a send operation matches the type specified for that entry by a receive operation if both operations use identical names: MPI INTEGER matches MPI INTEGER, MPI REAL matches MPI REAL, and so on.

The type of a variable in a host program matches the type specified in the communication operation if the datatype name used by that operation corresponds to the basic type of the host program variable: an entry with type name MPI INTEGER matches a Fortran variable of type INTEGER, an entry with type name MPI REAL matches a Fortran variable of type REAL, and so on. There is one exception to this last rule: An entry with type name MPI BYTE can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte. The value of the message entry will be the binary value of the corresponding byte in memory.

We thus have two cases:

• Communication of typed values (e.g., with datatype different from MPI BYTE), where the datatypes of the corresponding entries in the sender program, in the send call, in the receive call and in the receiver program should all match.

• Communication of untyped values (e.g., of datatype MPI BYTE), where both sender and receiver use the datatype MPI BYTE. In this case, there are no requirements on the types of the corresponding entries in the sender and the receiver programs, nor is it required that they be the same.

The following examples illustrate these two cases.

First program:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0)
  THEN
    CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
  ELSE
    CALL MPI_RECV(a(1), 15, MPI_REAL, 0, tag, comm, status, ierr)
END IF

This code is correct if both sender and receiver programs have allocated consecutive storage for ten real numbers, starting from a(1).

Second program:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0)
  THEN
    CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
  ELSE
    CALL MPI_RECV(a(1), 40, MPI_BYTE, 0, tag, comm, status, ierr)
END IF

This code is erroneous, since sender and receiver do not provide matching datatype arguments.

Third program:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0)
  THEN
    CALL MPI_SEND(a(1), 40, MPI_BYTE, 1, tag, comm, ierr)
  ELSE
    CALL MPI_RECV(a(1), 60, MPI_BYTE, 0, tag, comm, status, ierr)
END IF

This code is correct, irrespective of the type of a(1) and following variables in store. Some implementations may disallow the transfer of Fortran character strings using the

MPI type MPI BYTE.

3.6 Data conversion

One of the goals of MPI is to support parallel computations across heterogeneous environments. Communication in a heterogeneous environment may require data conversions.

We use the following terminology:

type conversion changes the datatype of a value, e.g. by rounding a REAL to an INTE-GER.

representation conversion changes the binary representation of a value, e.g. from Hex floating point to IEEE floating point.

The type matching rules imply that MPI communication never entails type conversion. On the other hand, MPI requires that a representation conversion be performed when a typed value is transferred across environments that use different representations for the datatype of this value. MPI does not specify rules for representation conversion. Such conversion is expected to preserve integer, logical or character values, and to convert a floating point value to the nearest value that can represented on the target system.

Overflow and underflow exceptions may occur during floating point conversions; conversion of integers or characters may also lead to exceptions when a value that can represented in one system cannot be represented in the other system. An exception occurring during representation conversion results in a failure of the communication; an error occurs either in the send operation, or the receive operation, or both.

If a value sent in a message is untyped (i.e., of type MPI BYTE), then the binary representation of the byte stored at the receiver is identical to the binary representation of the byte loaded at the sender. This holds true, whether sender and receiver run in the same or in distinct environments; no representation conversion is required.

Note that no conversion ever occurs when an MPI program executes in a homogeneous system, where all processes run in the same environment. Also note the different behavior of MPI BYTE and of MPI CHAR. A buffer descriptor entry with datatype of MPI CHAR can only match a C variable of type CHAR; and representation conversion may occur when values of type MPI CHAR are transferred., e.g., from an EBCDIC encoding to an ASCII encoding.

Consider the previous three examples.

The first program is correct, assuming both sender and receiver declared ten consecutive real variables in storage starting at a(1). If the sender and receiver execute in different environments, then the ten real values that are fetched from the send buffer will be converted to the representation for reals on the receiver site before they are stored in the receive buffer.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

While the number of real elements fetched from the send buffer equal the number of real elements stored in the receive buffer, the number of bytes stored need not equal the number of bytes loaded: e.g. the sender may use a four byte representation and the receiver an eight byte representation for reals. If the send or receive buffer do not contain ten consecutive real variables, then the program is erroneous, and its behavior is undefined.

The second program is erroneous, and its behavior is undefined.

The third program is correct. The exact same sequence of forty bytes that were loaded from the send buffer will be stored in the receive buffer, even if sender and receiver run in a different environment. The message sent has exactly the same length (in bytes) and the same binary representation as the message received. If the variables in the send buffer are of different types from the variables in the receive buffer, or they are of the same type but different data representations are used, then the bits stored in the receive buffer may encode values that are different from the values they encoded in the send buffer.

Data representation conversion also applies to the envelope of a message: source, destination and tag are all integers that may need to be converted.

The current draft does not provide a mechanism for inter-language communication: messages sent by Fortran calls should be received by Fortran calls and messages sent by C calls should be received by C calls (this follows from the requirements of type matching and the fact that Fortran and C datatypes are distinct). If inter-language communication is needed then one needs to invoke C communication routines from a Fortran program or Fortran communication routines from a C program.

Implementation note:

The current definition does not require messages to carry data type information. A message can be composed and sent using only the information provided by the send call, and can be received and stored using only the information provided by the receive call. If messages are sent between different machines then one can either use a “universal” data encoding for messages, use knowledge of the receiver environment in order to convert data at the sender, or use knowledge of the sender environment in order to convert data at the receiver. In either case the datatype argument in the local call can be used to derive the types of the values transferred.

Additional type information can be used to provide better error detection.

3.7 Communication Modes

The basic send operation described in section 3.2 used the standard communication mode. In such a communication mode, a send operation can be started whether or not a matching receive was posted. The completion of the send operation indicates that the message and its envelope have been safely stored away and that the sender is free to access and modify the sender buffer. Thus, the operation is locally-blocking: it does not return until the send operation has completed locally, on the sender side.

The completion of a send operation gives no indication that the message was received on the receiver side. A blocking send may be implemented so that it returns only after a matching receive has been executed on the receiver side. This avoids the need to buffer message data out of sender or receiver memory. In this case the send operation completes only after the matching receive has started executing. On the other hand, it is also possible for MPI to buffer messages, so as to allow the sender to proceed ahead of the receiver. In

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

such a case the send operation may complete successfully before the message was received. There are two additional communication modes: A send that uses the ready communication mode may be started only if the matching

receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this allows the removal of a hand-shake operation that is otherwise required, and results in improved performance. The completion of the send operation does not depend on the status of a matching receive, and merely indicates that the send buffer can be reused. A send operation that uses the ready mode has the same semantics as a standard send operation; it is merely that it can be used only in some of the situations where a regular sent can be used.

A send that uses the synchronous mode can be started whether or not a matching receive was posted. However, the send will complete successfully only if a matching receive is posted, and the receive operation has started to receive the message sent by the synchronous send. (I.e., the receive has been posted, and the incoming message has been matched to this posted receive.) Thus, the completion of a synchronous send not only indicates that the send buffer can be reused, but also indicates that the receiver has reached a certain point in its execution, namely that it has started executing the matching receive. If both sends and receives are blocking operations then the use of the synchronous mode provides synchronous communication semantics: a communication does not complete at either end before both processes “attend” to the communication; the completion of a synchronous send is a global event, and the operation is globally-blocking.

Two additional send functions are provided for the two additional communication modes. The communication mode is indicated by a one letter prefix: R for ready and S for synchronous.

Send in ready mode

MPI RSEND (buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Rsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
MPI RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in synchronous mode

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI SSEND (buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Ssend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
MPI SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

There is only one receive mode, which can match any of the send modes. The receive operation described in the last section is blocking: it returns only after the receive buffer contains the newly received message. It is asynchronous: the completion of a receive is a local operation, and a receive can complete before the matching send has completed (of course, it can complete only after the matching send has started).

Communication imposes an order on the events occurring at the communicating nodes. It is always the case that the completion of a receive occurs after the start of the matching send. If the synchronous send mode is used, then it is also the case that the completion of a send occurs after the start of the matching receive. Of course, on each process, a communication completes after it is started. No other order is imposed by MPI. E.g., if the standard send mode is used, the send operation may complete before the matching receive was started.

In a multi-threaded implementation of MPI, the system may deschedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it the user responsibility not to access or modify a communication buffer until the communication completes; otherwise the outcome of the computation is undefined.

Implementation note: A ready send can be implemented as a standard send; in such a case there will be no performance advantage (or disadvantage) for the use of ready send.

A standard send can be implemented as a synchronous send. In such a case, no data buffering is needed.

A possible communication protocol for the various communication modes is outlined below: ready send: The message is sent as soon as possible. synchronous send: The sender sends a request-to-send message. The receiver stores this

request. When a matching receive is posted, the receiver sends back a permission-to-send message, and the sender now sends the message.

standard send: First protocol may be used for short messages, and second protocol for long messages.

Additional control messages might be needed for flow control and error recovery. Of course, there are many other possible choices.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

3.8 Nonblocking communication

One can improve performance on many systems by overlapping communication and computation. This is especially true on systems where communication can be executed autonomously by an intelligent communication controller. Light-weight threads are one mechanism for achieving such overlap. An alternative mechanism that often leads to better performance is to use nonblocking communication. A nonblocking send call initiates the send operation, but does not complete it. The send call will return before the message was copied out of the send buffer. A separate call is needed to complete the communication, i.e. to verify that the data has been copied out of the send buffer. With suitable hardware, the transfer of data out of the sender memory may proceed concurrently with computations done at the sender after the send was initiated and before it completed. Similarly, a nonblocking receive initiates the receive operation, but does not complete it. The call will return before a message is stored into the receive buffer. A separate call is needed to complete the receive operation and verify that the data has been received into the receive buffer. With suitable hardware, the transfer of data into the receiver memory may proceed concurrently with computations done after the receive was initiated and before it completed. The use of nonblocking receives may also avoid system buffering and memory to memory copying, as information is provided early on the location of the receive buffer.

Nonblocking sends can use the same three modes as blocking sends: standard, ready and synchronous. These carry the same meaning. The initiation and completion of a standard send do not depend on the status of a matching receive. A ready send can be initiated only if a matching receive has already been initiated, otherwise the call is erroneous; its completion does not depend on the status of a matching receive. A synchronous send can be initiated before a matching receive has been initiated, but will complete only after a matching receive has been initiated, and has started receiving the message generated by the send operation.

In all cases the operation that initiates the communication (send or receive) is a local, nonblocking call. Such a call may fail because the system has exhausted available resources (e.g., exceeded the limit on number of pending communications per node). Good quality implementations of MPI will set these limits high enough so that a nonblocking communication initiation call will fail only in “pathological” cases.

Nonblocking sends can be matched with blocking receives, and vice-versa.

3.8.1 Communication Objects

Nonblocking communications use opaque communication request objects to identify communication operations and match the operation that initiates the communication with the operation that terminates it. These are system objects that are accessed via a handle. A request object identifies various properties of a communication operation, such as the (send or receive) buffer that is associated with it, its context, the tag and destination arguments to be used for a send, or the tag and source arguments to be used for a receive. In addition, this object stores information about the status of the pending communication operation that is performed with this object.

3.8.2 Communication initiation

We use the same naming conventions as for blocking communication: a prefix of R (S) is used for READY (SYNCHRONOUS) mode. In addition a prefix of I (for IMMEDIATE) indicates

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

that the call is nonblocking.

Initiate a standard mode nonblocking send.

MPI ISEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Isend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
MPI ISEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Initiate a ready mode nonblocking send.

MPI IRSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Irsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
MPI IRSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Initiate a synchronous mode nonblocking send.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI ISSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Issend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
MPI ISSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Initiate a nonblocking receive.

MPI IRECV (buf, count, datatype, source, tag, comm, request)

OUT  buf  initial address of receive buffer (choice)
IN  count  number of elements in receive buffer (integer)
IN  datatype  datatype of each receive buffer element (handle)
IN  source  rank of source (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Irecv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Comm request request)
MPI IRECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR

These calls allocate a communication request and associate it with the handle. The request can be used later to query the status of the communication or wait for its completion.

A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not access any part of the send buffer after a nonblocking send operation returns, until the send completes.

A nonblocking receive call indicates that the sytem may start writing data into the receive buffer. The receiver should not access any part of the receive buffer after a nonblocking receive operation returns, until the receive completes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

3.8.3 Communication Completion

The functions MPI WAIT and MPI TEST are used to complete a nonblocking communication. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer (the send operation itself leaves the content of the send buffer unchanged). It does not indicate that the message has been received; rather, it may have been buffered by the communication subsystem. However, if a synchronous mode send was used, the completion of the send operation indicates that a matching receive was initiated, and that the message will eventually be received by this matching receive.

The completion of a receive operation indicates that the receive buffer contains the received message, and that the status object is set; the receiver is now free to access the receive buffer. It does not indicate that the matching send operation has completed (but indicates, of course, that the send was initiated).

MPI WAIT(request, status)

INOUT request communication request (handle)

OUT  status  status object (Status)

int MPI Wait(MPI Comm request *request, MPI Status *status)
MPI WAIT(REQUEST, STATUS, IERROR)
    INTEGER REQUEST, STATUS(MPI STATUS SIZE), IERROR

A call to MPI WAIT returns when the send operation identified by request is complete. If the communication object associated with this request was created by a nonblocking send or receive call, then the object is deallocated by the call to MPI WAIT and the request becomes null.

The call returns in status information on the completed operation. The status object for a receive operation can be queried using the functions described in section 3.3.1. The status argument is not used or updated for a send operation, and an arbitrary value may be passed for this argument in the call.

MPI TEST(request, flag, status)

INOUT request communication request (handle)

OUT  flag  (logical)
OUT  status  status object (Status)

int MPI Test(MPI Comm request *request, int flag, MPI Status *status)
MPI TEST(REQUEST, FLAG, STATUS, IERROR)
    LOGICAL FLAG
    INTEGER REQUEST, STATUS(MPI STATUS SIZE), IERROR

A call to MPI TEST returns flag = true if the operation identified by request is complete. In such a case, the status object is set to contain information on the completed operation; if the communication object was created by a nonblocking send or receive, then it is deallocated and the request becomes null. The call returns flag=false, otherwise. In such a case, the value of the status object is undefined.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI TEST is a local, nonblocking operation.

The return status object for a receive operation carries information that can be accessed using the functions described in section 3.3.1. The status argument is not used or updated for a send operation and an arbitrary value can be passed for this argument in the call.

In a multi-threaded environment, the use of a blocking receive operation ( MPI WAIT) may allow the operating system to de-schedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation (MPI TEST) allows the user to schedule alternative activities within a single thread of execution.

Example:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0)
  THEN
    CALL MPI_ISEND(request, a(1), 10, MPI_REAL, 1, tag, comm, ierr)
    **** do some computation to mask latency ****
    CALL MPI_WAIT(request, status, ierr)
  ELSE
    CALL MPI_IRECV(request, a(1), 15, MPI_REAL, 0, tag, comm, ierr)
    **** do some computation to mask latency ****
    CALL MPI_WAIT(request, status, ierr)
END IF

The functions MPI WAIT and MPI TEST can be used to complete both sends and receives; they are also used to complete any other nonblocking communication call provided by MPI.

3.8.4 Semantics of Nonblocking Communications

The semantics of nonblocking communication is defined by suitably extending the definitions in section 3.4 for order and progress.

Nonblocking communication operations are ordered according to the execution order of the calls that initiate the communication; this defines the meaning of non-overtaking communication.

Example:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (RANK.EQ.0)
   THEN
      CALL MPI_ISEND(h1, a, 1, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_ISEND(h2, b, 1, MPI_REAL, 1, 0, comm, ierr)
   ELSE    ! rank.EQ.1
      CALL MPI_IRECV( h1, a, 1, MPI_REAL, 0, MPI_ANY_TAG, comm, ierr)
      CALL MPI_IRECV( h2, b, 1, MPI_REAL, 0, 0, comm, ierr)
END IF
CALL MPI_WAIT(h1,status)
CALL MPI_WAIT(h2,status)

The first send of process zero will match the first receive of process one, even if both messages are sent before process one executes either receive. This precludes an implementation where different queues are used to post the two receives.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

A call to MPI WAIT that completes a receive will eventually terminate and return if a matching send has been started, unless the send is satisfied by another receive. In particular, if the matching send is nonblocking, then the receive should complete even if no call is executed by the sender to complete the send. Similarly, a call to MPI WAIT that completes a send will eventually return if a matching receive has been started, unless the receive is satisfied by another send, and even if no call is executed to complete the receive.

Example:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (RANK.EQ.0)
   THEN
      CALL MPI_ISEND( h, a, 1, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_RECV( b, 1, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_WAIT( h, status)
   ELSE    ! rank.EQ.1
      CALL MPI_RECV( a, 1, MPI_REAL, 0, 0, comm, ierr)
      CALL MPI_SEND( b, 1, MPI_REAL, 0, 0, comm, ierr)
END IF

This code should not deadlock in a correct MPI implementation.

If MPI TEST that completes a receive is repeatedly called with the same arguments, and a matching send has been started, then the call will eventually return flag=true, unless the send is satisfied by another receive. If MPI TEST that completes a send is repeatedly called with the same arguments, and a matching receive has been started, then the call will eventually return flag=true, unless the receive is satisfied by another send.

Nonblocking communications cannot be replaced by blocking communications, even in the synchronous communication mode. Consider the following example:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0)
  THEN
    CALL MPI_ISEND(request1, a(1), count, MPI_REAL, 1, tag1, comm, ierr)
    CALL MPI_ISEND(request2, a(2), count, MPI_REAL, 1, tag2, comm, ierr)
    CALL MPI_WAIT(request1, status, ierr)
    CALL MPI_WAIT(request2, status, ierr)
  ELSE
    CALL MPI_IRECV(request2, a(2), count, MPI_REAL, 0, tag2, comm, ierr)
    CALL MPI_IRECV(request1, a(1), count, MPI_REAL, 0, tag1, comm, ierr)
    CALL MPI_WAIT(request2, status, ierr)
    CALL MPI_WAIT(request1, status, ierr)
END IF

The code is guaranteed to execute correctly, even in an implementation that does not buffer messages (the only resource requirement is the ability to have two pending communications). If one replaces the nonblocking sends (receives) by blocking sends (receives), and deletes the calls to MPI WAIT, then the code will deadlock. The code will also deadlock if one deletes the calls that initiate communication and replaces each call to MPI WAIT by the appropriate blocking send or receive.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

3.8.5 Multiple Completions

It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for a specific message. A call to MPI WAITANY or MPI TESTANY can be used to wait for the completion of one out of several operations; a call to MPI WAITALL or MPI TESTALL can be used to wait for all pending operations in a list.

MPI WAITANY (count, array of requests, index, status)

IN  count  list length (integer)

INOUT array of requests array of requests (array of handle)

OUT  index  index of handle for operation that completed (integer)
OUT  status  status object (Status)

int MPI Waitany(int count, MPI Comm request *array of requests, int *index,
             MPI Status *status)
MPI WAITANY(COUNT, ARRAY OF REQUESTS, INDEX, STATUS, IERROR)
    INTEGER COUNT, ARRAY OF REQUESTS(*), INDEX, STATUS(MPI STATUS SIZE),
    IERROR

Blocks until one of the operations associated with the communication requests in the array has completed. Returns in index the index of that request in the array and returns in status the status of the completing communication.

The execution of MPI WAITANY(count, array of requests, index, status) has the same effect as the execution of MPI WAIT(array of requests[i], status), where i is the value returned by index. In particular, the associated communication object is deallocated, and the handle to it in array of requests is set to null. If more then one operation is enabled and can terminate, one is arbitrarily chosen. There is no requirement that the choice satisfy any fairness criterion.

MPI TESTANY(count, array of requests, index, flag, status)

IN  count  list length (integer)

INOUT array of requests array of requests (array of handle)

OUT  index  index of operation that completed, or MPI UNDEFINED

if none completed (integer)

OUT  flag  TRUE if one of the operations is complete (logical)
OUT  status  status object (Status)

int MPI Testany(int count, MPI Comm request *array of requests, int *index,
             int *flag, MPI Status *status)
MPI TESTANY(COUNT, ARRAY OF REQUESTS, INDEX, FLAG, STATUS, IERROR)
    LOGICAL FLAG
    INTEGER COUNT, ARRAY OF REQUESTS(*), INDEX, STATUS(MPI STATUS SIZE),
    IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Causes either one or none of the operations associated with the communication requests to return. In the former case, it returns in index the index of its request in the array, returns in status the status of that operation, deallocates the associated communication object, and sets the request to null. In the latter case, it returns a value of MPI UNDEFINED in index and status is undefined.

The execution of MPI TESTANY(count, array of requests, index, status) has the same effect as the execution of MPI TEST(array of requests[i], flag, status), for i=0, 1 ,..., count-1, in some arbitrary order, until one call returns flag=true, or all fail. In the former case, index is set to the last value of i; in the later case, it is set to MPI UNDEFINED.

MPI WAITALL( count, array of requests, array of statuses)

IN  count  lists length (integer)

INOUT array of requests array of requests (array of handle)

OUT  array of statuses  array of status objects (Status)

int MPI Waitall(int count, MPI Comm request *array of requests,
             MPI Status *array of statuses)
MPI WAITALL(COUNT, ARRAY OF REQUESTS, ARRAY OF STATUSES, IERROR)
    INTEGER COUNT, ARRAY OF REQUESTS(*),
    ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR

Blocks until all communication operations associated with requests in the list complete, and return the status of all these operations. Both arrays have the same number of valid entries. The i-th entry in array of statuses is set to the return status of the i-th operation. All communication objects are deallocated and all requests are set to null.

The execution of MPI WAITALL(count, array of requests, array of statuses has the same effect as the execution of MPI WAIT(array of request[i], array of statuses[i]), for i=0 ,..., count-1, in some arbitrary order.

MPI TESTALL(count, array of requests, flag, array of statuses)

IN  count  lists length (integer)

INOUT array of requests array of requests (array of handle)

OUT  flag  (logical)
OUT  array of statuses  array of status objects (Status)

int MPI Testall(int count, MPI Comm request *array of requests, int *flag,
             MPI Status *array of statuses)
MPI TESTALL(COUNT, ARRAY OF REQUESTS, FLAG, ARRAY OF STATUSES, IERROR)
    LOGICAL FLAG
    INTEGER COUNT, ARRAY OF REQUESTS(*),
    ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR

Causes either all or none of the operations associated with the communication requests to complete. It returns flag = true if all communications associated with requests in the array have completed. In this case, each status entry is set to the status of the corresponding

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

communication. All communication objects are deallocated, and all requests are set to null. Otherwise, flag = false is returned, no request is modified and the values of the status entries are undefined. This is a nonblocking operation that returns immediately.

Example:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank < 2)
  THEN         ! rank=0 or rank=1
    CALL MPI_ISEND(request, a, n, MPI_REAL, 2, tag, comm, ierr)
    **** do some computation to mask latency ****
    CALL MPI_WAIT(request, status, ierr)
  ELSE         ! rank=2
    CALL MPI_IRECV(request_list(0), a, n, MPI_REAL, 0, tag, comm, ierr)
    CALL MPI_IRECV(request_list(1), b, n, MPI_REAL, 1, tag, comm, ierr)
    **** do some computation to mask latency ****
    CALL MPI_WAITANY(2, request_list, index, status, ierr)
    IF(index.EQ.0)
      THEN
        **** handle message from process 0 ****
        CALL MPI_WAIT(request_list(1), status, ierr)
        **** handle message from process 1 ****
      ELSE
        **** handle message from process 1 ****
        CALL MPI_WAIT(request_list(0), status, ierr)
        **** handle message from process 0 ****
    END IF
END IF

The calls introduced in this subsection can be used to wait or test for the completion of an arbitrary mix of nonblocking communication calls, including a mix of sends and receives and of any additional nonblocking communication calls provided by MPI.

The status argument in MPI WAITANY and MPI WAITALL and the entries of the array of statuses argument in MPI WAITALL and MPI TESTALL are not used or updated if the completing communication is a send. If all operations in the list provided by the argument array of requests are send operations, then an arbitrary value can be passed to the status (resp. array of statuses) argument.

An array of requests argument that is passed to any of the last four functions may contain null entries. These entries are ignored and the call executes as if it was supplied an array of requests containing only the non-null entries. As a request in array of requests is set to null when the corresponding operation completes, this allows to reuse the same array for completing in successive calls the operations associated with this array. Thus, the last example can be rewritten as follows:

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank < 2)
  THEN         ! rank=0 or rank=1
    CALL MPI_ISEND(request, a, n, MPI_REAL, 2, tag, comm, ierr)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

    CALL MPI_WAIT(request, status, ierr)
  ELSE         ! rank=2
    CALL MPI_IRECV(request_list(0), a, n, MPI_REAL, 0, tag, comm, ierr)
    CALL MPI_IRECV(request_list(1), b, n, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_WAITANY(2, request_list, index, status, ierr)
        **** handle first arriving message  ****
    CALL MPI_WAITANY(2, request_list, index, status, ierr)
        **** handle second arriving message  ****

If MPI WAITANY or MPI TESTANY are called with an empty array of requests argument (i.e., an array of length zero, or an array with all entries null) then they return the value MPI UNDEFINED in index; MPI WAITALL with an empty array of requests argument has no effect; MPI TESTALL with an empty array of requests argument returns flag=true.

3.9 Probe and Cancel

The MPI PROBE and MPI IPROBE operations allow incoming messages to be checked for, without actually receiving them. The user can then decide how to receive them, based on the information returned by the probe (basically, the information returned by status). In particular, the user may allocate memory for the receive buffer, according to the length of the probed message.

An MPI implementation of MPI PROBE needs to guarantee progress: if a call to MPI PROBE has been issued by a process, and a send that matches the probe has been initiated by some process, then the call to MPI PROBE will return, unless the message is received by another concurrent receive operation (that is executed by another thread at the probing process). Similarly, if a process busy waits with MPI IPROBE and a matching message has been issued, then the call to MPI IPROBE will eventually return flag = true unless the message is received by another concurrent receive operation.

The MPI CANCEL operation allows pending communications to be cancelled. This is required for cleanup. Posting a send or a receive ties up user resources (send or receive buffers), and a cancel may be needed to free these resources gracefully.

MPI IPROBE(source, tag, comm, flag, status)

IN  source  source rank, or MPI ANY SOURCE (integer)
IN  tag  tag value or MPI ANY TAG (integer)
IN  comm  communicator (handle)
OUT  flag  (logical)
OUT  status  status object (Status)

int MPI Iprobe(int source, int tag, MPI Comm comm, int *flag,
             MPI Status *status)
MPI IPROBE(SOURCE, TAG, COMM, FLAG, STATUS, IERROR)
    LOGICAL FLAG
    INTEGER SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE), IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI IPROBE returns flag = true if there is a message that can be received and that matches the pattern specified by the arguments source, tag, and comm. It returns flag = false, otherwise. If MPI IPROBE returns flag = true, then the status object can be subsequently queried using the functions described in section 3.3.1 to find the source, tag and length of the message. The values returned by these queries are the same values that would have been returned by a call to MPI RECV executed at the same point in the program. The return status object is undefined if flag = false.

A subsequent receive executed with the same context, and the source and tag returned by querying the status returned by MPI IPROBE will receive the message that was matched by the probe, if no other intervening receive occurred after the probe. If the receiving process is multi-threaded, it is the user’s responsibility to ensure that the last condition holds.

MPI PROBE(source, tag, comm, status)

IN  source  source rank, or MPI ANY SOURCE (integer)
IN  tag  tag value, or MPI ANY TAG (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Probe(int source, int tag, MPI Comm comm, MPI Status *status)
MPI PROBE(SOURCE, TAG, COMM, STATUS, IERROR)
    INTEGER SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE), IERROR

MPI PROBE behaves like MPI IPROBE except that it is a blocking call which returns only after a matching message has been found.

MPI CANCEL(request)

IN  request  communication request (handle)

int MPI Cancel(MPI Comm request request)
MPI CANCEL(REQUEST, IERROR)
    INTEGER REQUEST, IERROR

A call to MPI CANCEL marks for cancellation a pending nonblocking communication operation (send or receive). The cancel call is nonblocking, and local. It returns immediately, possibly before the communication is actually cancelled.

It is still necessary to complete a communication that has been marked for cancellation, using a call to MPI WAIT or MPI TEST (or any of the derived operations). If the operation has been cancelled, then information to that effect will be returned in the status argument of the operation that completes the communication. If a communication is marked for cancellation, then a MPI WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI WAIT behaves as a local function); similarly if MPI TEST is repeatedly called in a busy wait loop for a cancelled communication, then MPI TEST will eventually be successful.

Either the cancellation succeeds, or the communication succeeds, but not both. If a send is marked for cancellation, then it must be the case that either the send completes

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

normally, in which case the message sent was received at the destination process, or that the send is successfully cancelled, in which case no part of the message was received at the destination. Then, any matching receive has to be satisfied by another send. If a receive is marked for cancellation, then it must be the case that either the receive completes normally, or that the receive is successfully cancelled, in which case no part of the receive buffer is altered. Then, any matching send has to be satisfied by another receive.

MPI TEST CANCELLED(status, flag)

IN  status  status object (Status)
IN  flag  (logical)

int MPI Test cancelled(MPI Status status, int *flag)
MPI TEST CANCELLED(STATUS, FLAG, IERROR)
    LOGICAL FLAG
    INTEGER STATUS(MPI STATUS SIZE), IERROR

Returns flag = true if the communication associated with the status object was cancelled successfully. In such a case, all other fields of status (such as count or tag) are undefined. Returns flag = false, otherwise. If a receive operation might be cancelled then one should call MPI TEST CANCELLED first, to check whether the operation was cancelled, before checking on the other fields of the return status.

Implementation note: If a send operation uses an “eager” protocol (data is transferred to the receiver before a matching receive is posted), then the cancelation of this send may require communication with the intended receiver, in order to free allocated buffers. On some systems this may require an interrupt to the intended receiver. Therefore, it is expected that MPI CANCEL will be an expensive operation on many systems, to be used only exceptionally. Note that, while communication may be needed to implement MPI CANCEL, this is still a local operation, in the sense that its completion does not depend on the code executed by other processes. If processing is required on another process, this should be transparent to the application (hence the need for an interupt and an interupt handler).

3.10 Persistent communication objects

Often a communication with the same argument list is repeatedly executed within the inner loop of a parallel computation. In such a situation, it may be possible to optimize the communication by binding the list of communication arguments to a communication request once and, then, repeatedly using the communication request to initiate and complete messages. The communication request thus created can be thought of as a communication port or a “half-channel” . It does not provide the full functionality of a conventional channel, since there is no binding of the send port to the receive port: this construct allows reduction of the overhead for communication between processor and communication controller, but not the overhead for communication between one communication controller and another.

A communication request is created using one of the four following calls. These calls involve no communication.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The function MPI CREATE SEND creates a communication request for a standard mode send operation, and binds to it all the arguments of a send operation.

MPI CREATE SEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Create send(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
MPI CREATE SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER REQUEST, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

The function MPI CREATE RSEND creates a communication object for a ready mode send operation.

MPI CREATE RSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Create rsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
MPI CREATE RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

The function MPI CREATE SSEND creates a communication object for a synchronous mode send operation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI CREATE SSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Create ssend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
MPI CREATE SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

The function MPI CREATE RECV creates a communication object for a receive operation.

MPI CREATE RECV(buf, count, datatype, source, tag, comm, request)

OUT  buf  initial address of receive buffer (choice)
IN  count  number of elements received (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of source or MPI ANY SOURCE (integer)
IN  tag  message tag or MPI ANY TAG (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Create recv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Comm request request)
MPI CREATE RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR

A communication (send or receive) that uses a predefined request is initiated by the function MPI START.

MPI START(request)

INOUT request communication request (handle)

int MPI Start(MPI Comm request request)
MPI START(REQUEST, IERROR)
    INTEGER REQUEST, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The communication object associated with request should be an object that was created by one of the previous four functions, so that all the communication arguments are already defined. A send can be started provided that the previous send using the same object has completed, or as soon as the object is created, if it has not yet been used in a communication. In addition, if the communication mode is ready then a matching receive should be posted. The send buffer should not be accessed after the send is started, and until the operation completes.

A receive can be started provided that the preceding receive using the same object has completed, or as soon as the object is created, if it has not yet been used in a communication. The receive buffer should not be accessed after the receive is started, until the operation completes.

The call is nonblocking, with similar semantics to the nonblocking communication operations described in section 3.8.

MPI STARTALL(count, array of requests)

IN  count  list length (integer)

INOUT array of requests array of requests (array of handle)

int MPI Startall(int count, MPI Comm request *array of requests)
MPI STARTALL(COUNT, ARRAY OF REQUESTS, IERROR)
    INTEGER COUNT, ARRAY OF REQUESTS(*), IERROR

Start all communications associated with requests in array of requests. A call to MPI STARTALL(count, array of requests) has the same effect as calls to MPI START(array of requests[i]), executed for i=0 ,..., count-1, in some arbitrary order.

A communication started with a call to MPI START or MPI STARTALL is completed by a call to MPI WAIT, MPI TEST, or one of the derived functions described in section 3.8.5. These communication completion functions do not deallocate the request, and this can be reused anew by a MPI START call. The object needs to be explicitly deallocated by a call to the function MPI REQUEST FREE, below. We thus have two types of communication objects: persistent objects, which are allocated by a call to MPI CREATE xxx, and are explicitly deallocated by MPI REQUEST FREE, and ephemeral objects that persist for one communication only: they are created by a nonblocking communication initiation function, and are freed by the communication completion call.

MPI REQUEST FREE(request)

INOUT request communication request (handle)

int MPI Request free(MPI Comm request request)
MPI REQUEST FREE(REQUEST, IERROR)
    INTEGER REQUEST, IERROR

deallocated when there are no pending communications involving this object, at which point the request becomes null.

The call is nonblocking. It is permissible to call MPI REQUEST FREE(request) after

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

a communication that uses request has been initiated, but before it has completed. The object will be deallocated after the communication completes. It is erroneous to initiate a communication with a request that has been freed.

A correct invocation of the functions described in this section will occur in a sequence of the form

Create (Start Complete)^{∗} Free ,

where ∗ indicates zero or more repetitions. If the same communication object is used in several concurrent threads, it is the user responsibility to coordinate calls so that the correct sequence is obeyed.

A send operation initiated with MPI START can be matched with any receive operation and, likewise, a receive operation initiated with MPI START can receive messages generated by any send operation.

3.11 Send-receive

The send-receive operations combine in one call the sending of a message to one destination and the receiving of another message, from another destination, possibly the same. A send-receive operation is very useful for executing a shift operation across a chain of processes. If blocking sends and receives are used for such shift, then one needs to order the sends and receives correctly (e.g., even processes send, next receive, odd processes receive first, next send) so as to prevent cyclic dependencies that lead to deadlock. When a send-receive operation is used, the communication subsystem takes care of these issues. The send-receive operation can be used in conjunction with the functions described in Chapter 6 in order to perform shifts on various logical topologies. Also, a send-receive operation is useful for implementing remote procedure calls.

A message sent by a send-receive operation can be received by a regular receive operation, and vice versa.

MPI SENDRECV(sendbuf, sendcount, sendtype, dest, sendtag, recvbuf, recvcount, recvtype, source, recvtag, comm, status)

IN  sendbuf  initial address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  type of elements in send buffer (handle)
IN  dest  rank of destination (integer)
IN  sendtag  send tag (integer)
OUT  recvbuf  initial address of receive buffer (choice)
IN  recvcount  number of elements in receive buffer (integer)
IN  recvtype  type of elements in receive buffer (handle)
IN  source  rank of source (integer)
IN  recvtag  receive tag (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Sendrecv(void *sendbuf, int sendcount, MPI Datatype sendtype,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

             int dest, int sendtag, void *recvbuf, int recvcount,
             MPI Datatype recvtype, int source, MPI Datatype recvtag,
             MPI Comm comm, MPI Status *status)
MPI SENDRECV(SENDBUF, SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVBUF,
             RECVCOUNT, RECVTYPE, SOURCE, RECVTAG, COMM, STATUS, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVCOUNT, RECVTYPE,
    SOURCE, RECVTAG, COMM, STATUS(MPI STATUS SIZE), IERROR

Execute a blocking send and receive operation. Both send and receive use the same communicator, but possibly different tags. The send buffer and receive buffers are disjoint, and may have different lengths and datatypes.

MPI SENDRECV REPLACE(buf, count, datatype, dest, sendtag, source, recvtag, comm, status)

INOUT buf initial address of send and receive buffer (choice)

IN  count  number of elements in send and receive buffer (integer)
IN  datatype  type of elements in send and receive buffer (handle)
IN  dest  rank of destination (integer)
IN  sendtag  send message tag (integer)
IN  source  rank of source (integer)
IN  recvtag  receive message tag (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Sendrecv replace(void* buf, int count, MPI Datatype datatype,
             int dest, int sendtag, int source, int recvtag, MPI Comm comm,
             MPI Status *status)
MPI SENDRECV REPLACE(BUF, COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG,
             COMM, STATUS, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG, COMM,
    STATUS(MPI STATUS SIZE), IERROR

Execute a blocking send and receive; the same buffer is used both for the send and for the receive, so that the message sent is replaced by the message received.

The semantics of a send-receive operation is what would obtain if the caller forked two concurrent threads, one to execute the send, and one to execute the receive, followed by a join of these two threads. Additional intermediate buffering is needed for the “replace” variant.

A send-receive cannot be implemented by a blocking send followed by a blocking receive or a blocking receive, followed by a blocking send. Consider the following code, where two processes exchange messages:

CALL MPI_COMM_RANK(comm, rank, ierr)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

IF (rank.EQ.0)
  THEN
    CALL MPI_SENDRECV(send_buff, send_count, MPI_REAL, 0, send_tag,
       recv_buff, recv_count, MPI_REAL, 1, recv_tag, comm, status, ierr)
  ELSE
    CALL MPI_SENDRECV(send_buff, send_count, MPI_REAL, 1, send_tag,
       recv_buff, recv_count, MPI_REAL, 0, tag, comm, status, ierr)
END IF

If the send receives are replaced either by blocking send, followed by blocking receive, or blocking receive, followed by blocking send, then the code may deadlock. On the other hand, send-receive can be implemented using nonblocking sends and receives.

3.12 Null processes

In many instances, it is convenient to specify a “dummy” source or destination for communication. This simplifies the code that is needed for dealing with boundaries, e.g., in the case of a non-circular shift done with calls to send-receive.

The special value MPI PROCNULL can be used instead of a rank wherever a source or a destination argument is required in a call. A communication with process MPI PROCNULL has no effect: a send to MPI PROCNULL succeeds and returns as soon as possible. A receive from MPI PROCNULL succeeds and returns as soon as possible with no modifications to the receive buffer. When a receive with source = MPI PROCNULL is executed then the status object returns source = MPI PROCNULL, tag = MPI ANY TAG and count=0.

3.13 Derived datatypes

Up to now, all point to point communication involved only contiguous buffers containing a sequence of elements of the same type. This is too constraining on two accounts: One often wants to pass messages that contain values with different datatypes (e.g., an integer count, followed by a sequence of real numbers); and one often wants to send noncontiguous data (e.g. a sub-block of a matrix). One solution is to provide functions that pack noncontiguous data into a contiguous buffer at the sender site and unpack it back at the receiver site. This has the disadvantage of requiring additional memory-to-memory copy operations at both sites, even when the communication subsystem has scatter-gather capabilities. Instead, MPI provides mechanisms to specify more general, mixed and noncontiguous communication buffers. It is up to the implementation to decide whether data should be first packed in a contiguous buffer before being transmitted, or whether it can be collected directly from where it resides.

The general mechanisms provided here allow one to transfer directly, without copying, objects of various shape and size. It is not assumed that the MPI library is cognizant of the objects declared in the host language; thus, if one wants to transfer a structure, or an array section, it will be necessary to provide in MPI a definition of a communication buffer that mimics the definition of the structure or array section in question. These facilities can be used by library designers to define communication functions that can transfer objects defined in the host language – by decoding their definitions as available in a symbol table or a dope vector. Such higher-level communication functions are not part of MPI.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

More general communication buffers are specified by replacing the basic datatypes that have been used so far with derived datatypes that are constructed from basic datatypes using the constructors described in this section. These methods of constructing derived datatypes can be applied recursively.

A general datatype is an opaque object that specifies two things:

• A sequence of basic datatypes

• A sequence of integer (byte) displacements

The displacements are not required to be positive, distinct, or in increasing order; therefore the order of items need not coincide with their order in store, and an item may appear more than once. We call such pair of sequences (or sequence of pairs) a type map. The sequence of basic datatypes (displacements ignored) is the type signature of the datatype.

Let

T ypemap = {(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

be such a type map, where type_{i} are basic types, and disp_{i} are displacements. Let

T ypesig = {type_{0}, ..., type_{n−1}}

be the associated type signature. This type map, together with a base address buf, specifies a communication buffer: the communication buffer that consists of n entries, where the i-th entry is at address buf + disp_{i} and has type type_{i}. A message assembled from such communication buffer will consist of n values, of the types defined by T ypesig.

We can use a handle to a general datatype as an argument in a send or receive operation, in replacement of a basic datatype argument. The operation MPI SEND(buf, 1, datatype,...) will use the send buffer defined by the base address buf and the general datatype associated with datatype; it will generate a message with the type signature determined by the datatype argument. MPI RECV(buf, 1, datatype,...) will use the receive buffer defined by the base address buf and the general datatype associated with datatype.

General datatypes can be used in all send and receive operations. We address later in section 3.13.5 the case where the second argument count has value > 1.

The basic datatypes presented in section 3.2.1 are particular cases of a general datatype, and are predefined. Thus, MPI INT is a predefined handle to a datatype with type map {(int, 0)}, with one entry of type int and displacement zero. And similarly for all other basic datatypes.

The extent of a datatype is defined to be the span from the first byte to the last byte occupied by entries in this datatype, rounded up to satisfy alignment requirements. That is, if

T ypemap = {(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

then

lb(T ypemap) = min_{j} disp_{j},
ub(T ypemap) = max_{j}(disp_{j} + sizeof(type_{j})), and
extent(T ypemap) = ub(T ypemap) − lb(T ypemap) + ϵ.  (3.1)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

If type_{i} requires alignment to a byte address that is is a multiple of k_{i}, then ϵ is the least nonnegative increment needed to round extent(T ypemap) to the next multiple of max_{i} k_{i}.

Example: Assume that T ype = {(double, 0), (char, 8)} (a double at displacement zero, followed by a char at displacement eight). Assume, furthermore, that doubles have to be strictly aligned at addresses that are multiple of eight. Then, the extent of this datatype is 16 (9 rounded to the next multiple of 8). A datatype that consists of a character immediately followed by a double will also have an extent of 16.

3.13.1 Datatype constructors

Contiguous The simplest datatype constructor is MPI TYPE CONTIGUOUS which allows replication of a datatype into contiguous locations.

MPI TYPE CONTIGUOUS(count, oldtype, newtype)

IN  count  replication count (nonnegative integer)
IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type contiguous(int count, MPI Datatype oldtype,
             MPI Datatype *newtype)
MPI TYPE CONTIGUOUS(COUNT, OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, OLDTYPE, NEWTYPE, IERROR

newtype is the datatype obtained by concatenating count copies of oldtype. Concatenation is defined using extent as the size of the concatenated copies.

Example: let oldtype have type map {(double, 0), (char, 8)}, with extent 16, and let count = 3. The type map of the datatype returned by newtype is

{(double, 0), (char, 8), (double, 16), (char, 24), (double, 32), (char, 40)};

i.e., alternating double and char elements, with displacements 0, 8, 16, 24, 32, 40. In general, assume that the type map of oldtype is

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent. Then newtype has a type map with count · n entries defined by:

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1}), (type_{0}, disp_{0} + extent), ..., (type_{n−1}, disp_{n−1} + extent),

..., (type_{0}, disp_{0} + extent · (count − 1)), ..., (type_{n−1}, disp_{n−1} + extent · (count − 1))}.

Vector The function MPI TYPE VECTOR is a more general constructor that allows replication of a datatype into locations that consist of equally spaced blocks. Each block is obtained by concatenating the same number of copies of the old datatype. The spacing between blocks is a multiple of the extent of the old datatype.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI TYPE VECTOR( count, blocklength, stride, oldtype, newtype)

IN  count  number of blocks (nonnegative integer)
IN  blocklength  number of elements in each block (nonnegative inte-

ger)

IN stride number of elements between start of each block (inte-

ger)

IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type vector(int count, int blocklength, int stride,
         MPI Datatype oldtype, MPI Datatype *newtype)
MPI TYPE VECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR

Example: Assume, again, that oldtype has type map {(double, 0), (char, 8)}, with extent 16. A call to MPI TYPE VECTOR( 2, 3, 4, oldtype, newtype) will create the datatype with type map

{(double, 0), (char, 8), (double, 16), (char, 24), (double, 32), (char, 40),

(double, 64), (char, 72), (double, 80), (char, 88), (double, 96), (char, 104)} :

two blocks with three copies each of the old type, starting 4*16 apart.

A call to MPI TYPE VECTOR(3, 1, -2, oldtype, newtype) will create the datatype

{(double, 0), (char, 8), (double, −32), (char, −24), (double, −64), (char, −56)}.

In general, assume that oldtype has type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent. The newly created datatype has a type map with count·blocklength·n entries:

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1}),

(type_{0}, disp_{0} + extent), ..., (type_{n−1}, disp_{n−1} + extent), ...,

(type_{0}, disp_{0} + (blocklength − 1) · extent), ..., (type_{n−1}, disp_{n−1} + (blocklength − 1) · extent),

(type_{0}, disp_{0} + stride · extent), ..., (type_{n−1}, disp_{n−1} + stride · extent), ...,

(type_{0}, disp_{0} + (stride + blocklength − 1) · extent), ...,

(type_{n−1}, disp_{n−1} + (stride + blocklength − 1) · extent), ....,

(type_{0}, disp_{0} + stride · (count − 1) · extent), ...,

(type_{n−1}, disp_{n−1} + stride · (count − 1) · extent), ...,

(type_{0}, disp_{0} + (stride · (count − 1) + blocklength − 1) · extent), ...,

(type_{n−1}, disp_{n−1} + (stride · (count − 1) + blocklength − 1) · extent)}

A call to MPI TYPE CONTIGUOUS(count, oldtype, newtype) is equivalent to a call to MPI TYPE VECTOR(count, 1, 1, oldtype, newtype), or to a call to MPI TYPE VECTOR(1, count, n, oldtype, newtype), n arbitrary.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Hvector The function MPI TYPE HVECTOR is identical to MPI TYPE VECTOR, except that stride is given in bytes, rather then in elements. The use for both types of vector constructors is illustrated in section 3.13.6. (H is used for “heterogeneous”).

MPI TYPE HVECTOR( count, blocklength, stride, oldtype, newtype)

IN  count  number of blocks (nonnegative integer)
IN  blocklength  number of elements in each block (nonnegative inte-

ger)

IN  stride  number of bytes between start of each block (integer)

funcargINoldtypeold datatype (handle)

OUT  newtype  new datatype (handle)

int MPI Type hvector(int count, int blocklength, MPI Aint stride,
         MPI Datatype oldtype, MPI Datatype *newtype)
MPI TYPE HVECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR

Assume that oldtype has type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent. The newly created datatype has a type map with count·blocklength·n entries:

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1}),

(type_{0}, disp_{0} + extent), ..., (type_{n−1}, disp_{n−1} + extent), ...,

(type_{0}, disp_{0} + (blocklength − 1) · extent), ..., (type_{n−1}, disp_{n−1} + (blocklength − 1) · extent),

(type_{0}, disp_{0} + stride), ..., (type_{n−1}, disp_{n−1} + stride), ...,

(type_{0}, disp_{0} + stride + (blocklength − 1) · extent), ...,

(type_{n−1}, disp_{n−1} + stride + (blocklength − 1) · extent), ....,

(type_{0}, disp_{0} + stride · (count − 1)), ...,

(type_{n−1}, disp_{n−1} + stride · (count − 1)), ...,

(type_{0}, disp_{0} + stride · (count − 1) + (blocklength − 1) · extent), ...,

(type_{n−1}, disp_{n−1} + stride · (count − 1) + (blocklength − 1) · extent)}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Indexed The function MPI TYPE INDEXED allows replication of an old datatype into a sequence of contiguous blocks, where each block can contain different number of copies and have a different displacement. All block displacements are multiples of the old type extent.

MPI TYPE INDEXED( count, array of blocklengths, array of displacements, oldtype, newtype)

IN  count  number of blocks – also number of entries in

array of displacements and array of blocklengths (integer)

IN  array of blocklengths  number of elements per block (array of nonnegative

integers)

IN  array of displacements  displacement for each block, in multiples of oldtype

extent (array of integer)

IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type indexed(int count, int *array of blocklengths,
             int *array of displacements, MPI Datatype oldtype,
             MPI Datatype *newtype)
MPI TYPE INDEXED(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
             OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
    OLDTYPE, NEWTYPE, IERROR

Example: let oldtype have type map {(double, 0), (char, 8)}, with extent 16. Let D = (4, 0) and let B = (3, 1). A call to MPI TYPE INDEXED(2, B, D, oldtype, newtype) returns a dataype with type map

{(double, 64), (char, 72), (double, 80), (char, 88), (double, 96), (char, 104)

(double, 0), (char, 8)} :

three copies of the old type starting at displacement 64, and one copy starting at displacement 0.

In general, assume that oldtype has type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent. Let B be the array of blocklength argument and D be the array of displacements argument. The newly created datatype has n · ∑_{i=0}^{count−1} B[i] entries:

{(type_{0}, disp_{0} + D[0] · extent), ..., (type_{n−1}, disp_{n−1} + D[0] · extent), ...,
(type_{0}, disp_{0} + (D[0] + B[0] − 1) · extent), ..., (type_{n−1}, disp_{n−1} + (D[0] + B[0] − 1) · extent), ...,
(type_{0}, disp_{0} + D[count − 1] · extent), ..., (type_{n−1}, disp_{n−1} + D[count − 1] · extent), ...,
(type_{0}, disp_{0} + (D[count − 1] + B[count − 1] − 1) · extent), ...,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

(type_{n−1}, disp_{n−1} + (D[count − 1] + B[count − 1] − 1) · extent)}.

A call to MPI TYPE VECTOR(count, blocklength, stride, oldtype, newtype) is equivalent to a call to MPI TYPE INDEX(count, B, D, oldtype, newtype) where

D[j] = j · stride , j = 0, ..., count − 1,

and

B[j] = blocklength , j = 0, ..., count − 1.

Hindexed The function MPI TYPE HINDEXED is identical to MPI TYPE INDEXED, except that block displacements in array of displacements are specified in bytes, rather than in multiples of the oldtype extent.

MPI TYPE HINDEXED( count, array of blocklengths, array of displacements, oldtype, newtype)

IN  count  number of blocks – also number of entries in

array of displacements and array of blocklengths (integer)

IN  array of blocklengths  number of elements in each block (array of nonnega-

tive integers)

IN  array of displacements  byte displacement of each block (array of integer)
IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type hindexed(int count, int *array of blocklengths,
             MPI Aint *array of displacements, MPI Datatype oldtype,
             MPI Datatype *newtype)
MPI TYPE HINDEXED(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
             OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
    OLDTYPE, NEWTYPE, IERROR

Assume that oldtype has type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent. Let B be the array of blocklength argument and D be the array of displacements argument. The newly created datatype has a type map with n · ∑_{i=0}^{count−1} B[i] entries:

{(type_{0}, disp_{0} + D[0]), ..., (type_{n−1}, disp_{n−1} + D[0]), ...,

(type_{0}, disp_{0} + D[0] + (B[0] − 1) · extent), ..., (type_{n−1}, disp_{n−1} + D[0] + (B[0] − 1) · extent), ...,

(type_{0}, disp_{0} + D[count − 1]), ..., (type_{n−1}, disp_{n−1} + D[count − 1]), ...,
(type_{0}, disp_{0} + D[count − 1] + (B[count − 1] − 1) · extent), ...,
(type_{n−1}, disp_{n−1} + D[count − 1] + (B[count − 1] − 1) · extent)}.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Struct MPI TYPE STRUCT is the most general constructor: It further generalizes the previous one in that it allows each block to consist of replications of a different datatype.

MPI TYPE STRUCT(count, array of blocklengths, array of displacements, array of types, newtype)

IN  count  number of blocks (integer) – also number of entries

in arrays array of types, array of displacements and array of blocklengths

IN  array of blocklength  number of elements in each block (array of integer)
IN  array of displacements  byte displacement of each block (array of integer)
IN  array of types  type of elements in each block (array of handles to

datatype objects)

OUT  newtype  new datatype (handle)

int MPI Type struct(int count, int *array of blocklengths,
             MPI Aint *array of displacements, MPI Datatype *array of types,
             MPI Datatype *newtype)
MPI TYPE STRUCT(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
             ARRAY OF TYPES, NEWTYPE, IERROR)
    INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
    ARRAY OF TYPES(*), NEWTYPE, IERROR

Example: Let type1 have type map

{(double, 0), (char, 8)},

with extent 16. Let D = (0, 16, 26), B = (2, 1, 3) and T = (MPI FLOAT, type1,

MPI CHAR). Then a call to MPI TYPE STRUCT(3, B, D, T, newtype) returns datatype with type map,

{(float, 0), (float, 4), (double, 16), (char, 24), (char, 26), (char, 27), (char, 28)} :

two copies of MPI FLOAT starting at 0, followed by one copy of type1 starting at 16, followed by three copies of MPI CHAR, starting at 26. (We assume that a float occupies four bytes.)

In general, let T be the array of types argument, where T[i] is a handle to

type_{i} = {(type_{0}^{i}, disp_{0}^{i}), ..., (type_{n_{i}−1}^{i}, disp_{n_{i}−1}^{i})},

with extent extent_{i}. Let B be the array of blocklength argument and D be the array of displacements argument. Then the newly created datatype has a type map with ∑_{i=0}^{count−1} B[i] · n_{i} entries

{(type_{0}^{0}, disp_{0}^{0} + D[0]), ..., (type_{n_{0}}^{0}, disp_{n_{0}}^{0} + D[0]), ...,

(type_{0}^{0}, disp_{0}^{0} + D[0] + (B[0] − 1) · extent_{0}), ..., (type_{n_{0}}^{0}, disp_{n_{0}}^{0} + D[0] + (B[0] − 1) · extent_{0}), ...,

(type_{0}^{count−1}, disp_{0}^{count−1} + D[count − 1]), ..., (type_{n_{count−1}−1}^{count−1}, disp_{n_{count−1}−1}^{count−1} + D[count − 1]), ...,

(type_{0}^{count−1}, disp_{0}^{count−1} + D[count − 1] + (B[count − 1] − 1) · extent_{count−1}), ...,

(type_{n_{count−1}−1}^{count−1}, disp_{n_{count−1}−1}^{count−1} + D[count − 1] + (B[count − 1] − 1) · extent_{count−1})}

A call to MPI TYPE HINDEXED( count, B, D, oldtype, newtype) is equivalent to a call to MPI TYPE STRUCT( count, B, D, T, newtype), where each entry of T is equal to oldtype.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

3.13.2 Address and extent functions

The displacements in a general datatype are relative to some initial buffer address. Absolute addresses can be substituted for these displacements: we treat them as displacements relative to “address zero”, the start of the address space. This initial address zero is indicated by the constant MPI BOTTOM. Thus, a datatype can specify the absolute address of the entries in the communication buffer, in which case the buf argument is passed the value

MPI BOTTOM.

The address of a location in memory can be found by invoking the function MPI ADDRESS.

MPI ADDRESS(location, address)

IN  location  location in caller memory (choice)
OUT  address  address of location (integer)

int MPI Address(void* location, MPI Aint address)
MPI ADDRESS(LOCATION, ADDRESS, IERROR)
    <type> LOCATION(*)
    INTEGER ADDRESS, IERROR

Returns the (byte) address of location.

Example:

   REAL A(100,100)
   INTEGER I1, I2, DIFF
   CALL MPI_ADDRESS(A(1,1), I1, IERROR)
   CALL MPI_ADDRESS(A(10,10), I2, IERROR)
   DIFF = I2 - I1
! The value of DIFF is 909*sizeofreal; the values of I1 and I2 are
! implementation dependent.

Some implementations may not support calls to MPI ADDRESS in Fortran, where the location argument is of type CHARACTER.

The following auxiliary functions provide useful information on derived datatypes.

MPI TYPE EXTENT(datatype, extent)

IN  datatype  datatype (handle)
OUT  extent  datatype extent (integer)

int MPI Type extent(MPI Datatype datatype, MPI Aint *extent)
MPI TYPE EXTENT(DATATYPE, EXTENT, IERROR)
    INTEGER DATATYPE, EXTENT, IERROR

Returns the extent of a datatype – where extent is as defined in Eq. 3.1.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI TYPE SIZE(datatype, size)

IN  datatype  datatype (handle)
OUT  size  datatype size (integer)

int MPI Type size(MPI Datatype datatype, MPI Aint *size)
MPI TYPE SIZE(DATATYPE, SIZE, IERROR)
    INTEGER DATATYPE, SIZE, IERROR

Returns the number of bytes occupied by entries in the datatype – i.e. the sum of the sizes of the datatype elements.

MPI TYPE COUNT(datatype, count)

IN  datatype  datatype (handle)
OUT  count  datatype count (integer)

int MPI Type count(MPI Datatype datatype, int *count)
MPI TYPE COUNT(DATATYPE, COUNT, IERROR)
    INTEGER DATATYPE, COUNT, IERROR

Returns the number of entries in the datatype.

3.13.3 Lower-bound and upper-bound markers

It is often convenient to define explicitly the lower bound and upper bound of a type map, and overwrite the definition given by Equation 3.1. This allows to define a datatype that has “holes” at its beginning or its end; or a datatype with entries that extend above the upper bound or below the lower bound. Examples of such a usage are provided in section 3.13.6. To achieve this, we add two additional “pseudo-datatypes” MPI LB, and MPI UB that can be used, respectively, to mark the lower bound or the upper bound of a datatype. These pseudo-datatypes occupy no space (extent(MPI LB) = extent(MPI UB) = 0). They do not affect the size or count of a datatype, and do not affect the the content of a message created with this datatype. However, they do affect the definition of the extent of a datatype and, therefore, affect the outcome of a replication of this datatype by a datatype constructor.

    Example: Let D = (-3, 0, 6); T = (MPI LB, MPI INT, MPI UB), and B = (1, 1,

1). Then a call to MPI TYPE STRUCT(3, B, D, T, type1) creates a new datatype that has an extent of 9 (from -3 to 5), and contains an integer at displacement 0. This is the datatype defined by the sequence {(lb, -3), (int, 0), (ub, 6)} . If this type is replicated twice by a call to MPI TYPE CONTIGUOUS(2, type1, type2) then the newly created type can be described by the sequence {(lb, -3), (int, 0), (int,9), (ub, 15)} . (Entries of type lb or ub can be deleted if they are not at the endpoints of the datatype.)

In general, if

T ypemap = {(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

then the lower bound of T ypemap is defined to be

lb(T ypemap) = \{ \begin{matrix}min_{j} disp_{j} & if no entry has basic type lb \\ min\{disp_{j} : type_{j} = lb\} & otherwise \end{matrix}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Similarly, the upper bound of T ypemap is defined to be

ub(T ypemap) = \{ \begin{matrix}max_{j} disp_{j} + sizeof(type_{j}) & if no entry has basic type ub \\ max\{disp_{j} : type_{j} = ub\} & otherwise \end{matrix}

Then

extent(T ypemap) = ub(T ypemap) − lb(T ypemap) + ϵ

If type_{i} requires alignment to a byte address that is a multiple of k_{i}, then ϵ is the least nonnegative increment needed to round extent(T ypemap) to the next multiple of max_{i} k_{i}.

The formal definitions given for the various datatype constructors apply now, with the amended definition of extent.

3.13.4 Commit and free

A datatype object has to be committed before it can be used in a communication. The system may “compile” at commit time an internal representation for the datatype that facilitates communication, e.g. change from a compacted representation to a flat representation of the datatype, and select the most convenient transfer mechanism. A commited datatype can still be used as a argument in datatype constructors.

Note: the commit operation commits the datatype, i.e., the formal description of a communication buffer, not the content of that buffer. Thus, after a datatype has been committed, it can be repeatedly reused to communicate the changing content of a buffer or, indeed, the content of different buffers, with different starting addresses.

MPI TYPE COMMIT(datatype)

INOUT datatype datatype that is committed (handle)

int MPI Type commit(MPI Datatype *datatype)
MPI TYPE COMMIT(DATATYPE, IERROR)
    INTEGER DATATYPE, IERROR

There is no need to commit basic datatypes; they are “pre-committed”.

MPI TYPE FREE(datatype)

INOUT datatype datatype that is freed (handle)

int MPI Type free(MPI Datatype *datatype)
MPI TYPE FREE(DATATYPE, IERROR)
    INTEGER DATATYPE, IERROR

Marks the datatype object associated with datatype for deallocation. The object will be deallocated after any pending communication that uses this object completes, at which point datatype becomes null. It is erroneous to use a datatype as argument to an MPI function after it was freed.

Example:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

INTEGER type1, type2
CALL MPI_TYPE_CONTIGUOUS(5, MPI_REAL, type1, ierr)
              ! new type object created
CALL MPI_TYPE_COMMIT(type1, ierr)
              ! now type1 can be used for communication
type2 = type1
              ! type2 can be used for communication
              ! (it is a handle to same object as type1)
CALL MPI_TYPE_VECTOR(3, 5, 4, MPI_REAL, type1, ierr)
              ! new uncommitted type object created
CALL MPI_TYPE_COMMIT(type1, ierr)
              ! now type1 can be used anew for communication

Freeing a datatype does not affect any other datatype that was built from the freed datatype. The system should behave as if a separate copy of the datatype was created whenever this datatype was incorporated in a new derived datatype.

Implementation note:

The implementation may keep a reference count of active communications that use the datatype, in order to decide when to free it. Also, one may implement constructors of derived datatypes so that they keep pointers to their datatype arguments, rather then copying them. In such case, one needs to keep track of active datatype definition references in order to know when a datatype object can be freed.

3.13.5 Use of general datatypes in communication

Handles to derived datatypes can be passed to a communication call wherever a datatype argument is required. A call of the form MPI SEND(buf, count, datatype , ...), where count > 1, is interpreted as if the call was passed a new datatype which is the concatenation of count copies of datatype. Thus,

MPI SEND(buf, count, datatype, dest, tag, comm) is equivalent to

MPI_TYPE_CONTIGUOUS(count, datatype, newtype)
MPI_TYPE_COMMIT(newtype)
MPI_SEND(buf, 1, newtype, dest, tag, comm)

The same applies to all other communication functions that have a count and datatype argument.

Let datatype have type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent (empty entries deleted). Then the send buffer used by the send operation consists of n × count entries, where entry i × n + j is at location

addr_{i,j} = buf + extent × i + disp_{j}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

and has type type_{j}, for i = 0, ..., count − 1 and j = 0, ..., n − 1. These entries need not be contiguous, nor distinct; their order can be arbitrary.

The variable stored at address addr_{i,j} in the calling program should be of a type that matches type_{j}, where type matching is defined as in section 3.5. The message sent contains n × count entries, where entry i × n + j has type type_{j}.

Similarly, suppose that a receive operation MPI RECV(buf, count, datatype, source, tag, comm, status) is executed, where datatype has type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent (empty entries deleted). Then the receive buffer used for this receive operation consists of the n × count entries, where entry i × n + j is at location buf + extent × i + disp_{j} and has type type_{j}. If the incoming message consists of k elements, then we must have k ≤ n × count; the i × n + j-th element of the message should have a type that matches type_{j}.

Note that type matching is defined according to the type signature of the corresponding datatypes, i.e., the sequence of basic type components, and does not depend on the datatype definition.

Example:

...
CALL MPI_TYPE_CONTIGUOUS( MPI_REAL, 2, type2, ...)
CALL MPI_TYPE_CONTIGUOUS( MPI_REAL, 4, type4, ...)
CALL MPI_TYPE_CONTIGUOUS( type2, 2, type22, ...)
...
CALL MPI_SEND( a, 4, MPI_REAL, ...)
CALL MPI_SEND( a, 2, type2, ...)
CALL MPI_SEND( a, 1, type22, ...)
CALL MPI_SEND( a, 1, type4, ...)
...
CALL MPI_RECV( a, 4, MPI_REAL, ...)
CALL MPI_RECV( a, 2, type2, ...)
CALL MPI_RECV( a, 1, type22, ...)
CALL MPI_RECV( a, 1, type4, ...)

From a type matching viewpoint any of the sends legally matches any of the receives. A datatype used in a receive operation may not have overlapping entries in its type

map: the same location in the receive buffer cannot be written twice during a receive.

Type matching requirements imply that (nonempty) entries in the type map of a datatype cannot overlap, unless they coincide and have the same basic type, or one of them has basic type byte: otherwise, when the datatype is used in a communication, at least one of the two overlapping entries will not be type-consistent with the communication buffer.

Suppose, again, that MPI RECV(buf, count, datatype, dest, tag, comm, status) is executed, where datatype has type map

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})}.

The received message need not fill all the receive buffer, nor does it need to fill a number of locations which is a multiple of n; any number k of basic elements can be received, where 0 ≤ k ≤ count × n.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The number of elements received can be retrieved from status using the query function MPI GET ELEMENTS:

MPI GET ELEMENTS( status, datatype, count)

IN  status  return status of receive operation (Status)
IN  datatype  datatype used by receive operation (handle)
OUT  count  number of received basic elements (integer)

int MPI Get elements(MPI Status status, MPI Datatype datatype, int *count)
MPI GET ELEMENTS(STATUS, DATATYPE, COUNT, IERROR)
    INTEGER STATUS(MPI STATUS SIZE), DATATYPE, COUNT, IERROR

The previously used function MPI GET COUNT has a different behavior: it returns the number of “top elements” received. In the previous example, MPI GET COUNT may return any integer value k, where 0 ≤ k ≤ count. If MPI GET COUNT returns k, then the number of basic elements received (and the value returned by MPI GET ELEMENTS) is n × k. If the number of basic elements received is not a multiple of n, i.e. if the receive operation has not received an integral number of datatype “copies”, then MPI GET COUNT returns

the value MPI UNDEFINED.

Example:

...
CALL MPI_TYPE_CONTIGUOUS(2, MPI_REAL, Type2, ierr)
CALL MPI_COMMIT(Type2, ierr)
...
CALL MPI_RANK(comm, rank, ierr)
IF(rank.EQ.0)
   THEN
      CALL MPI_SEND(a, 2, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_SEND(a, 3, MPI_REAL, 1, 0, comm, ierr)
   ELSE
      CALL MPI_RECV(a, 2, Type2, 0, 0, comm, stat, ierr)
      CALL MPI_GET_COUNT(stat, i, ierr)        ! returns i=1
      CALL MPI_GET_ELEMENTS(stat, i, ierr)     ! returns i=2
      CALL MPI_RECV(a, 2, Type2, 0, 0, comm, stat, ierr)
      CALL MPI_GET_COUNT(stat, i, ierr)        ! returns i=MPI_UNDEFINED
      CALL MPI_GET_ELEMENTS(stat, i, ierr)     ! returns i=3
END IF

The function MPI GET ELEMENTS can also be used after a probe to find the number of elements in the probed message. Note that the two functions MPI GET COUNT and MPI GET ELEMENTS return the same values when they are used with basic datatypes.

Implementation note: The current definition implies that a receive cannot change the value of storage between the entries defined to compose the communication buffer. In particular, the definition implies that padding space in a structure should not be modified when such structure is copied from one process to another. This would prevents the obvious optimization of copying the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

structure, together with the padding, as one contiguous block. The implementation is free to do this optimization when it does not impact the outcome of the computation. The user can “force” this optimization by explicitly including padding as part of the message.

3.13.6 Examples

The following examples illustrate the use of derived datatypes.

First example: Send and receive a section of a 3D array.

REAL a(100,100,100), e(9,9,9)
INTEGER oneslice, twoslice, threeslice, sizeofreal, myrank, ierr
MPI_STATUS status
C      extract the section a(1:17:2, 3:11, 2:10)
C      and store it in e(*,*,*).
CALL MPI_COMM_RANK(MPI_ALL, myrank)
CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)
C     create datatype for a 1D section
CALL MPI_TYPE_VECTOR( 9, 1, 2, MPI_REAL, oneslice, ierr)
C     create datatype for a 2D section
CALL MPI_TYPE_HVECTOR(9, 1, 100*sizeofreal, oneslice, twoslice,ierr)
C     create datatype for the entire section
CALL MPI_TYPE_HVECTOR( 9, 1, 100*100*sizeofreal, twoslice, 1,
                       threeslice, ierr)
CALL MPI_TYPE_COMMIT( threeslice, ierr)
CALL MPI_SENDRECV(a(1,3,2), 1, threeslice, myrank, 0, e, 9*9*9,
                  MPI_REAL, myrank, 0, MPI_ALL, status, ierr)

Second example: Copy the (strictly) lower triangular part of a matrix.

REAL a(100,100), b(100,100)
INTEGER  disp(100), blocklen(100), ltype, myrank, ierr
MPI_STATUS status
C     copy lower triangular part of array a
C     onto lower triangular part of array b
CALL MPI_COMM_RANK(MPI_ALL, myrank)
C     compute start and size of each column

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

DO i=1, 100
  disp(i) = 100*(i-1) + i
  block(i) = 100-i
END DO
C     create datatype for lower triangular part
CALL MPI_TYPE_INDEX( 100, block, disp, MPI_REAL, ltype, ierr)
CALL MPI_TYPE_COMMIT(ltype, ierr)
CALL MPI_SENDRECV( a, 1, ltype, myrank, 0, b, 1,
              ltype, myrank, 0, MPI_ALL, status, ierr)

Third example: Transpose a matrix

REAL a(100,100), b(100,100)
INTEGER row, xpose, sizeofreal, myrank, ierr
MPI_STATUS status
C     transpose matrix a onto b
CALL MPI_COMM_RANK(MPI_ALL, myrank)
CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)
C     create datatype for one row
CALL MPI_TYPE_VECTOR( 100, 1, 100, MPI_REAL, row, ierr)
C     create datatype for matrix in row-major order
CALL MPI_TYPE_HVECTOR( 100, 1, sizeofreal, row, xpose, ierr)
CALL MPI_TYPE_COMMIT( xpose, ierr)
C     send matrix in row-major order and receive in column major order
CALL MPI_SENDRECV( a, 1, xpose, myrank, 0, b, 100*100,
          MPI_REAL, myrank, 0, MPI_ALL, status, ierr)

Another approach to the transpose problem:

REAL a(100,100), b(100,100)
INTEGER  disp(2), blocklen(2), type(2), row, row1, sizeofreal
INTEGER  myrank, ierr
MPI_STATUS status
CALL MPI_COMM_RANK(MPI_ALL, myrank)
C     transpose matrix a onto b
CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

C     create datatype for one row
      CALL MPI_TYPE_VECTOR( 100, 1, 100, MPI_REAL, row, ierr)
C     create datatype for one row, with the extent of one real number
      disp(1) = 0
      disp(2) = sizeofreal
      type(1)  = row
      type(2)  = MPI_UB
      blocklen(1)  = 1
      blocklen(2)  = 1
      CALL MPI_TYPE_STRUCT( 2, blocklen, disp, type, row1, ierr)
      CALL MPI_TYPE_COMMIT( row1, ierr)
C     send 100 rows and receive in column major order
      CALL MPI_SENDRECV( a, 100, row1, myrank, 0, b, 100*100,
                MPI_REAL, myrank, 0, MPI_ALL, status, ierr)

Fourth example: manipulate an array of structures.

struct Partstruct
   {
   int    class;  /* particle class */
   double d[6];   /* particle coordinates */
   char   b[7];   /* some additional information */
   };
struct Partstruct    particle[1000];
int                  i, dest, rank;
MPI_Comm     comm;

/* build datatype describing structure */
MPI_Datatype Particletype;
MPI_Datatype type[3] = {MPI_INT, MPI_DOUBLE, MPI_CHAR};
int          blocklen[3] = {1, 6, 7};
MPI_Aint     disp[3];

/* compute displacements of structure components */
MPI_Address( particle, disp);
MPI_Address( particle[0].d, disp+1);
MPI_Address( particle[0].b, disp+2);
for (i=0; i <3; i++) disp[i] -= disp[0];

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Type_struct( 3, blocklen, disp, type, &Particletype);
   /* If compiler does padding in mysterious ways,
   the following may be safer */
MPI_Datatype type1[4] = {MPI_INT, MPI_DOUBLE, MPI_CHAR, MPI_UB};
int          blocklen1[4] = {1, 6, 7, 1};
MPI_Aint     disp1[4];
/* compute displacements of structure components */
MPI_Address( particle, disp1);
MPI_Address( particle[0].d, disp1+1);
MPI_Address( particle[0].b, disp1+2);
MPI_Address( particle+1, disp1+3);
for (i=0; i <4; i++) disp1[i] -= disp1[0];
/* build datatype describing structure */
MPI_Type_struct( 4, blocklen1, disp1, type1, &Particletype);

              /* 4.1:
        send the entire array */
MPI_Commit( Particletype);
MPI_Send( particle, 1000, Particletype, dest, tag, comm);

              /* 4.2:
        send only the entries of class zero particles,
        preceded by the number of such entries */
MPI_Datatype Zparticles;   /* datatype describing all particles
                              with class zero (needs to be recomputed
                              if classes change) */
MPI_Datatype Ztype;
MPI_Aint     zdisp[1000];
int zblock[1000], j, k;
int zzblock[2] = {1,1};
MPI_Aint     zzdisp[2];
MPI_Datatype zztype[2];
/* compute displacements of class zero particles */
j = 0;
for(i=0; i < 1000; i++)
  if (particle[i].class==0)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

     {
     zdisp[j] = i;
     zblock[j] = 1;
     j++;
     }
/* create datatype for class zero particles  */
MPI_Type_indexed( j, zblock, zdisp, Particletype, &Zparticles);
/* prepend particle count */
MPI_Address(&j, zzdisp);
MPI_Address(particle, zzdisp+1);
zztype[0] = MPI_INT;
zztype[1] = Zparticles;
MPI_Type_struct(2, zzblock, zzdisp, zztype, &Ztype);
MPI_Type_commit( Ztype);
MPI_Send( MPI_BOTTOM, 1, Ztype, dest, tag, comm);

       /* A probably more efficient way of defining Zparticles */
/* consecutive particles with index zero are handled as one block */
j=0;
for (i=0; i < 1000; i++)
  if (particle[i].index==0)
    {
    for (k=i+1; (k < 1000)&&(particle[k].index == 0) ; k++);
    zdisp[j] = i;
    zblock[j] = k-i;
    j++;
    i = k;
    }
MPI_Type_indexed( j, zblock, zdisp, Particletype, &Zparticles);

                /* 4.3:
          send the first two coordinates of all entries */
MPI_Datatype Allpairs;      /* datatype for all pairs of coordinates */
MPI_Aint sizeofentry;
MPI_Type_extent( Particletype, sizeofentry);
     /* sizeofentry can also be computed by subtracting the address
        of particle[0] from the address of particle[1] */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Type_hvector( 1000, 2, sizeofentry, MPI_REAL, &Allpairs);
MPI_Type_commit( Allpairs);
MPI_Send( particle.d, 1, Allpairs, dest, tag, comm);
      /* an alternative solution to 4.3 */
MPI_Datatype Onepair;   /* datatype for one pair of coordinates, with
                          the extent of one particle entry */
MPI_Aint disp2[3];
MPI_Datatype type2[3] = {MPI_LB, MPI_DOUBLE, MPI_UB};
int blocklen2[3] = {1, 2, 1};
MPI_Address( particle, disp2);
MPI_Address( particle[0].d, disp2+1);
MPI_Address( particle+1, disp2+2);
for (i=0; i<2; i++) disp2[i] -= disp2[0];
MPI_Type_struct( 3, blocklen2, disp2, type2, &Onepair);
MPI_Commit( Onepair);
MPI_Send( particle[0].d, 1000, Onepair, dest, tag, comm);

Fifth example: same array manipulations as in the previous example, but using absolute addresses in datatypes.

struct Partstruct
   {
   int class;
   double d[6];
   char b[7];
   };
struct Partstruct particle[1000];
           /* build datatype describing first array entry */
MPI_Datatype Particletype;
MPI_Datatype type[3] = {MPI_INT, MPI_DOUBLE, MPI_CHAR};
int          block[3] = {1, 6, 7};
MPI_Aint     disp[3];
MPI_Address( particle, disp);
MPI_Address( particle[0].d, disp+1);
MPI_Address( particle[0].b, disp+2);
MPI_Type_struct( 3, block, disp, type, &Particletype);
/* Particletype describes first array entry -- using absolute
   addresses */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

                  /* 5.1:
            send the entire array */
MPI_Commit( Particletype);
MPI_Send( MPI_BOTTOM, 1000, Particletype, dest, tag, comm);

                 /* 5.2:
         send the entries of class zero,
         preceded by the number of such entries */
MPI_Datatype Zparticles, Ztype;
MPI_Aint zdisp[1000]
int zblock[1000], i, j, k;
int zzblock[2] = {1,1};
MPI_Datatype zztype[2];
MPI_Aint     zzdisp[2];
j=0;
for (i=0; i < 1000; i++)
  if (particle[i].index==0)
    {
    for (k=i+1; (k < 1000)&&(particle[k].index == 0) ; k++);
    zdisp[j] = i;
    zblock[j] = k-i;
    j++;
    i = k;
    }
MPI_Type_indexed( j, zblock, zdisp, Particletype, &Zparticles);
/* Zparticles describe particles with class zero, using
   their absolute addresses*/
/* prepend particle count */
MPI_Address(&j, zzdisp);
zzdisp[1] = MPI_BOTTOM;
zztype[0] = MPI_INT;
zztype[1] = Zparticles;
MPI_Type_struct(2, zzblock, zzdisp, zztype, &Ztype);
MPI_Type_commit( Ztype);
MPI_Send( MPI_BOTTOM, 1, Ztype, dest, tag, comm);

Sixth example: handling of unions.

union {

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

   int     ival;
   float   fval;
      } u[1000]
int utype;
/* All entries of u have identical type; variable
   utype keeps track of their current type */
MPI_Datatype   type[2];
int            blocklen[2] = {1,1};
MPI_Aint       disp[2];
MPI_Datatype   mpi_utype[2];
MPI_Aint       i,j;
/* compute an MPI datatype for each possible union type;
   assume values are left-aligned in union storage. */
MPI_Address( u, &i);
MPI_Address( u+1, &j);
disp[0] = 0; disp[1] = j-i
type[1] = MPI_UB;
type[0] = MPI_INT;
MPI_Type_struct(2, blocklen, disp, type, &mpi_utype[0]);
type[0] = MPI_FLOAT;
MPI_Type_struct(2, blocklen, disp, type, &mpi_utype[1]);
for(i=0; i<2; i++) MPI_Type_commit(mpi_utype[i]);
/* actual communication */
MPI_Send(u, 1000, mpi_utype[utype], dest, tag, comm);

3.13.7 Correct use of addresses

Successively declared variables in C or Fortran are not necessarily stored at contiguous locations. Thus, care must be exercised that displacements do not cross from one variable to another. Also, in machines with segmented address space, addresses are not unique and address arithmetic has some peculiar properties. Thus, use of addresses, i.e. displacements relative to the start address MPI BOTTOM, has to be restricted.

Variables belong to the same sequential storage if they belong to the same array, to the same COMMON block in Fortran, or to the same structure in C. Valid addresses are defined recursively as follows:

1. The function MPI ADDRESS returns a valid address, when passed as argument a variable of the calling program.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

2. The buf argument of a communication function evaluates to a valid address, when passed as argument a variable of the calling program.

3. If v is a valid address, and i is an integer, then v+i is a valid address, provided v and v+i are in the same sequential storage.

4. If v is a valid address then MPI BOTTOM + v is a valid address.

A correct program uses only valid addresses to identify the locations of entries in communication buffers. Furthermore, if u and v are two valid addresses, then the (integer) difference u - v can be computed only if both u and v are in the same sequential storage; no other arithmetic operations can be meaningfully executed on addresses.

We say that a datatype is absolute if all displacements within this datatype are valid (absolute) addresses; it is relative otherwise.

A correct program obeys the following constraints in the use of datatypes:

• If the oldtype argument used in MPI TYPE CONTIGUOUS, MPI TYPE VECTOR, MPI TYPE HVECTOR, MPI TYPE INDEX, or MPI TYPE HINDEX is absolute, then all addresses within it must be to variables contained within the same array or structure; the result datatype is also absolute, and all computed addresses must also fall within the same array or structure.

• If all entries of the array of types argument of MPI TYPE STRUCT are absolute addresses (computed by MPI ADDRESS), then the result datatype is also absolute. Each new address computed from an old address must fall within the same array or record as the old address. (Addresses in the old type may not be all within the same array or record.)

• If a communication call is invoked with arguments buf and datatype, then either buf = MPI BOTTOM and datatype is a handle to an absolute datatype, or buf is set to the address of a program variable, and all displacements in datatype relative to this base yield addresses that are within the same sequential storage as this variable.

In summary, the type constructors MPI TYPE CONTIGUOUS, MPI TYPE VECTOR, MPI TYPE HVECTOR, MPI TYPE INDEX and MPI TYPE HINDEX can be recursively applied to build datatypes that will combine variables that belong to the same sequential storage; variables that do not belong to the same sequential storage can be combined together using one application of MPI TYPE STRUCT.

Implementation note: There is no need to distinguish (absolute) addresses and (relative) displacements on a machine

with contiguous address space: MPI BOTTOM is zero, and both addresses and displacements are integers. On machines where the distinction is required, addresses are recognized as expressions that involve MPI BOTTOM.

3.14 Universal communication functions

This section is not part of the MPI definition. Its goal is to indicate how the definition or implementation of the point to point communication functions can be reduced to a small number of primitives.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

We assume that the function MPI TYPE COMMIT is not changing the representation of a datatype, so that this function is redundant.

Communication with a count argument that is > 1 can be reduced to communication with count =1, by replicating the datatype argument count times. Communication with a a buf argument that is not MPI BOTTOM can be reduced to communication with buf = MPI BOTTOM by creating a suitable displaced new datatype. Thus:

MPI SEND(buf, count, datatype, dest, tag, comm) is

Type[1] = datatype
MPI_ADDRESS( buf, Disp[1]);
Blocklen[1] = count;
MPI_TYPE_STRUCT( 1, Type, Disp, Blocklen, Newtype)
MPI_SEND( MPI_BOTTOM, 1, Newtype, dest, tag, comm)

The same construction applies to all other communication functions with arguments count and datatype. We shall henceforth restrict ourselves to communication that involves only one element with an absolute address (count =1, and buf = MPI BOTTOM).

It is convenient to to make explicit the persistence attribute of communication objects. A persistent communication object need be explicitly deallocated by a MPI REQUEST FREE operation. On the other hand, an ephemeral communication object is good for a single communication. It is deallocated by the system when the first communication it is used for completes.

The function MPI COMM INIT is a new universal function for the creation of communication objects:

MPI COMM INIT(datatype, source-dest, tag, comm, op-mode, persistence, request)

IN  datatype  datatype of element sent or received (handle)
IN  source-dest  rank of destination or source (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
IN  op-mode  one of  MPI STANDARD,  MPI READY,

MPI SYNCHRONOUS or MPI RECV

IN  persistence  one of MPI PERSISTENT or MPI EPHEMERAL
OUT  request  communication request (handle)

3.14.1 Persistent communication objects

MPI CREATE SEND(MPI BOTTOM, 1, datatype, dest, tag, comm, request) is

MPI_COMM_INIT(datatype, dest, tag, comm, MPI_STANDARD,
    MPI_PERSISTENT, request)

The functions MPI CREATE RSEND, MPI CREATE RSEND and MPI CREATE RECV are dealt with in a similar manner.

The functions MPI START and MPI REQUEST FREE are primitive.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

3.14.2 Nonblocking communication initiation

MPI ISEND(MPI BOTTOM, 1, datatype, dest, tag, comm, request) is

MPI_COMM_INIT(datatype, dest, tag, comm, MPI_STANDARD,
     MPI_EPHEMERAL, request)
MPI_START(request)

The functions MPI IRSEND, MPI ISSEND and MPI IRECV are handled in a similar manner.

3.14.3 Communication completion

The two primitive completion operations are MPI WAITANY and MPI TESTALL.

The function MPI WAIT can be implemented by a call to MPI WAITANY with an arrayof-requests argument of length one.

The function MPI WAITALL can be implemented as a loop where an MPI WAIT call is executed for each successive request in the array-of-requests.

The function MPI TEST can be implemented by a call to MPI TESTALL with an arrayof-requests of length one.

The function MPI TESTANY can be implemented as a loop where an MPI TEST call is executed for each successive request in the array-of-request.

3.14.4 Blocking communication

MPI SEND(MPI BOTTOM, 1, datatype, dest, tag, comm) is

MPI_COMM_INIT(datatype, dest, tag, comm, MPI_STANDARD,
      MPI_EPHEMERAL, request)
MPI_START(request)
MPI_WAIT(request, dontcarestatus)

The functions MPI RSEND and MPI SSEND are handled in a similar manner.

MPI RECV(MPI BOTTOM, 1, datatype, source, tag, comm) is

MPI_COMM_INIT(datatype, dest, tag, comm, MPI_RECV, MPI_EPHEMERAL, request)
MPI_START(request)
MPI_WAIT(request, status)

3.14.5 Probe and cancel

The functions MPI PROBE and MPI CANCEL are primitive.

3.14.6 Return status

The functions MPI GET SOURCE, MPI GET TAG, MPI GET LEN, MPI PROBE LEN, and MPI TEST CANCELLED are primitive.

These functions are simple macros that access records in a structure.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

3.14.7 send-receive and exchange

MPI SENDRECV( send buf, send count, send type, dest, recv buf, recv count, recv type, source, tag, comm, status) is

MPI_ISEND(send_buf, send_count, send_type, dest, tag, comm, request[0])
MPI_IRECV(recv_buf, recv_count, recv_type, source, tag, comm, request[1])
MPI_WAITALL(2, request, status_array)
status = status_array[1]

The nonblocking sends and receives can be replaced by their primitive implementation. The function MPI SENDRECV REPLACE can be handled in a similar manner; a tem-

porary buffer need be allocated to replicate the send and receive buffer. A size for this buffer can be computed using the function MPI TYPE SIZE.

3.14.8 Derived datatypes

We have outlined in section 3.13.1 how each datatype constructor can be expressed in terms of the next one. Thus all datatype constructors can be expressed in terms of the constructor MPI TYPE STRUCT; the functions MPI TYPE FREE, MPI TYPE EXTENT, MPI TYPE SIZE, MPI TYPE COUNT and MPI ADDRESS are primitive.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 4

Collective Communication

4.1 Introduction

Collective communication is defined to be communication that involves a group of processes. The functions provided by the MPI collective communication include:

• Broadcast from one member to all members of a group.

• Barrier synchronization across all group members

• Gather data from all group members to one member.

• Scatter data from one member to all members of a group.

• Global operations such as sum, max, min, etc., were the result is known by all group members and a variation where the result is known by only one member. The ability to have user defined global operations.

• Scan across all members of a group (also called parallel prefix).

• Broadcast from all members to all members of a group.

• Scatter/Gather data from all members to all members of a group (also called complete exchange or all-to-all).

While vendors may optimize some of these collective routines for their architectures, a complete library of the collective communication routines can be written entirely using the MPI point-to-point communication functions and a few auxiliary functions.

A collective operation is executed by having all processes in the group call the communication routine, with matching arguments. The syntax and semantics of the collective operations are defined to be consistent with the syntax and semantics of the point-to-point operations. Thus general datatypes are allowed and must match between sending and receiving processes as specified in the chapter on point-to-point functions. One of the key arguments is a communicator that defines the group of participating processes and provides a context for the operation. Several collective routines such as broadcast and gather have a single originating or receiving process. In this chapter such processes are called the root. Some arguments in the collective functions are specified as “significant only at root”. These arguments are ignored for all participants except the root, and can be set to any value. The reader is referred to chapter 3 for information concerning communication buffers, general

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

datatypes and type matching rules; and to chapter 5 for information on how to define groups and create communicators.

Collective routines can (but are not required to) return as soon as their participation in the collective communication is complete. The completion of a call indicates that the caller is now free to access the locations in the communication buffer. It does not indicate that other processes in the group have started the operation (unless otherwise indicated in the description of the operation). The successful completion of a collective communication call may depend on the execution of a matching call at all processes in the group. Thus, a collective communication call may, or may not, have the effect of synchronizing all calling processes. Collective communication calls may use the same communicators as point to point communication; MPI guarantees that messages generated on behalf of collective communication calls will not be confused with messages generated by point to point communication.

A more detailed discussion of the correct use of the collective routines can be found at the end of this chapter.

4.2 Communication Functions

The key concept of the collective functions is to have a “group” of participating processes. The routines do not have a group identifier as an explicit argument. Instead, there is a communicator argument. In this chapter a communicator can be thought of as a group identifier linked with a context. (Inter-communicators, that is between groups communicators, are not allowed in the collective functions.)

4.3 Barrier synchronization

MPI BARRIER( comm )

IN  comm  communicator (handle)

int MPI Barrier(MPI Comm comm )
MPI BARRIER(COMM, IERROR)
    INTEGER COMM, IERROR

MPI BARRIER blocks the caller until all group members have called it; the call returns at any process only after all group members have entered the call.

4.4 Data move functions

Figure 4.1 illustrates the the different collective move functions supported by MPI.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.1: Collective move functions illustrated for a group of six processes. In each case, each row of boxes represents data locations in one process. Thus, in the one-all broadcast, initially just the first process contains the data A_{0}, but after the broadcast all processes contain it.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

4.4.1 Broadcast

MPI BCAST( buffer, count, datatype, root, comm )

INOUT buffer starting address of buffer (choice)

IN  count  number of entries in buffer (integer)
IN  datatype  data type of buffer (handle)
IN  root  rank of broadcast root (integer)
IN  comm  communicator (handle)

int  MPI Bcast(void* buf, int count, MPI Datatype datatype, int root,
             MPI Comm comm )
MPI BCAST(BUFFER, COUNT, DATATYPE, ROOT, COMM, IERROR)
    <type> BUFFER(*)
    INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR

MPI BCAST broadcasts a message from the process with rank root to all other processes of the group. It is called by all members of group using the same arguments for comm, root and matching arguments for count, datatype. On return the contents of the buffer of the process with rank root is contained in the buffer of the calling process.

4.4.2 Gather

MPI GATHER( sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)

IN  sendbuf  starting address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice, significant only at

root)

IN  recvcount  number of elements for any single receive (integer, sig-

nificant only at root)

IN  recvtype  data type of recv buffer elements (significant only at

root) (handle)

IN  root  rank of receiving process (integer)
IN  comm  communicator (handle)

int MPI Gather(void* sendbuf, int sendcount, MPI Datatype sendtype,
             void* recvbuf, int recvcount, MPI Datatype recvtype, int root,
             MPI Comm comm)
MPI GATHER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
             ROOT, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, ROOT, COMM, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Each process (including the root process) sends the contents of its send buffer to the root process. The root process receives the messages at stores them in rank order. The outcome is as if each of the n processes in the group had executed a call to

     MPI Send(sendbuf, sendcount, sendtype, root, ...),

and the root had executed n calls to

     MPI Recv(recvbuf[i], recvcount, recvtype, i, ...),

where,

     recvbuf[i] = recvbuf + i × recvcount × extent(recvtype).

An alternative description is that the n messages sent by the processes in the group are concatenated in rank order, and the resulting message is received by the root as if by a call to MPI RECV(recvbuf, recvcount ×n, recvtype, ...).

The receive buffer is ignored for all non-root processes.

General derived datatypes are allowed for both sendtype and recvtype. The type signature of sendcount, sendtype on process i must be equal to the type signature of recvcount, recvtype at the root. Note that the amount of data sent must be equal to the amount received (pairwise between each process and the root). MPI GATHER and all other data movement collective routines make this restriction and provide no facility (such as the status argument of MPI RECV) for discovering how much data was sent.

All arguments to the function are significant on process root, while on other processes, only arguments sendbuf, sendcount, sendtype, root, comm are significant. The arguments root and comm must have identical values on all processes.

MPI GATHERV( sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm)

IN  sendbuf  starting address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice, significant only at

root)

IN recvcounts integer array (of length group size) containing the number of elements that are received from each process (significant only at root)

IN displs integer array (of length group size). Entry i specifies the displacement relative to recvbuf at which to place the incoming data from process i (significant only at root)

IN recvtype data type of recv buffer elements (significant only at

root) (handle)

IN  root  rank of receiving process (integer)
IN  comm  communicator (handle)

int MPI Gatherv(void* sendbuf, int sendcount, MPI Datatype sendtype,
             void* recvbuf, int *recvcounts, int *displs,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

             MPI Datatype recvtype, int root, MPI Comm comm)
MPI GATHERV(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNTS, DISPLS,
             RECVTYPE, ROOT, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, RECVCOUNTS(*), DISPLS(*), RECVTYPE, ROOT,
    COMM, IERROR

MPI GATHERV extends the functionality of MPI GATHER by allowing a varying count of data from each process (since recvcounts is now an array), and also allows more flexibility as to where the data is placed on the root (by providing the new argument, displs).

The outcome is as if each process has sent a message to the root,

     MPI Send(sendbuf, sendcount, sendtype, root, ...),

and the root executed n receives,

     MPI Recv(recvbuf + disp[i] × extent(recvtype), recvcounts[i], recvtype, i, ...).

Messages are placed in the receive buffer of the root process in rank order, that is, the data sent from process j is placed in the j-th portion of the receive buffer recvbuf on process root. The j-th portion of recvbuf begins at offset displs[j] elements (in terms of recvtype) into recvbuf.

The type signature implied by sendcount, sendtype on process i must be equal to the type signature implied by recvcounts[i], recvtype at the root (however, the type maps may be different).

All arguments to the function are significant on process root, while on other processes, only arguments sendbuf, sendcount, sendtype, root, comm are significant. The arguments root and comm must have identical values on all processes.

For both functions, the specification of count(s), type(s), (displacements), should not cause any location on the root to be written more than once. Such a call is erroneous.

We illustrate the matching conditions with the following examples.

4.4.3 Examples of Usage of MPI GATHER, MPI GATHERV

Example 1

Gather 100 ints from every process in group to root. See figure 4.2.

    MPI_Comm comm;
    int gsize,sendarray[100];
    int root, *rbuf;
    ...
    /* The variable comm is set elsewhere in the program
     */
    MPI_Comm_size( comm, &gsize);
    rbuf = (int *)malloc(gsize*100*sizeof(int));
    MPI_Gather( sendarray, 100, MPI_INT, rbuf, 100, MPI_INT, root, comm);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Figure 4.2: The root process gathers 100 ints from each process in the group.

Example 2

Do the same as in previous example, but use a derived datatype. Note that the type cannot be the entire set of gsize*100 ints since type matching is defined pairwise between the root and each process in the gather.

MPI_Comm comm;
int gsize,sendarray[100];
int root, *rbuf;
MPI_Datatype rtype;
...
/* The variable comm is set elsewhere in the program
 */
MPI_Comm_size( comm, &gsize);
rbuf = (int *)malloc(gsize*100*sizeof(int));
MPI_Type_contiguous( 100, MPI_INT, &rtype );
MPI_Type_commit( rtype );
MPI_Gather( sendarray, 100, MPI_INT, rbuf, 1, rtype, root, comm);

Example 3

Now have each process send 100 ints to root, but place each set (of 100) stride ints apart at receiving end. Use MPI GATHERV and the displs argument to achieve this effect. Assume stride ≥ 100. See figure 4.3.

MPI_Comm comm;
int gsize,sendarray[100];
int root, *rbuf, stride;
int *displs,i,*rcounts;
...
MPI_Comm_size( comm, &gsize);
rbuf = (int *)malloc(gsize*stride*sizeof(int));
displs = (int *)malloc(gsize*sizeof(int));
rcounts = (int *)malloc(gsize*sizeof(int));

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.3: The root process gathers 100 ints from each process in the group, each set is placed stride ints apart.

for (i=0; i<gsize; ++i) {
    displs[i] = i*stride;
    rcounts[i] = 100;
}
MPI_Gatherv( sendarray, 100, MPI_INT, rbuf, rcounts, displs, MPI_INT,
                                                           root, comm);

Note that the program is erroneous if stride < 100.

Example 4

Same as Example 3 on the receiving side, but send the 100 ints from the 0th column of a 100×150 int array, in C. See figure 4.4.

MPI_Comm comm;
int gsize,sendarray[100][150];
int root, *rbuf, stride;
MPI_Datatype stype;
int *displs,i,*rcounts;
...
MPI_Comm_size( comm, &gsize);
rbuf = (int *)malloc(gsize*stride*sizeof(int));
displs = (int *)malloc(gsize*sizeof(int));
rcounts = (int *)malloc(gsize*sizeof(int));
for (i=0; i<gsize; ++i) {
    displs[i] = i*stride;
    rcounts[i] = 100;
}
/* Create datatype for 1 column of array
 */
MPI_Type_vector( 100, 1, 150, MPI_INT, &stype);
MPI_Type_commit( stype );
MPI_Gatherv( sendarray, 1, stype, rbuf, rcounts, displs, MPI_INT,
                                                         root, comm);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Figure 4.4: The root process gathers column 0 of a 100×150 C array, and each set is placed stride ints apart.

Example 5

Process i sends (100-i) ints from the i-th column of a 100 × 150 int array, in C. It is received into a buffer with stride, as in the previous two examples. See figure 4.5.

MPI_Comm comm;
int gsize,sendarray[100][150],*sptr;
int root, *rbuf, stride, myrank;
MPI_Datatype stype;
int *displs,i,*rcounts;
...
MPI_Comm_size( comm, &gsize);
MPI_Comm_rank( comm, &myrank );
rbuf = (int *)malloc(gsize*stride*sizeof(int));
displs = (int *)malloc(gsize*sizeof(int));
rcounts = (int *)malloc(gsize*sizeof(int));
for (i=0; i<gsize; ++i) {
    displs[i] = i*stride;
    rcounts[i] = 100-i;     /* note change from previous example */
}
/* Create datatype for the column we are sending
 */
MPI_Type_vector( 100-myrank, 1, 150, MPI_INT, &stype);
MPI_Type_commit( stype );
/* sptr is the address of start of "myrank" column
 */
sptr = &sendarray[0][myrank];
MPI_Gatherv( sptr, 1, stype, rbuf, rcounts, displs, MPI_INT, root, comm);

Note that a different amount of data is received from each process.

Example 6

Same as Example 5, but done in a different way at the sending end. We create a datatype that causes the correct striding at the sending end so that that we read a column of a C

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.5: The root process gathers 100-i ints from column i of a 100×150 C array, and each set is placed stride ints apart.

array — this was also done in Example 3, 2nd part, section 3.13.6.

MPI_Comm comm;
int gsize,sendarray[100][150],*sptr;
int root, *rbuf, stride, myrank, disp[2], blocklen[2];
MPI_Datatype stype,type[2];
int *displs,i,*rcounts;
...
MPI_Comm_size( comm, &gsize);
MPI_Comm_rank( comm, &myrank );
rbuf = (int *)malloc(gsize*stride*sizeof(int));
displs = (int *)malloc(gsize*sizeof(int));
rcounts = (int *)malloc(gsize*sizeof(int));
for (i=0; i<gsize; ++i) {
    displs[i] = i*stride;
    rcounts[i] = 100-i;
}
/* Create datatype for one int, with extent of entire row
 */
disp[0] = 0;       disp[1] = 150*sizeof(int);
type[0] = MPI_INT; type[1] = MPI_UB;
blocklen[0] = 1;   blocklen[1] = 1;
MPI_Type_struct( 2, blocklen, disp, type, &stype );
MPI_Type_commit( stype );
sptr = &sendarray[0][myrank];
MPI_Gatherv( sptr, 100-myrank, stype, rbuf, rcounts, displs, MPI_INT,
                                                           root, comm);

Example 7

Same as Example 5 at sending side, but at receiving side we make the stride between received blocks vary from block to block. See figure 4.6.

MPI_Comm comm;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

int gsize,sendarray[100][150],*sptr;
int root, *rbuf, *stride, myrank, bufsize;
MPI_Datatype stype;
int *displs,i,*rcounts,offset;
...
MPI_Comm_size( comm, &gsize);
MPI_Comm_rank( comm, &myrank );
stride = (int *)malloc(gsize*sizeof(int));
...
/* stride[i] for i = 0 to gsize-1 is set somehow
 */
/* set up displs and rcounts vectors first
 */
displs = (int *)malloc(gsize*sizeof(int));
rcounts = (int *)malloc(gsize*sizeof(int));
offset = 0;
for (i=0; i<gsize; ++i) {
    displs[i] = offset;
    offset += stride[i];
    rcounts[i] = 100-i;
}
/* the required buffer size for rbuf is now easily obtained
 */
bufsize = displs[gsize-1]+rcounts[gsize-1];
rbuf = (int *)malloc(bufsize*sizeof(int));
/* Create datatype for the column we are sending
 */
MPI_Type_vector( 100-myrank, 1, 150, MPI_INT, &stype);
MPI_Type_commit( stype );
sptr = &sendarray[0][myrank];
MPI_Gatherv( sptr, 1, stype, rbuf, rcounts, displs, MPI_INT,
                                                    root, comm);

Example 8

Process i sends num ints from the i-th column of a 100 × 150 int array, in C. The complicating factor is that the various values of num are not known to root, so a separate gather must first be run to find these out. The data is placed contiguously at the receiving end.

MPI_Comm comm;
int gsize,sendarray[100][150],*sptr;
int root, *rbuf, stride, myrank, disp[2], blocklen[2];
MPI_Datatype stype,types[2];
int *displs,i,*rcounts,num;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.6: The root process gathers 100-i ints from column i of a 100×150 C array, and each set is placed stride[i] ints apart (a varying stride).

...
MPI_Comm_size( comm, &gsize);
MPI_Comm_rank( comm, &myrank );
/* First, gather nums to root
 */
rcounts = (int *)malloc(gsize*sizeof(int));
MPI_Gather( &num, 1, MPI_INT, rcounts, 1, MPI_INT, root, comm);
/* root now has correct rcounts, using these we set displs[] so
 * that data is placed contiguously (or concatenated) at receive end
 */
displs = (int *)malloc(gsize*sizeof(int));
displs[0] = 0;
for (i=1; i<gsize; ++i) {
    displs[i] = displs[i-1]+rcounts[i-1];
}
/* And, create receive buffer
 */
rbuf = (int *)malloc(gsize*(displs[gsize-1]+rcounts[gsize-1])*sizeof(int));
/* Create datatype for one int, with extent of entire row
 */
disp[0] = 0;       disp[1] = 150*sizeof(int);
type[0] = MPI_INT; type[1] = MPI_UB;
blocklen[0] = 1;   blocklen[1] = 1;
MPI_Type_struct( 2, blocklen, disp, type, &stype );
MPI_Type_commit( stype );
sptr = &sendarray[0][myrank];
MPI_Gatherv( sptr, num, stype, rbuf, rcounts, displs, MPI_INT,
                                                           root, comm);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

4.4.4 Scatter

MPI SCATTER( sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)

IN  sendbuf  address of send buffer (choice, significant only at root)
IN  sendcount  number of elements sent to each process (integer, sig-

nificant only at root)

IN  sendtype  data type of send buffer elements (significant only at

root) (handle)

OUT  recvbuf  address of receive buffer (choice)
IN  recvcount  number of elements in receive buffer (integer)
IN  recvtype  data type of receive buffer elements (handle)
IN  root  rank of sending process (integer)
IN  comm  communicator (handle)

int MPI Scatter(void* sendbuf, int sendcount, MPI Datatype sendtype,
             void* recvbuf, int recvcount, MPI Datatype recvtype, int root,
             MPI Comm comm)
MPI SCATTER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
             ROOT, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, ROOT, COMM, IERROR

MPI SCATTER is the inverse operation to MPI GATHER.

The root process sends a message to each process (including itself); each process stores the incoming message in its receive buffer. The outcome is as if the root executed n send operations,

     MPI Send(sendbuf + i × sendcount × extent(sendtype), sendcount, sendtype, i, ...),

and each process executed a receive,

     MPI Recv(recvbuf, recvcount, recvtype, i, ...).

An alternative description is that the root send a message with MPI Send(sendbuf, sendcount×n, sendtype, ...); this message is split into n equal segments, and the i − th segment is sent to the i-th process in the group; and each process receives this message as above.

The type signature associated with sendcount, sendtype at the root must be equal to the type signature associated with recvcount, recvtype at all processes (however, the type maps may be different).

All arguments to the function are significant on process root, while on other processes, only arguments recvbuf, recvcount, recvtype, root, comm are significant. The arguments root and comm must have identical values on all processes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI SCATTERV( sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm)

IN  sendbuf  address of send buffer (choice, significant only at root)
IN  sendcounts  integer array (of length group size) specifying the num-

ber of elements to send to each processor

IN  displs  integer array (of length group size). Entry i specifies

the displacement (relative to sendbuf from which to take the outgoing data to process i

IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice)
IN  recvcount  number of elements in receive buffer (integer)
IN  recvtype  data type of receive buffer elements (handle)
IN  root  rank of sending process (integer)
IN  comm  communicator (handle)

int MPI Scatterv(void* sendbuf, int *sendcounts, int *displs,
             MPI Datatype sendtype, void* recvbuf, int recvcount,
             MPI Datatype recvtype, int root, MPI Comm comm)
MPI SCATTERV(SENDBUF, SENDCOUNTS, DISPLS, SENDTYPE, RECVBUF, RECVCOUNT,
             RECVTYPE, ROOT, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNTS(*), DISPLS(*), SENDTYPE, RECVCOUNT, RECVTYPE, ROOT,
    COMM, IERROR

MPI SCATTERV is the inverse operation to MPI GATHERV.

MPI SCATTERV extends the functionality of MPI SCATTER by allowing a varying count of data to be sent to each process (since sendcounts is now an array), and also allows more flexibility as to where the data is taken from on the root (by providing the new argument, displs).

The outcome is as if the root excuted n send operations,

     MPI Send(sendbuf + displs[i] × extent(sendtype), sendcounts[i], sendtype, i, ...),

and each process executed a receive,

     MPI Recv(recvbuf, recvcount, recvtype, i, ...).

The type signature implied by sendcount[i], sendtype at the root must be equal to the type signature implied by recvcount, recvtype at process i (however, the type maps may be different).

All arguments to the function are significant on process root, while on other processes, only arguments recvbuf, recvcount, recvtype, root, comm are significant. The arguments root and comm must have identical values on all processes.

For both functions, the specification of count(s), type(s), (displacements), should not cause any location on the root to be read more than once.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.7: The root process scatters sets of 100 ints to each process in the group.

4.4.5 Examples of Usage of MPI SCATTER, MPI SCATTERV

Example 9

The reverse of Example 1. Scatter sets of 100 ints from the root to each process in the group. See figure 4.7.

MPI_Comm comm;
int gsize,*sendbuf;
int root, rbuf[100];
...
/* The variable comm is set elsewhere in the program
 */
MPI_Comm_size( comm, &gsize);
sendbuf = (int *)malloc(gsize*100*sizeof(int));
MPI_Scatter( sendbuf, 100, MPI_INT, rbuf, 100, MPI_INT, root, comm);

Example 10

The reverse of Example 3. The root process scatters sets of 100 ints to the other processes, but the sets of 100 are stride ints apart in the sending buffer. Requires use of MPI SCATTERV. Assume stride ≥ 100. See figure 4.8.

MPI_Comm comm;
int gsize,*sendbuf;
int root, rbuf[100], i, *displs, *scounts;
...
MPI_Comm_size( comm, &gsize);
sendbuf = (int *)malloc(gsize*stride*sizeof(int));
displs = (int *)malloc(gsize*sizeof(int));
scounts = (int *)malloc(gsize*sizeof(int));
for (i=0; i<gsize; ++i) {
    displs[i] = i*stride;
    scounts[i] = 100;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.8: The root process scatters sets of 100 ints, moving by stride ints from send to send in the scatter.

}
MPI_Scatterv( sendbuf, displs, scounts, MPI_INT, rbuf, 100, MPI_INT,
                                                          root, comm);

Example 11

The reverse of Example 7. We have a varying stride between blocks at sending (root) side, at the receiving side we receive into the i-th column of a 100×150 C array. See figure 4.9.

MPI_Comm comm;
int gsize,recvarray[100][150],*rptr;
int root, *sendbuf, myrank, bufsize, *stride;
MPI_Datatype rtype;
int i, *displs, *scounts, offset;
...
MPI_Comm_size( comm, &gsize);
MPI_Comm_rank( comm, &myrank );
stride = (int *)malloc(gsize*sizeof(int));
...
/* stride[i] for i = 0 to gsize-1 is set somehow
 */
/* sendbuf comes from elsewhere
 */
...
displs = (int *)malloc(gsize*sizeof(int));
scounts = (int *)malloc(gsize*sizeof(int));
offset = 0;
for (i=0; i<gsize; ++i) {
    displs[i] = offset;
    offset += stride[i];
    scounts[i] = 100 - i;
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.9: The root scatters gathers blocks of 100-i ints into column i of a 100×150 C array. At the sending side, the blocks are stride[i] ints apart.

    /* Create datatype for the column we are receiving
     */
    MPI_Type_vector( 100-myrank, 1, 150, MPI_INT, &rtype);
    MPI_Type_commit( rtype );
    rptr = &recvarray[0][myrank];
    MPI_Scatterv( sendbuf, scounts, displs, MPI_INT, rptr, 1, rtype,
                                                            root, comm);

4.4.6 Gather-to-all

MPI ALLGATHER( sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm)

IN  sendbuf  starting address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice)
IN  recvcount  number of elements received from any process (inte-

ger)

IN  recvtype  data type of receive buffer elements (handle)
IN  comm  communicator (handle)

int MPI Allgather(void* sendbuf, int sendcount, MPI Datatype sendtype,
             void* recvbuf, int recvcount, MPI Datatype recvtype,
             MPI Comm comm)
MPI ALLGATHER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
             COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, COMM, IERROR

MPI ALLGATHER can be thought of as MPI GATHER, but where all processes receive the result, instead of just the root. The j-th block of data sent from each process is received by every process and placed in the j-th block of the buffer recvbuf.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The type signature associated with sendcount, sendtype, at a process must be equal to the type signature associated with recvcount, recvtype at any other process.

Thus, the outcome of a call to MPI ALLGATHER(...) is as if all processes executed n calls to

     MPI GATHER(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm),

for root = 0, ⋯, n − 1.

MPI ALLGATHERV( sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm)

IN  sendbuf  starting address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice)
IN  recvcounts  integer array (of length group size) containing the num-

ber of elements that are received from each process

IN  displs  integer array (of length group size). Entry i specifies

the displacement (relative to recvbuf at which to place the incoming data from process i

IN  recvtype  data type of receive buffer elements (handle)
IN  comm  communicator (handle)

int MPI Allgatherv(void* sendbuf, int sendcount, MPI Datatype sendtype,
             void* recvbuf, int *recvcounts, int *displs,
             MPI Datatype recvtype, MPI Comm comm)
MPI ALLGATHERV(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNTS, DISPLS,
             RECVTYPE, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, RECVCOUNTS(*), DISPLS(*), RECVTYPE, COMM,
    IERROR

MPI ALLGATHERV can be thought of as MPI GATHERV, but where all processes receive the result, instead of just the root. The j-th block of data sent from each process is received by every process and placed in the j-th block of the buffer recvbuf. These blocks need not all be the same size.

The type signature associated with sendcount, sendtype, at process j must be equal to the type signature associated with recvcounts[j], recvtype at any other process.

The outcome is as if all processes executed calls to

     MPI GATHERV(sendbuf, sendcount, sendtype, recvbuf, displs, recvcounts, recvtype,
     root, comm),

for root = 0, ⋯, n − 1.

For both MPI ALLGATHER and MPI ALLGATHERV, all arguments on all processes are significant. The argument comm must have identical values on all processes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

4.4.7 All-to-All Scatter/Gather

MPI ALLTOALL(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm)

IN  sendbuf  starting address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice)
IN  recvcount  number of elements received from any process (inte-

ger)

IN  recvtype  data type of receive buffer elements (handle)
IN  comm  communicator (handle)

int MPI Alltoall(void* sendbuf, int sendcount, MPI Datatype sendtype,
             void* recvbuf, int recvcount, MPI Datatype recvtype,
             MPI Comm comm)
MPI ALLTOALL(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
             COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, COMM, IERROR

MPI ALLTOALL is an extension of MPI ALLGATHER to the case where each process sends distinct data to each of the receivers. The j-th block sent from process i is received by process j and is placed in the i-th block of recvbuf.

The type signature associated with sendcount, sendtype, at a process must be equal to the type signature associated with recvcount, recvtype at any other process.

The outcome is as if each process executed a send to each process (itself included) with a call to

     MPI Send(sendbuf + i × sendcount × extent(sendtype), sendcount, sendtype, i, ...),

and a receive from every other process with a call to

     MPI Recv(recvbuf + i × recvcount × extent(recvtype), recvcount, i, ...).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI ALLTOALLV(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm)

IN  sendbuf  starting address of send buffer (choice)
IN  sendcounts  integer array equal to the group size specifying the

number of elements to send to each processor

IN  sdispls  integer array (of length group size). Entry j specifies

the displacement (relative to sendbuf from which to take the outgoing data destined for process j

IN  sendtype  data type of send buffer elements (handle)
OUT  recvbuf  address of receive buffer (choice)
IN  recvcounts  integer array equal to the group size specifying the

maximum number of elements that can be received from each processor

IN  rdispls  integer array (of length group size). Entry i specifies

the displacement (relative to recvbuf at which to place the incoming data from process i

IN  recvtype  data type of receive buffer elements (handle)
IN  comm  communicator (handle)

int MPI Alltoallv(void* sendbuf, int *sendcounts, int *sdispls,
             MPI Datatype sendtype, void* recvbuf, int *recvcounts,
             int *rdispls, MPI Datatype recvtype, MPI Comm comm)
MPI ALLTOALLV(SENDBUF, SENDCOUNTS, SDISPLS, SENDTYPE, RECVBUF, RECVCOUNTS,
             RDISPLS, RECVTYPE, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNTS(*), SDISPLS(*), SENDTYPE, RECVCOUNTS(*), RDISPLS(*),
    RECVTYPE, COMM, IERROR

MPI ALLTOALLV adds flexibility to MPI ALLTOALL in that the location of data for the send is specified by sdispls and the location of the placement of the data on the receive side is specified by rdispls.

The j-th block sent from process i is received by process j and is placed in the i-th block of recvbuf. These blocks need not all have the same size.

The type signature associated with sendcount[j], sendtype at process i must be equal to the type signature associated with recvcount[i], recvtype at process j.

The outcome is as if each process sent a message to each other process with,

     MPI Send(sendbuf + displs[i], sendcounts[i], sendtype, i, ...),

and received a message from every other process with a call to

     MPI Recv(recvbuf + displs[i], recvcounts[i], recvtype, i, ...).

For both MPI ALLTOALL and MPI ALLTOALLV, all arguments on all processes are significant. The argument comm must have identical values on all processes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Implementation note: Although the discussion of collective communication in terms of point to point operation implies that each message is transfered directly from sender to receiver, implementations may use a tree communication pattern, where messages are forwarded by intermediate nodes where they are split (for scatter) or concatenated (for gather), if this is more efficient.

4.5 Global Compute Operations

The functions in this section perform one of the following operations across all the members of a group:

global max on integer and floating point data types

global min on integer and floating point data types

global sum on integer and floating point data types

global product on integer and floating point data types

global AND on logical and integer data types

global OR on logical and integer data types

global XOR on logical and integer data types

rank of process with maximum value

rank of process with minimum value

user defined (associative) operation

user defined (associative and commutative) operation

4.5.1 Reduce

MPI REDUCE( sendbuf, recvbuf, count, datatype, op, root, comm)

IN  sendbuf  address of send buffer (choice)
OUT  recvbuf  address of receive buffer (choice, significant only at

root)

IN  count  number of elements in send buffer (integer)
IN  datatype  data type of elements of send buffer (basic types only)

(handle)

IN  op  operation (state)
IN  root  rank of root process (integer)
IN  comm  communicator (handle)

int MPI Reduce(void* sendbuf, void* recvbuf, int count,
         MPI Datatype datatype, MPI Op op, int root, MPI Comm comm)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER COUNT, DATATYPE, OP, ROOT, COMM, IERROR

Combines the values provided in the send buffer of each process in the group, using the operation op, and returns the combined value in the receive buffer of the process with rank root. Each process can provide one value, or a sequence of values, in which case the combine operation is executed element-wise on each entry of the sequence. For example, if the operation is MPI MAX and the send buffer contains two floating point numbers, then recvbuf(1) = global max(sendbuf(1)) and recvbuf(2) = global max(sendbuf(2)). All send buffers should define sequences of equal length of entries all of the same data type, where the type is a basic MPI datatype and one of those allowed for operands of op (see below). For all operations the number of elements in the send buffer are the same as for the receive buffers. For all operations, the type of elements in the send buffer are the same as for the receive buffers.

The operation defined by op is associative and commutative, and the implementation can take advantage of associativity and commutativity in order to change order of evaluation. The routine is called by all group members using the same arguments for count, datatype, op, root and comm.

We list below the supported options for op.

MPI MAX maximum

MPI MIN minimum

MPI SUM sum

MPI PROD product

MPI LAND logical and

MPI BAND bit-wise and

MPI LOR logical or

MPI BOR bit-wise or

MPI LXOR logical xor

MPI BXOR bit-wise xor

MPI MAXLOC maximum value and rank of process with it MPI MINLOC minimum value and rank of process with it

The MPI MINLOC ( MPI MAXLOC) operations return both minimum (maximum) values and the ranks of processes containing those values. The potentially mixed-type nature of the output buffer is a concern, so MPI treats the buffers uniformly and coerces the ranks to the same type as the values. When MPI MINLOC or MPI MAXLOC are invoked, the input buffer should contain m elements of a data type to which the operation MPI MIN or MPI MAX can be applied, followed by space for another m elements of the same type. Internally, the function will coerce the rank of the calling process (an integer type) to the type of the data values, and carry these values along during the reduction. The operation returns at the root the m minimum (or maximum) values, followed by the m ranks of the processes containing these values. To recover the ranks of the processes as integers, the second set of values can now be copied out of the buffer and coerced to integer values.

The operation that defines MPI MAXLOC is

( \begin{matrix}u \\ i \end{matrix} ) ∘ ( \begin{matrix}v \\ j \end{matrix} ) = ( \begin{matrix}w \\ k \end{matrix} )

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

where

w = max(u, v)

and

k = \{ \begin{matrix}i & if u > v \\ min(i, j) & if u = v \\ j & if u < v \end{matrix}

Note that ties are resolved in favor of the process with lower rank and hence this operation is associative and commutative.

MPI MINLOC is defined similarly:

( \begin{matrix}u \\ i \end{matrix} ) ∘ ( \begin{matrix}v \\ j \end{matrix} ) = ( \begin{matrix}w \\ k \end{matrix} )

where

w = min(u, v)

and

k = \{ \begin{matrix}i & if u < v \\ min(i, j) & if u = v \\ j & if u > v \end{matrix}

Operation / Type Compatibility

All of the options for op in MPI REDUCE do not apply for each of the possible MPI basic datatypes. We enumerate the allowed combinations here. First, define groups of MPI basic datatypes in the following way.

C integer: MPI CHAR, MPI INT, MPI LONG, MPI SHORT,
     MPI UNSIGNED SHORT, MPI UNSIGNED, MPI UNSIGNED LONG

Fortran integer: MPI INTEGER

Floating point: MPI FLOAT, MPI DOUBLE, MPI REAL,
     MPI DOUBLE PRECISION
Logical: MPI LOGICAL
Complex: MPI COMPLEX
Byte: MPI BYTE

Now, the valid datatypes for each option is specified below.

MPI MAX, MPI MIN, MPI MAXLOC, MPI MINLOC C integer, Fortran integer, Floating point MPI SUM, MPI PROD C integer, Fortran integer, Floating point, Com-

plex

MPI LAND, MPI LOR, MPI LXOR C integer, Logical

MPI BAND, MPI BOR, MPI BXOR C integer, Byte

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Example of Reduce

Each process has an array of 30 doubles, in C. For each of the 30 locations, compute the value and rank of the process containing the largest value.

...
/* each process has an array of 30 double: a[]
 */
double a[30];
double in[60],out[60];
int i,outranks[30];
for (i=0; i<30; ++i) {
    in[i] = a[i];
}
MPI_Reduce( in, out, 30, MPI_DOUBLE, MPI_MAXLOC, root, comm );
/* At this point, the answer resides on process root
 */
if (myrank == root) {
    /* read ranks out and coerce back to ints
     */
    for (i=0; i<30; ++i) {
        outranks[i] = out[30+i];
    }
}

4.5.2 User-Reduce

MPI USER REDUCE( sendbuf, recvbuf, count, datatype, function, root, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice, significant

only at root)

IN  count  number of elements in input buffer (integer)
IN  datatype  data type of elements of input buffer (handle)
IN  function  user defined function (function)
IN  root  rank of root process (integer)
IN  comm  communicator (handle)

int MPI User reduce(void* sendbuf, void* recvbuf, int count,
         MPI Datatype datatype, void function, int root, MPI Comm comm)
MPI USER REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, ROOT, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Similar to the reduce operation function above except that a user supplied function is used. function is a function with three arguments. A C prototype for such function is void f( invec, inoutvec, *len). Both invec and inoutvec are arrays with *len entries. The function computes element-wise a commutative and associative operation on each pair of entries and returns the result in inoutvec. A pseudo-code for function is given below, where op is the commutative and associative operation defined by function.

            for(i=0; i < *len; i++) {
                    inoutvec[i] op= invec[i]
            }

The type of the elements of invec and of inoutvec match the type of the elements of the send buffers and the receive buffer. No MPI functions may be called inside the user defined function.

The addition of the third argument, *len, in function allows the system to avoid calling function for each element in the input buffer. Rather, the system can choose to apply function to chunks of input.

Example of User-Reduce

Compute the product of an array of complex numbers, in C.

typedef struct _complex {
    double real,imag;
} Complex;
/* the user-defined function
 */
void myProd( Complex invec[], Complex inoutvec[], int *len )
{
    int i;
    Complex c;
    for (i=0; i< *len; ++i) {
        c.real = inoutvec[i].real*invec[i].real -
                   inoutvec[i].imag*invec[i].imag;
        c.imag = inoutvec[i].real*invec[i].imag +
                   inoutvec[i].imag*invec[i].real;
        inoutvec[i] = c;
    }
}
/* and, to call it...
 */
...
    /* each process has an array of 100 Complexes
     */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Complex a[100], answer[100];
MPI_Datatype ctype;
/* explain to MPI what a "complex" is (remember, this is C!)
 * Note we are assuming compiler implements type Complex as
 * two contiguous doubles
 */
MPI_Type_contiguous( 2, MPI_DOUBLE, &ctype );
MPI_Type_commit( ctype );
MPI_User_reduce( a, answer, 100, ctype, myProd, root, comm );
/* At this point, the answer, which consists of 100 Complexes,
 * resides on process root
 */

MPI USER REDUCEA( sendbuf, recvbuf, count, datatype, function, root, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice, significant

only at root)

IN  count  number of elements in input buffer (integer)
IN  datatype  data type of elements of input buffer (handle)
IN  function  user defined function (function)
IN  root  rank of root process (integer)
IN  comm  communicator (handle)

int MPI User reducea(void* sendbuf, void* recvbuf, int count,
         MPI Datatype datatype, void function, int root, MPI Comm comm)
MPI USER REDUCEA(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, ROOT, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR

Identical to MPI USER REDUCE, except that the operation defined by function is not required to be commutative, but only associative. Thus, the order of computation can be modified only using associativity. Use of this function means that the implementation cannot try and change the order of computation of the reduce by assuming commutativity.

Implementation note: MPI USER REDUCEA and MPI USER REDUCE can have identical implementations, if one does

not wish to take advantage of commutativity.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

4.5.3 All-Reduce

MPI includes variants of each of the reduce operations where the result is known to all processes in the group on return. MPI requires that all processes participating in any of the all-reduce operations receive exactly identical results.

MPI ALLREDUCE( sendbuf, recvbuf, count, datatype, op, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  data type of elements of send buffer (handle)
IN  op  operation (state)
IN  comm  communicator (handle)

int MPI Allreduce(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, MPI Op op, MPI Comm comm)
MPI ALLREDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER COUNT, DATATYPE, OP, COMM, IERROR

Same as the MPI REDUCE operation function except that the result appears in the receive buffer of all the group members.

MPI USER ALLREDUCE( sendbuf, recvbuf, count, datatype, function, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  data type of elements of send buffer (handle)
IN  function  user defined function (function)
IN  comm  communicator (handle)

int MPI User allreduce(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, void function, MPI Comm comm)
MPI USER ALLREDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM,
             IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    EXTERNAL FUNCTION
    INTEGER COUNT, DATATYPE, COMM, IERROR

Same as the MPI USER REDUCE operation function except that the result appears in the receive buffer of all the group members.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI USER ALLREDUCEA( sendbuf, recvbuf, count, datatype, function, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  data type of elements of send buffer (handle)
IN  function  user defined function (function)
IN  comm  communicator (handle)

int MPI User allreducea(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, void function, MPI Comm comm)
MPI USER ALLREDUCEA(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM,
             IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    EXTERNAL FUNCTION
    INTEGER COUNT, DATATYPE, COMM, IERROR

Same as MPI USER REDUCEA, except that the result appears in the receive buffer of all the group members.

Implementation note: The all-reduce operations can be implemented as a reduce, followed by a broadcast. However, a direct implementation can lead to better performance.

4.5.4 Reduce-Scatter

MPI also includes variants of each of the reduce operations where the result is scattered to all processes in the group on return.

MPI REDUCE SCATTER( sendbuf, recvbuf, recvcounts, datatype, op, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  recvcounts  integer array specifying the number of elements in re-

sult distributed to each process. Array must be identical on all calling processes.

IN  datatype  data type of elements of input buffer (handle)
IN  op  operation (state)
IN  comm  communicator (handle)

int MPI Reduce scatter(void* sendbuf, void* recvbuf, int *recvcounts,
             MPI Datatype datatype, MPI Op op, MPI Comm comm)
MPI REDUCE SCATTER(SENDBUF, RECVBUF, RECVCOUNTS, DATATYPE, OP, COMM,
             IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

98 CHAPTER 4. COLLECTIVE COMMUNICATION

  <type> SENDBUF(*), RECVBUF(*)
  INTEGER RECVCOUNTS, DATATYPE, OP, COMM, IERROR

MPI REDUCE SCATTER first does a componentwise reduction on vectors provided by the processes. Next, the resulting vector of results is split into n disjoint segments, where n is the number of members in the group; segment i contains recvcounts[i] elements. The i-th segment is sent to process with rank i.

Implementation note: The MPI REDUCE SCATTER routine is functionally equivalent to: A MPI REDUCE operation function with count equal to the sum of recvcounts[i] followed by MPI SCATTERV with sendcounts equal to recvcounts. However, a direct implementation may run faster.

MPI USER REDUCE SCATTER( sendbuf, recvbuf, recvcnts, type, function, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  recvcnts  integer array specifying the number of elements in re-

sult distributed to each process. Array must be identical on all calling processes.

IN  type  data type of elements of input buffer (handle)
IN  function  user defined function (function)
IN  comm  communicator (handle)

Same as the MPI REDUCE SCATTER operation function except that the user specifies the reduction operation as in MPI USER REDUCE.

MPI USER REDUCE SCATTERA( sendbuf, recvbuf, recvcnts, type, function, comm)

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  recvcnts  integer array specifying the number of elements in re-

sult distributed to each process. Array must be identical on all calling processes.

IN  type  data type of elements of input buffer (handle)
IN  function  user defined function (function)
IN  comm  communicator (handle)

The “A” version of MPI USER REDUCE SCATTER.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

4.5.5 Scan

MPI SCAN( sendbuf, recvbuf, count, datatype, op, comm )

IN  sendbuf  starting address of send buffer (choice)
OUT  recvbuf  starting address of receive buffer (choice)
IN  count  number of elements in input buffer (integer)
IN  datatype  data type of elements of input buffer (handle)
IN  op  operation (state)
IN  comm  communicator (handle)

int MPI Scan(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, MPI Op op, MPI Comm comm )
MPI SCAN(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    INTEGER COUNT, DATATYPE, OP, COMM, IERROR

MPI SCAN is used to perform a parallel prefix with respect to an associative and commutative reduction operation on data distributed across the group. The operation returns in the receive buffer of the process with rank i the reduction of the values in the send buffers of processes with ranks 0,...,i. The type of operations supported, their semantics, and the constraints on send and receive buffers are as for MPI REDUCE.

MPI USER SCAN( sendbuf, recvbuf, count, datatype, function, comm)

IN  sendbuf  address of input buffer
OUT  recvbuf  address of output buffer
IN  count  number of elements in input and output buffer (inte-

ger)

IN  datatype  data type of buffer (handle)
IN  function  user provided function
IN  comm  communicator (handle)

int MPI User scan(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, void function, MPI Comm comm)
MPI USER SCAN(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    EXTERNAL FUNCTION
    INTEGER COUNT, DATATYPE, COMM, IERROR

Same as the MPI SCAN operation function except that a user supplied function is used. function is an associative and commutative function with an input vector, an inout vector, and a length argument. The types of the two vectors and of the returned values all agree. See MPI USER REDUCE for more details.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI USER SCANA( sendbuf, recvbuf, count, datatype, function, comm)

IN  sendbuf  address of input buffer (choice)
OUT  recvbuf  address of output buffer (choice)
IN  count  number of elements in input and output buffer (inte-

ger)

IN  datatype  data type of buffer (handle)
IN  function  user defined function (function)
IN  comm  communicator (handle)

int MPI User scana(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, void function, MPI Comm comm)
MPI USER SCANA(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM, IERROR)
    <type> SENDBUF(*), RECVBUF(*)
    EXTERNAL FUNCTION
    INTEGER COUNT, DATATYPE, COMM, IERROR

Same as MPI USER SCAN, except that the user-defined operation need not be commutative.

Implementation note:

MPI USER SCAN could be implemented as MPI USER SCANA, though for some architectures it may be possible to arrive at a faster MPI USER SCAN by taking advantage of commutativity.

4.6 Correctness

4.6.1 Synchronization Side-Effects

A correct program should invoke collective communications so that deadlock will not occur, whether collective communication is synchronizing or not. The following examples illustrate dangerous use of collective routines.

The following example illustrates an erroneous program.

/* Example A */
switch(rank)
   {
   case 0: { MPI_Bcast(var1, count, type, 0, comm);
             MPI_Send(var2, count, type, 1, tag, comm);
             break;
           }
   case 1: { MPI_Recv(var2, count, type, 0, tag, comm);
             MPI_Bcast(var1, count, type, 0, comm);
             break;
           }
   }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Process zero executes a broadcast, followed by a blocking send operation; process one first executes a matching blocking receive, followed by the matching broadcast call. This program may deadlock. The broadcast call on process zero may block until process one executes the matching broadcast call, so that the send is not executed. Process one will definitely block on the receive and so, in this case, never executes the broadcast.

The following example is correct, but non-deterministic:

/* Example B */
switch(rank)
   {
    case 0: { MPI_Bcast(var1, count, type, 0, comm);
              MPI_Send(var2, count, type, 1, tag, comm);
              break;
            }
    case 1: { MPI_Recv(var2, count, type, MPI_ANY_SOURCE, tag, comm);
              MPI_Bcast(var1, count, type, 0, comm);
              MPI_Recv(var2, count, type, MPI_ANY_SOURCE, tag, comm);
              break;
            }
    case 2: { MPI_Send(var2, count, type, 1, tag, comm);
              MPI_Bcast(var1, count, type, 0, comm);
              break;
            }
    }

All three processes participate in a broadcast. Process 0 sends a message to process 1 after the broadcast, and process 2 sends a message to process 1 before the broadcast. Process 1 receives before and after the broadcast, with a wildcard source argument.

Two possible executions, with different matchings of sends and receives, are illustrated in figure 4.10.

Note that the second execution has the peculiar effect that a send executed after the broadcast is received at another node before the broadcast. This example illustrates the fact that one should not rely on collective communication functions to have particular synchronization effects. To assume that collective communication functions do or do not have certain synchronizing side-effects is non-portable.

4.6.2 Multiple Calls to Collective

It is the user’s responsibility to make sure that there are no two concurrently executing collective calls that use the same communicator on the same process. Since all collective communication calls are blocking this restriction only affects multithreaded implementations. On the other hand, it is legitimate for one process to start a new collective communication call even though a previous call that uses the same communicator has not yet terminated on another process. As illustrated in the following example:

/* Example C */
 MPI_Bcast(var1, count, type, 0, comm);
 MPI_Bcast(var2, count, type, 1, comm);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 4.10: A race condition causes non-deterministic matching of sends and receives. One cannot rely on synchronization from a broadcast to make the program deterministic.

In a nonsynchronizing implementation of broadcast, process zero may start executing the second broadcast before process one terminated the first broadcast. Both process zero and one may terminate their two broadcast calls before other processes have started their calls. It is the implementor’s responsibility to ensure this will not cause any error.

Implementation note:

Assume that broadcast is implemented using point-to-point MPI communication. The following two rules are satisfied:

1. All receives specify their source explicitly (no wildcards).

2. Each process sends all messages that pertain to one collective call before sending any message that pertain to a subsequent collective call.

Then messages belonging to successive broadcasts cannot be confused, as the order of pointto-point messages is preserved. This is true, in general, for any collective library.

A collective communication may execute in a context while point-to-point communications that use the same context are pending, or occur concurrently. This is illustrated in example B above, the first process may receive a message sent with the context of communicator comm while it is executing a broadcast with the same communicator. It is the implementor’s responsibility to ensure this will not cause any confusion.

Implementation note: Assume that collective communications are implemented using point-to-point MPI communication. Then, in order to avoid confusion, whenever a communicator is created, a “hidden communicator” need be created for collective communication. A direct implementation of MPI collective communication can achieve a similar effect more cheaply, e.g., by

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

4.6. CORRECTNESS 103

using a hidden tag or context bit to indicate whether the communicator is used for point-to-point or collective communication.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 5

Groups, Contexts, Communicators, and Caching

5.1 Introduction

We begin this chapter with a perspective of why MPI has included features that are not found in existing vendor systems, and why these are important. One of the main motivations for the features described here is to allow for the systematic development of parallel libraries.

5.1.1 Why we need libraries

Parallel libraries are needed to encapsulate the distracting complications associated with distributed realizations of key algorithms. They ensure consistent correctness of such procedures, and provide a “higher level” of portability than MPI itself can provide. As such, libraries prevent each programmer from repeating the work of defining consistent data structures, data layouts, and methods that implement key algorithms (such as matrix operations). Since the best libraries come with several variations on parallel systems (different data layouts, different strategies dependent on size of system, or type of floating point), this too needs to be hidden from the user.

5.1.2 Why existing systems do not support libraries

It is difficult to write libraries with message passing systems that vendors produce, because they have no way for the library to insulate itself effectively from the other point-to-point message passing that goes on in the system. For instance, message tags are a means for a library to hide its messages from the user. However, many libraries may choose the same tags, or the vendor’s library may itself indiscriminately use tags when implementing collective operations (e.g., global sum); in that case, collective and point-to-point messages may get mixed up.

Collective operations, such as reduce, broadcast, and so on, are not defined in all vendor systems, and when defined are most often only available over the full set of processors (or processes) allocated to the user. Libraries need to work on different collections of processes, and to have available collective operations that only impact the processes that care about the collective operation.

A third issue is that libraries don’t want to describe point-to-point communication in terms of low-level, hardware-dependent names. In fact, they might like to use virtual

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

names for processes that somehow reflect the relationship of processes to one another in the distributed data structure natural to an algorithm. Libraries (or program modules) that execute concurrently on different processes may want to use the same process names for their internal communication.

5.1.3 What features are needed to support libraries

So, we have established three key areas that will support the creation of libraries:

• Safe communication space, that guarantees that a library can communicate as it needs to,

• Group scope for collective operations, that allow libraries to avoid unnecessarily synchronizing uninvolved processors,

• Abstract process naming to allow libraries to describe their communication in terms suitable to their own data structures.

5.1.4 What these features are called in MPI

The features that implement these important concepts are as follows, in MPI:

• Contexts provide the ability to have separate safe “universes” of message-passing in MPI. A context is something like an additional tag that differentiates messages, but the system manages it, and the user never manipulates contexts. We say that communicators (below) are a specific context of communication, when we are speaking loosely. The use of separate communication contexts by distinct libraries (or distinct library invocations) will insulate communication internal to the library execution from external communication. This allows the invocation of the library even if there are pending communications, and avoids the need to synchronize entry or exit into library code.

• Groups define an ordered collection of processes, each with a rank, and it is this group that defines the low-level names for inter-process communication (ranks are used for sending and receiving). Thus, groups define a scope for process names in point to point communication. In addition, groups define the scope of collective operations.

• Communicators (see [13]) encapsulate all of these ideas in an object that provides the right scope for all communication operations in MPI.

The current practice in many communication libraries is that there is a a unique, predefined communication universe that includes all processes available when the library is initiated; the processes are assigned consecutive ranks. Participants in a point-to-point communication are identified by their rank; a collective communication (such as broadcast) involves all processes. This practice can be followed in MPI by using the predefined communicator MPI COMM WORLD. Users who are satisfied with such practice can plug in MPI COMM WORLD wherever a communicator argument is required, and can consequently disregard the rest of this chapter.

The discussion has dealt so far with intra-communication: communication within a group. MPI also supports inter-communication: communication across groups. When an application is built by composing several parallel modules, it is convenient to allow one

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

module to communicate with the other using local ranks within the second module for addressing. This is especially convenient in a client-server computing paradigm, where either client or server are parallel. The support of inter-communication also provides a mechanism for the extension of MPI to a dynamic model where not all processes are preallocated at initialization time. In such a situation, it becomes necessary to support communication across “universes.” Inter-communication is supported by objects called inter-communicators. These bind two groups together with a context shared by both groups. MPI provides mechanisms for creating and manipulating inter-communicators. They are used in point-to-point communication in the same manner as regular intra-communicators. Users who do not need inter-communication in their applications can safely ignore this extension.

Communicators also provide a “caching” mechanism that allows one to associate new attributes with communicators, on a par with MPI built-in features. This can be used by advanced users to further adorn communicators, and by MPI to implement some communicator functions. For example, the virtual topology functions described in Chapter 6 are likely to be supported this way.

5.2 Basic Concepts

5.2.1 Groups

A group is an ordered set of process identifiers (henceforth processes); process identifiers are implementation-dependent objects. Each process in a group is associated with an integer rank, starting from zero. Groups are represented by opaque group objects, and hence cannot be directly transferred from one process to another. A group is used within a communicator to describe the participants in a communication “universe” and to rank such participants (thus giving them unique names within that “universe” of communication).

Implementation note: A group may be represented by a virtual-to-real process-address-translation table. Each communicator object would have a pointer to such a table.

5.2.2 Contexts

A context is the MPI mechanism for partitioning communication space. A defining property of a context is that a message sent in a context cannot be received in another context. Contexts are not explicit MPI objects; they appear only along with groups in communicators. Distinct communicators in the same process have distinct contexts. A context is essentially a tag (or tags) needed to make a communicator safe for point-to-point and MPI-defined collective communication.

Implementation note: A possible implementation is for a context to be a label attached to messages and matched on receive. Each intra-communicator stores the value of that label. Communicator generating functions use a collective communication to agree on a new group-wide unique context. In inter-communication, it is more convenient to use two labels, one used by group A to send and group B to receive, and the second used by group B to send and group A to receive. Since contexts are not explicit objects, other implementations are possible.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

5.2.3 Communicators

Communicators bring together the concepts of group and context. Furthermore, to support implementation-specific optimizations, and virtual topologies, they “cache” additional information. The source and destination of a message is identified by the rank of that process within the group. For collective communication, the communicator specifies the set of processes that participate in the collective operation and their order, when significant. Thus, the communicator restricts the “spatial” scope of communication, and provides local process addressing.

Communicators are represented by opaque communicator objects, and hence cannot be directly transferred from one process to another.

5.2.4 Predefined Communicators

An initial communicator MPI COMM WORLD of all processes the local process can communicate with after initialization (itself included) is defined once MPI INIT has been called.

In a static-process-model implementation of MPI, all processes that participate in the computation are available after MPI is initialized. For this case, MPI COMM WORLD is a communicator of all processes available for the computation; this communicator has the same value in all processes. In an implementation of MPI where processes can dynamically join an MPI execution, it may be the case that a process starts an MPI computation without having access to all other processes. In such situations, MPI COMM WORLD is a communicator incorporating all processes with which the joining process can immediately communicate. Therefore, MPI COMM WORLD may simultaneouly have different values in different processes.

All MPI implementations are required to provide the MPI COMM WORLD communicator. It cannot be deallocated during the life of a process. The group corresponding to this communicator does not appear as a pre-defined constant, but it may be accessed using MPI COMM GROUP (see below). MPI does not specify the correspondence between the process rank in MPI COMM WORLD and its (machine-dependent) absolute address. Neither does MPI specify the function of the host process, if any. An inquiry function MPI GET HOST is provided in Chapter 7 so that MPI programs can determine if there is a host process, and find its rank in MPI COMM WORLD if it should exist. Other implementation-dependent, predefined communicators may also be provided.

5.3 Group Management

This section describes the manipulation of process groups in MPI. These operations are local and their execution do not require interprocess communication.

5.3.1 Group Accessors

MPI GROUP SIZE(group, size)

IN  group  group (handle)
OUT  size  number of processes in the group (integer)

int MPI Group size(MPI Group group, int *size)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GROUP SIZE(GROUP, SIZE, IERROR)
    INTEGER GROUP, SIZE, IERROR

MPI GROUP RANK(group, rank)

IN  group  group (handle)
OUT  rank  rank of the calling process in group, or

MPI UNDEFINED if the process is not a member (integer)

int MPI Group rank(MPI Group group, int *rank)
MPI GROUP RANK(GROUP, RANK, IERROR)
    INTEGER GROUP, RANK, IERROR

MPI GROUP TRANSLATE RANKS (group1, n, ranks1, group2, ranks2)

IN  group1  group1 (handle)
IN  n  number of ranks in ranks1 and ranks2 arrays (integer)
IN  ranks1  array of zero or more valid ranks in group1
IN  group2  group2 (handle)
OUT  ranks2  array of corresponding ranks in group2,

MPI UNDEFINED when no correspondence exists.

int MPI Group translate ranks (MPI Group group1, int n, int *ranks1,
             MPI Group group2, int *ranks2)
MPI GROUP TRANSLATE RANKS(GROUP1, N, RANKS1, GROUP2, RANKS2, IERROR)
    INTEGER GROUP1, N, RANKS1(*), GROUP2, RANKS2(*), IERROR

5.3.2 Group Constructors

These functions allow construction of new groups from existing groups. These are local operations, and distinct groups may be defined on different processes; a process may also define a group that does not include itself. Consistent definitions are required when groups are used as arguments in communicator-building functions. MPI does not provide a mechanism to build a group from scratch, but only from other, previously defined groups. The base group, upon which all other groups are defined, is the group associated with the initial communicator MPI COMM WORLD (accessible through the function MPI COMM GROUP).

MPI GROUP UNION(group1, group2, newgroup)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  newgroup  union group (handle)

int MPI Group union(MPI Group group1, MPI Group group2, MPI Group *newgroup)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GROUP UNION(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR

MPI GROUP INTERSECTION(group1, group2, newgroup)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  newgroup  intersection group (handle)

int MPI Group intersection(MPI Group group1, MPI Group group2,
             MPI Group *newgroup)
MPI GROUP INTERSECTION(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR

MPI GROUP DIFFERENCE(group1, group2, newgroup)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  newgroup  difference group (handle)

int MPI Group difference(MPI Group group1, MPI Group group2,
             MPI Group *newgroup)
MPI GROUP DIFFERENCE(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR

The set-like operations are defined as follows:

union All elements of the first group (group1), followed by all elements of second group (group2) not in first.

intersect all elements of the first group that are also in the second group, ordered as in first group.

difference all elements of the first group that are not in the second group, ordered as in the first group.

Note that for these operations the order of processes in the output group is determined primarily by order in the first group (if possible) and then, if necessary, by order in the second group. Neither union nor intersection are commutative, but both are associative.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GROUP INCL(group, n, ranks, newgroup)

IN  group  group (handle)
IN  n  number of elements in array ranks (and size of new-

group) (integer)

IN  ranks  ranks of processes in group to appear in newgroup (ar-

ray of integers)

OUT  newgroup  new group derived from above, in the order defined by

ranks (handle)

int MPI Group incl(MPI Group group, int n, int *ranks, MPI Group *newgroup)
MPI GROUP INCL(GROUP, N, RANKS, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR

The function MPI GROUP INCL creates a group newgroup that consists of the n processes in group with ranks rank[0],…, rank[n-1]; the process with rank i in newgroup is the process with rank ranks[i] in group. Each of the n elements of ranks must be a valid rank in group and all elements must be distinct, or else the program is erroneous. If n = 0, then newgroup is empty. This function can, for instance, be used to reorder the elements of a group.

MPI GROUP EXCL(group, n, ranks, newgroup)

IN  group  group (handle)
IN  n  number of elements in array ranks (integer)
IN  ranks  array of integer ranks in group not to appear in new-

group

OUT  newgroup  new group derived from above, preserving the order

defined by group (handle)

int MPI Group excl(MPI Group group, int n, int *ranks, MPI Group *newgroup)
MPI GROUP EXCL(GROUP, N, RANKS, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR

The function MPI GROUP EXCL creates a group of processes newgroup that is obtained by deleting from group those processes with ranks ranks[0] ,… ranks[n-1]. The ordering of processes in newgroup is identical to the ordering in group. Each of the n elements of ranks must be a valid rank in group and all elements must be distinct; otherwise, the program is erroneous. If n = 0, then newgroup is identical to group.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GROUP RANGE INCL(group, n, ranges, newgroup)

IN  group  group (handle)
IN  n  number of triplets in array ranges (integer)
IN  ranges  a one-dimensional array of integer triplets, of the form

(first rank, last rank, stride) indicating ranks in group or processes to be included in newgroup

OUT  newgroup  new group derived from above, in the order defined by

ranges (handle)

int MPI Group range incl(MPI Group group, int n, int **ranges,
             MPI Group *newgroup)
MPI GROUP RANGE INCL(GROUP, N, RANGES, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANGES(*,*), NEWGROUP, IERROR

If ranges consist of the triplets

(first_{1}, last_{1}, stride_{1}), ..., (first_{n}, last_{n}, stride_{n})

then newgroup consists of the sequence of processes in group with ranks

first_{1}, first_{1} + stride_{1}, ..., first_{1} + ⌊\frac{last_{1} − first_{1}}{stride_{1}} ⌋ stride_{1}, ...

first_{n}, first_{n} + stride_{n}, ..., first_{n} + ⌊\frac{last_{n} − first_{n}}{stride_{n}} ⌋ stride_{n}.

Each computed rank must be a valid rank in group and all computed ranks must be distinct, else the program is erroneous. Note that we may have first_{i} > last_{i}, and stride_{i} may be negative, but cannot be zero.

The functionality of this routine is specified to be equivalent to expanding the array of ranges to an array of the included ranks and passing the resulting array of ranks and other arguments to MPI GROUP INCL. A call to MPI GROUP INCL is equivalent to a call to MPI GROUP RANGE INCL with each rank i in ranks replaced by the triplet (i,i,1) in the argument ranges.

MPI GROUP RANGE EXCL(group, n, ranges, newgroup)

IN  group  group (handle)
IN  n  number of elements in array ranks (integer)
IN  ranges  a one-dimensional array of integer triplets of the form

(first rank, last rank, stride), indicating the ranks in group of processes to be excluded from the output group newgroup.

OUT  newgroup  new group derived from above, preserving the order

in group (handle)

int MPI Group range excl(MPI Group group, int n, int **ranges,
             MPI Group *newgroup)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GROUP RANGE EXCL(GROUP, N, RANGES, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANGES(*,*), NEWGROUP, IERROR

Each computed rank must be a valid rank in group and all computed ranks must be distinct, or else the program is erroneous.

The functionality of this routine is specified to be equivalent to expanding the array of ranges to an array of the excluded ranks and passing the resulting array of ranks and other arguments to MPI GROUP EXCL. A call to MPI GROUP EXCL is equivalent to a call to MPI GROUP RANGE EXCL with each rank i in ranks replaced by the triplet (i,i,1) in the argument ranges.

Note: the range operations do not explicitly enumerate ranks, and therefore are more scalable if implemented efficiently. Hence, we recommend MPI programmers to use them whenenever possible, as high-quality implementations will take advantage of this fact.

5.3.3 Group Destructors

MPI GROUP FREE(group)

INOUT group group (handle)

int MPI Group free(MPI Group *group)
MPI GROUP FREE(GROUP, IERROR)
    INTEGER GROUP, IERROR

This operation marks a group object for deallocation. The handle group is set to null by the call. The group object is actually deallocated only if there are no other active references to it. If a group has been bound to one or more communicators, then the user has to free all “copies” of the group.

Implementation note:

One can keep a reference count that is incremented for each call to MPI COMM MAKE and MPI COMM DUP, and and decremented for each call to MPI GROUP FREE or MPI COMM FREE; the group object is ultimately deallocated when the reference count drops to zero.

5.4 Communicator Management

This section describes the manipulation of communicators in MPI. Operations that access communicators are local and their execution does not require interprocess communication. Operations that create communicators are collective and may require interprocess communication. (High quality implementations will be able to amortize these overheads over several calls.)

5.4.1 Communicator Accessors

The following are all local operations.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI COMM SIZE(comm, size)

IN  comm  communicator (handle)
OUT  size  number of processes in the group of comm (integer)

int MPI Comm size(MPI Comm comm, int *size)
MPI COMM SIZE(COMM, SIZE, IERROR)
    INTEGER COMM, SIZE, IERROR

MPI COMM RANK(comm, rank)

IN  comm  communicator (handle)
OUT  rank  rank of the calling process in group of comm (integer)

int MPI Comm rank(MPI Comm comm, int *rank)
MPI COMM RANK(COMM, RANK, IERROR)
    INTEGER COMM, RANK, IERROR

MPI COMM GROUP(comm, group)

IN  comm  communicator (handle)
OUT  group  group corresponding to comm (handle)

int MPI Comm group(MPI Comm comm, MPI Group *group)
MPI COMM GROUP(COMM, GROUP, IERROR)
    INTEGER COMM, GROUP, IERROR

5.4.2 Communicator Constructors

The following are collective functions that are invoked by all processes in the group associated with comm.

Note that there is a chicken-and-egg aspect to MPI in that a communicator is needed to create a new communicator. The base communicator for all MPI communicators is predefined outside of MPI, and is MPI COMM WORLD (defined above).

MPI COMM DUP(comm, newcomm)

IN  comm  communicator (handle)
OUT  newcomm  copy of comm (handle)

int MPI Comm dup(MPI Comm comm, MPI Comm *newcomm)
MPI COMM DUP(COMM, NEWCOMM, IERROR)
    INTEGER COMM, NEWCOMM, IERROR

MPI COMM DUP Duplicates the existing communicator comm with cached attributes copied or not copied (according to the copy-callback policy of each individual cached at-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

114 CHAPTER 5. GROUPS, CONTEXTS, COMMUNICATORS, AND CACHING

tribute). Returns in newcomm a new communicator with the same group, same cached information, but a new context.

Implementation note: One need not actually copy the group information, but only add a new reference and increment the reference count. Copy on write can be used for the cached information.

MPI COMM MAKE(comm, group, newcomm)

IN  comm  communicator (handle)
IN  group  Group, which is a subset of the group of comm (han-

dle)

OUT  newcomm  new communicator (handle)

int MPI Comm make(MPI Comm comm, MPI Group group, MPI Comm *newcomm)
MPI COMM MAKE(COMM, GROUP, NEWCOMM, IERROR)
    INTEGER COMM, GROUP, NEWCOMM, IERROR

This function creates a new communicator newcomm with communication group defined by group and a new context. No cached information propagates from comm to newcomm. The function returns a null handle to processes that are not in group.

The call is erroneous if not all group arguments have the same value, or if group is not a subset of the group associated with comm. Note that the call is to be executed by all processes in comm, even if they do not belong to the new group.

Implementation note: Since all processes calling MPI COMM DUP or MPI COMM MAKE provide the same group argument, it is theoretically possible to agree on a group-wide unique context with no communication. However, local execution of these functions requires use of a larger context name space and reduces error-checking. Implementations may strike various compromises between these conflicting goals. If new communicators are created without synchronizing the processes involved then the commucation system should be able to cope with messages arriving in a context that has not yet been allocated at the receiving process.

MPI COMM SPLIT(comm, color, key, newcomm)

IN  comm  communicator (handle)
IN  color  control of subset assignment (integer)
IN  key  control of rank assigment (integer)
OUT  newcomm  new communicator (handle)

int MPI Comm split(MPI Comm comm, int color, int key, MPI Comm *newcomm)
MPI COMM SPLIT(COMM, COLOR, KEY, NEWCOMM, IERROR)
    INTEGER COMM, COLOR, KEY, NEWCOMM, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

This function partitions the group associated with comm into disjoint subgroups, one for each value of color. Each subgroup contains all processes of the same color. Within each subgroup, the processes are ranked in the order defined by the value of the argument key, with ties broken according to their rank in the old group. A new communicator is created for each subgroup and returned in newcomm. A process may supply the color value MPI UNDEFINED, in which case newcomm returns a null communicator handle. This is a collective call, but each process can provide different values for color and key.

A call to MPI COMM MAKE(comm, group, newcomm) is equivalent to a call to MPI COMM SPLIT(comm, color, key, newcomm), where all members of group provide color = 0 and key = rank in group, and all processes that are not members of group provide color = MPI UNDEFINED. The function MPI COMM SPLIT allows more general partitioning of a group into one or more subgroups with optional reordering.

5.4.3 Communicator Destructors

MPI COMM FREE(comm)

INOUT comm communicator to be destroyed (handle)

int MPI Comm free(MPI Comm *comm)
MPI COMM FREE(COMM, IERROR)
    INTEGER COMM, IERROR

This collective operation marks the communication object for deallocation. The handle is set to null. The object is actually deallocated only if there are no other active references to it. The associated group may also be deallocated, if there are no active references to it. It is the user responsibility to free each copy of a communicator object.

Implementation note: A reference count mechanism can be used: the reference count is incremented by each call to MPI COMM DUP, and decremented by each call to MPI COMM FREE. The object is ultimately deallocated when the count reaches zero.

Though collective, it is anticipated that this operation will normally be local, though the debugging version of an MPI library might choose to synchronize.

5.5 Inter-Communication

This section introduces the concept of inter-communication and describes the portions of MPI that support it. It describes support for writing programs that contain user-level servers. It also describes a name service that simplifies writing programs containing intercommunication.

All point-to-point communication described thus far has involved communication between processes that are members of the same group. This type of communication is called “intra-communication” and the communicator used is called an “intra-communicator.”

In modular and multi-disciplinary applications, different process groups execute distinct modules and processes within different modules communicate with one another in a pipeline or a more general module graph. In these applications, the most natural way for a process

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

to specify a target process is by the rank of the target process within the target group. In applications that contain internal user-level servers, each server may be a process group that provides services to one or more clients, and each client may be a process group that uses the services of one or more servers. It is again most natural to specify the target process by rank within the target group in these applications. This type of communication is called “inter-communication” and the communicator used is called an “inter-communicator.”

An inter-communication operation is a point-to-point communication between processes in different groups. The group containing a process that initiates an inter-commun-ication operation is called the “local group,” that is, the sender in a send and the receiver in a receive. The group containing the target process is called the “remote group,” that is, the receiver in a send and the sender in a receive. As in intra-communication, the target process is specified using a (communicator, rank) pair. Unlike intra-communication, the rank is relative to a second, remote group.

Local and remote groups of an inter-communicator are not required to be disjoint. An intra-communicator can be seen as a particular case of an inter-communicator, where local and remote groups happen to be identical. The semantics of accessor function are defined so as to be consistent with this interpretation. Here is a summary of the properties of inter-communication and inter-communicators:

• The syntax of point-to-point communication is the same for both inter- and intracommunication. The same communicator can be used both for send and for receive operations.

• A target process is addressed by its rank in the remote group, both for sends and for receives.

• Communications using an inter-communicator are guaranteed not to conflict with any communications that use a different communicator.

• An inter-communicator cannot be used for collective communication.

• A communicator will provide either intra- or inter-communication, never both.

The routine MPI COMM TEST INTER may be used to determine if a communicator is an inter- or intra-communicator. Inter-communicators can be used as arguments to some of the other communicator access routines. Inter-communicators cannot be used as input to any of the constructor routines for intra-communicators.

Implementation note: Communicators can be represented in each process by a tuple consisting of:

group

send context

receive context

source

For inter-communicators, group describes the remote group, and source is the rank of the process in the local group. For intra-communicators, group is the communicator group (remote=local), source is the rank of the process in this group, and send context and receive context are identical. A group is represented by a rank-to-absolute-address translation table.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The inter-communicator cannot be discussed sensibly without considering processes in both the local and remote groups. Imagine a process P in group P, which has an inter-communicator C_{P}, and a process Q in group Q, which has an inter-communicator C_{Q}. (Note that P and Q do not have to be distinct.) Then

• C_{P}.group describes the group Q and C_{Q}.group describes the group P.

• C_{P}.send context = C_{Q}.receive context and the context is unique in Q; C_{P}.receive context = C_{Q}.send context and this context is unique in P.

• C_{P}.source is rank of P in P and C_{Q}.source is rank of Q in Q.

Assume that P sends a message to Q using the inter-communicator. Then P uses the group table to find the absolute address of Q; source and send context are appended to the message.

Assume that Q posts a receive with an explicit source argument using the inter-communicator. Then Q matches receive context to the message context and source argument to the message source.

The same algorithm is appropriate for intra-communicators as well.

5.5.1 Communicator Accessors

MPI COMM TEST INTER(comm, flag)

IN  comm  communicator (handle)
OUT  flag  (logical)

int MPI Comm test inter(MPI Comm comm, int *flag)
MPI COMM TEST INTER(COMM, FLAG, IERROR)
    INTEGER COMM, IERROR
    LOGICAL FLAG

This local routine allows the calling process to determine if a communicator is an intercommunicator or an intra-communicator. It returns true if it is an intercommunicator.

When an inter-communicator is used as an input argument to the communicator accessors described under intra-communication, the following table describes behavior.

MPI COMM * Function Behavior (in Inter-Communication Mode)

MPI COMM SIZE returns the size of the remote group. MPI COMM GROUP returns the remote group. MPI COMM RANK returns the rank in the local group

5.5.2 Intercommunicator Constructors and Destructors

The functions MPI COMM DUP, MPI COMM MAKE and MPI COMM SPLIT do not apply to intercommunicators. In addition, the functions MPI INTERCOMM START and MPI INTERCOMM FINISH can be used to bind two groups into an intercommunicator; the function MPI INTERCOMM MERGE creates an intracommunicator by merging the two groups of an intercommunicator.

Construction of an inter-communicator from two intracommunicators requires separate collective operations in the local group and in the remote group, both for the starting and

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the finishing operations, as well as a point-to-point communication between a process in the local group and a process in the remote group. The explicit synchronization can cause deadlock in modular programs with cyclic communication graphs, even if within each local group calls are executed in the same order,

MPI INTERCOMM START and MPI INTERCOMM FINISH are both collective operations in their local group, but the call within each group need not be synchronized with the call in the other group.

The inter-communicator objects are destroyed in the same way as intra-communicator objects, that is, by calling MPI COMM FREE.

The functions MPI INTERCOMM START and MPI INTERCOMM FINISH can be used to create an intercommunicator from two existing intracommunicators, in the following situation: At least one selected member from each group (the “group leader”) has the ability to communicate with the selected member from the other group; i.e. a “parent” communicator exists where both leaders belong, and each leader knows the rank of the other leader in this parent communicator (the two leaders could be the same process). Furthermore, members of each group know the rank of their leader.

In static MPI implementation the MPI COMM WORLD communicator can be this parent communicator. Since this parent communicator may be shared for multiple intercommunicator creating functions, and the same process may be leaders for several such calls, an additional user-provided tag is used to differentiate calls.

In dynamic MPI implementations, where, e.g., a process may spawn new children processes during an MPI execution, the parent process may be the “bridge” between the old communication universe and the new communication world that includes the parent and its children.

MPI INTERCOMM START(local comm, local leader, peer comm, remote leader, tag, inter request)

IN  local comm  local intra-communicator (handle)
IN  local leader  rank of local group leader in local comm (integer)
IN  peer comm  “parent” intra-communicator; significant only at the

local leader (handle)

IN  remote leader  rank of remote group leader in peer comm; significant

only at the local leader (integer)

IN  tag  “safe” tag (integer)
OUT  inter request  handle for MPI INTERCOMM FINISH()

int MPI Intercomm start(MPI Comm local comm, int local leader,
             MPI Comm peer comm, int remote leader, int tag,
             MPI Intercomm request *inter request)
MPI INTERCOMM START(LOCAL COMM, LOCAL LEADER, PEER COMM, REMOTE LEADER, TAG,
             INTER REQUEST, IERROR)
    INTEGER LOCAL COMM, LOCAL LEADER, PEER COMM, REMOTE LEADER, TAG,
    INTER REQUEST, IERROR

This call starts off an inter-communicator creation operation, returning a handle for the completion of the operation in inter request; it is collective in local comm. The call does

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

not wait for the remote group to execute MPI INTERCOMM START(). inter request is conceptually similar to the communication request handle used by non-blocking point-to-point routines. The handle is constructed by a start routine and destroyed by the matching finish routine; these handles are not valid for any other use. It is erroneous to call this routine a second time with the same bridge comm, remote leader and tag arguments, without an intervening call to MPI INTERCOMM FINISH() to complete the pending call.

Processes should provide identical comm, local leader and tag arguments within each group.

MPI INTERCOMM FINISH(inter request, newcomm)

INOUT inter request handle returned by MPI INTERCOMM START()

OUT  newcomm  new inter-communicator (handle)

int MPI Intercomm finish(MPI Intercomm request *inter request,
             MPI Comm *newcomm)
MPI INTERCOMM FINISH(INTER REQUEST, IERROR)
    INTEGER INTER REQUEST, NEWCOMM, IERROR

MPI INTERCOMM FINISH completes an asynchronous inter-communicator creation operation. This routine is collective in the goup associated with communicator local comm of the corresponding call to MPI INTERCOMM START. It waits for the remote group to call MPI INTERCOMM START but does not wait for the remote group to call MPI INTERCOMM FINISH.

Implementation note: MPI INTERCOMM START initiates asynchronous sends and receives for an exchange of information between leaders. MPI INTERCOMM FINISH completes the exchange of information, next execute a broadcast in each local group, with the local leader at the root.

MPI INTERCOMM MERGE(intercomm, high, newintracomm)

IN  intercomm  Intercommunicator (handle)
IN  high  (logical)
OUT  newintracomm  new intra-communicator (handle)

int MPI Intercomm merge(MPI Comm intercomm, int high,
             MPI Comm *newintracomm)
MPI INTERCOMM MERGE(INTERCOMM, HIGH, NEWINTERCOMM, IERROR)
    INTEGER INTERCOMM, NEWINTRACOMM, IERROR
    LOGICAL HIGH

This function returns an intra-communicator for the union of the two groups that are associated with intercomm. All processes should provide the same high value within each of the two groups. If processes in one group provided the value high = false and processes in the other group provided the value high = true then the union orders the “low” group before the “high” group. If all processes provided the same high argument then the order

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

of the union is arbitrary. This call is blocking and collective within the union of the groups.

5.5.3 Name Service

MPI provides a name service to simplify construction of inter-communicators. This service allows a local process group to create an inter-communicator when the only available information about the remote group is a user-defined character string. A loosely synchronous version is provided by routines MPI INTERCOMM NAME START and MPI INTERCOMM FINISH. Note that for rendezvous to occur, all invocations of MPI INTERCOMM NAME START clearly must specify the same value for the name argument.

MPI INTERCOMM NAME START(comm, name, inter request)

IN  comm  local intra-communicator (handle)
IN  name  character string
OUT  inter request  handle for MPI INTERCOMM FINISH()

int MPI Intercomm name start(MPI Comm comm, char *name,
             MPI Intercomm request *inter request)
MPI INTERCOMM NAME START(COMM, NAME, NEW COMM, INTER REQUEST, IERROR)
    CHARACTER *(*) NAME
    INTEGER COMM, NEW COMM, INTER REQUEST, IERROR

This is the name-served equivalent of MPI INTERCOMM START. The operation is completed by a call to MPI INTERCOMM FINISH.

5.6 Caching

MPI provides a “caching” facility that allows an application to attach arbitrary pieces of information, called attributes, to communicators. More precisely, the caching facility allows a portable library to do the following:

• pass information between calls by associating it with an MPI communicator,

• quickly retrieve that information, and

• be guaranteed that out-of-date information is never retrieved, even if the communicator is freed and its handle subsequently reused by MPI.

These capabilities, in some form, are required by built-in MPI routines such as collective communication and virtual topology. Defining an interface to these capabilities as part of the MPI standard is valuable because it permits routines like collective communication and virtual topology to be implemented as portable code, and also because it makes MPI more extensible by allowing user-written routines to use standard MPI calling sequences.

5.6.1 Functionality

Attributes are local to the process and specific to the communicator to which they are attached. They are not propagated by MPI from one communicator to another except when

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the communicator is duplicated using MPI COMM DUP (and even then the application must give specific permission for the attribute to be copied). Attributes are scalar values, equal in size to, or larger than a C-language pointer.

The caching interface defined here represents that attributes are stored by MPI opaquely within a communicator. Accessor functions include the following:

• obtain a key value (used to identify an attribute); the user specifies “callback” functions by which MPI informs the application when the communicator is destroyed or copied.

• store and retrieve the value of an attribute;

MPI provides the following services related to caching. They are all local.

MPI ATTR CREATE KEY(copy fn, delete fn, keyval, extra state)

IN  copy fn  Copy callback function for keyval
IN  delete fn  Delete callback function for keyval
OUT  keyval  key value for future access (integer)
IN  extra state  Extra state for callback functions

int MPI Attr create key(int copy fn(), int delete fn(), int keyval,
             void* extra state)
MPI ATTR CREATE KEY(COPY FN, DELETE FN, KEYVAL, EXTRA STATE, IERROR)
    EXTERNAL COPY FN, DELETE FN
    INTEGER KEYVAL, EXTRA STATE, IERROR

Generates a new attribute key. Keys are locally unique in process, and opaque to user, though they are explicitly stored in integers. Once allocated, the key value can be used to associate attributes and access them on any locally defined communicator.

The copy fn function is invoked when a communicator is duplicated. A C prototype for such a function is

int copy fn(oldcomm, newcomm, keyval, extra state)

This function does whatever it needs to do with the new communicator newcomm based on what attribute information was in the old communicator oldcomm; it returns 1 if successful, 0 if not. A valid copy function is one that completely duplicates the information by making a full duplicate copy of the data structures implied by an attribute; another might just make another reference to that data structure, while using a reference-count mechanism. Other types of attributes might not copy at all (they might be specific to oldcomm only). MPI COMM DUP is the only function that calls the copy fn. copy fn may be specified as null, in which case no copy callback occurs for keyval.

Analogous to copy fn is a callback deletion function, defined as follows:

delete fn(comm, keyval, extra state)

This function is called by MPI COMM FREE and MPI ATTR DELETE to do whatever is needed to remove an attribute. It may be specified as null, in which case no delete callback occurs for keyval.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI ATTR FREE KEY(keyval)

IN  keyval  Frees the integer key value (integer)

int MPI Attr free key(int keyval)
MPI ATTR FREE KEY(KEYVAL, IERROR)
    INTEGER KEYVAL, IERROR

Frees an extant attribute key. It is not erroneous to free an attribute key that is in use, because the actual free does not transpire until after all references (in other communicators on the process) to the key have been freed. These references need to be explictly freed by the program, either via calls to MPI ATTR DELETE that free one attribute instance, or by calls to MPI COMM FREE that free all attribute instances associated with the freed communicator.

MPI PUT ATTR(comm, keyval, attribute val)

IN  comm  communicator to which attribute will be attached (han-

dle)

IN  keyval  key value, as returned by

MPI ATTR CREATE KEY (integer)

IN  attribute val  attribute value

int MPI Put attr(MPI Comm comm, int keyval, void* attribute val)
MPI PUT ATTR(COMM, KEYVAL, ATTRIBUTE VAL, IERROR)
    INTEGER COMM, KEYVAL, ATTRIBUTE VAL, IERROR

This function stores the stipulated attribute value attribute val for subsequent retrieval by MPI GET ATTR. If the value is already present, then the outcome is as if MPI DELETE ATTR was first called to delete the previous value (and the callback function delete fn was executed), and a new value was next stored. The call is erroneous if there is no key with value keyval.

MPI GET ATTR(comm, keyval, attribute val, found)

IN  comm  communicator to which attribute is attached (handle)
IN  keyval  key value (integer)
OUT  attribute val  attribute value, unless found = false
OUT  found  true if an attribute value was extracted; false if no

attribute is associated with the key

int MPI Get attr(MPI Comm comm, int keyval, void* attribute val, int *found)
MPI GET ATTR(COMM, KEYVAL, ATTRIBUTE VAL, FOUND, IERROR)
    INTEGER COMM, KEYVAL, ATTRIBUTE VAL, IERROR
    LOGICAL FOUND

Retrieves attribute value by key. The call is erroneous if there is no key with value keyval. On the other hand, the call is correct if the key value exists, but no attribute is

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

attached on comm for that key; in such case, the call returns found = false.

MPI DELETE ATTR(comm, keyval)

IN  comm  communicator to which attribute is attached (handle)
IN  keyval  The key value of the deleted attribute (integer)

int MPI Delete Attr(MPI Comm comm, int keyval)
MPI DELETE ATTR(COMM, KEYVAL, IERROR)
    INTEGER COMM, KEYVAL, IERROR

Delete attribute from cache by key. This function invokes the attribute delete function delete fn specified when the keyval was created.

Whenever a communicator is replicated using the function MPI COMM DUP, all callback copy functions for attributes that are currently set are invoked (in arbitrary order). Whenever a communicator is deleted using the function MPI COMM FREE all callback delete functions for attributes that are currently set are invoked.

Example

This example shows how to write a collective communication operation that uses caching to be more efficient after the first call. The coding style assumes that MPI function results return only error statuses.

   static int gop_key_assigned = 0;  /* 0 only on first entry */
   static int gop_key;               /* key for this module's stuff */
   typedef struct
   {
      int ref_count;          /* reference count */
      /* other stuff, whatever else we want */
   } gop_stuff_type;
   Efficient_Collective_Op (comm, ...)
   MPI_Comm comm;
   {
     gop_stuff_type *gop_stuff;
     MPI_Group       group;
     int             foundflag;
     MPI_Comm_group(comm, &group);
     if (!gop_key_assigned)     /* get a key on first call ever */
     {
       gop_key_assigned = 1;
       if ( ! MPI_Attr_get_key( gop_stuff_copier,
                                gop_stuff_destructor,
                                &gop_key, (void *)0));
       /* get the key while assigning its copy and delete callback

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

     behavior. */
  MPI_Abort ("Insufficient keys available");
}
MPI_Attr_get_value (comm, gop_key, &gop_stuff, &foundflag);
if (foundflag)
{ /* This module has executed in this group before.
     We will use the cached information */
}
else
{ /* This is a group that we have not yet cached anything in.
     We will now do so.
  */
  /* First, allocate storage for the stuff we want,
     and initialize the reference count */
  gop_stuff = (gop_stuff_type *) malloc (sizeof(gop_stuff_type));
  if (gop_stuff == NULL) { /* abort on out-of-memory error */ }
  gop_stuff -> ref_count = 1;
  /* Second, fill in *gop_stuff with whatever we want.
     This part isn't shown here */
  /* Third, store gop_stuff as the attribute value */
  MPI_Attr_put_value ( comm, gop_key, gop_stuff);
}
/* Then, in any case, use contents of *gop_stuff
   to do the global op ... */
}
/* The following routine is called by MPI when a group is freed */
gop_stuff_destructor (comm, keyval, gop_stuff, extra)
MPI_Comm comm;
int keyval;
gop_stuff_type *gop_stuff;
void *extra;
{
if (keyval != gop_key) { /* abort -- programming error */ }
/* The group's being freed removes one reference to gop_stuff */
gop_stuff -> ref_count -= 1;
/* If no references remain, then free the storage */
if (gop_stuff -> ref_count == 0) {

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

5.7. FORMALIZING THE LOOSELY SYNCHRONOUS MODEL (USAGE, SAFETY)125

       free((void *)gop_stuff);
     }
   }
   /* The following routine is called by MPI when a group is copied */
   gop_stuff_copier (comm, keyval, gop_stuff, extra)
   MPI_Comm comm;
   int keyval;
   gop_stuff_type *gop_stuff;
   void *extra;
   {
     if (keyval != gop_key) { /* abort -- programming error */ }
     /* The new group adds one reference to this gop_stuff */
     gop_stuff -> ref_count += 1;
   }

5.7 Formalizing the Loosely Synchronous Model (Usage, Safety)

5.7.1 Basic Statements

When a caller passes a communicator (that contains a context and group) to a callee, that communicator must be free of side effects throughout execution of the subprogram (quiescent). This provides one model in which libraries can be written, and work “safely.” For libraries so designated, the callee has permission to do whatever communication it likes with the communicator, and under the above guarantee knows that no other communications will interfere. Since we permit good implementations to create new communicators without synchronization (such as by preallocated contexts on communicators), this does not impose a significant overhead.

This form of safety is analogous to other common computer-science usages, such as passing a descriptor of an array to a library routine. The library routine has every right to expect such a descriptor to be valid and modifiable.

5.7.2 Models of Execution

We say that a parallel procedure is active in a process if the process belongs to a group that may collectively execute the procedure, and some member of that group is currently executing the procedure code. If a parallel procedure is active in a process, then this process may be receiving messages pertaining to this procedure, even if it does not currently execute the code of this procedure.

Non-reentrant parallel procedures

This covers the case where, at any point in time, at most one invocation of a parallel procedure can be active at any process. That is, concurrent invocations of the same parallel procedure may occur only within disjoint groups of processes. For example, all invocations of parallel procedures involve all processes, processes are single-threaded, and there are no recursive invocations.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

In such a case, a context can be statically allocated to each procedure. The static allocation can be done in a preamble, as part of initialization code. Or, it can be done a compile/link time, if the implementation has additional mechanisms to reserve context values. Communicators to be used by the different procedures can be built in a preamble, if the executing groups are statically defined; if the executing groups change dynamically, then a new communicator has to be built whenever the executing group changes, but this new communicator can be built using the same type of preallocated context. If the parallel procedures can be organized into libraries, so that only one procedure of each library can be concurrently active in each processor, then it is sufficient to allocate one context per library.

Parallel procedures that are non-reentrant within each executing group

This covers the case where, at any point in time, for each process group, there can be at most one active invocation of a parallel procedure by a process member. However, it might be possible that the same procedure is concurrently invoked in two partially (or completely) overlapping groups. For example, the same collective communication function may be concurrently invoked on two partially overlapping groups.

In such a case, a context is associated with each parallel procedure and each executing group, so that overlapping execution groups have distinct communication contexts. (One does not need a different context from each group; one merely needs a “coloring” of the groups, so that one can generate the communicators for each parallel procedure when the execution groups are defined. Here again, one only need one context for each library, if no two procedures from the same library can be concurrently active in the same group.

Note that, for collective communication libraries, we do allow several concurrent invocations within the same group: a broadcast in a group may be started in a process before the previous broadcast in that group ended at another process. In such a case, one cannot rely on context mechanisms to disambiguate successive invocations of the same parallel procedure within the same group: the procedure needs to be implemented so as to avoid confusion (back masking). For example, for broadcast, one may need to carry additional information in messages, such as the broadcast root, to help in such disambiguation; one also relies on the pair-wise preservation of message order by MPI. With such an approach, we may be gaining performance, but we lose modularity. It is insufficient to implement the parallel procedure so that it works correctly in isolation, when invoked only once; it needs to be implemented so that any number of successive invocations will execute correctly. Of course, the same approach can be used for other parallel libraries.

Well-nested parallel procedures

Calls of parallel procedures are well-nested if a new parallel procedure is always invoked in a subset of a group executing the same parallel procedure. Thus, processes that execute the same parallel procedure have the same execution stack.

In such a case, a new context needs to be dynamically allocated for each new invocation of a parallel procedure. However, a stack mechanism can be used for allocating new contexts. Thus, a possible mechanism is first to allocate a large number of contexts (up to the upper bound on the depth of nested parallel procedure calls), and then use a local stack management of these context’s on each process to create a new communicator (using MPI COMM MAKE) for each new invocation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The General case

In the general case, there may be multiple concurrently active invocations of the same parallel procedure within the same group; invocations may not be well-nested. A new context need to be created for each invocation. It is the user responsibility to make sure that, if two distinct parallel procedures are invoked concurrently on overlapping sets of processes, then context allocation or communicator creation is properly coordinated.

5.8 Motivating Examples

5.8.1 Current Practice #1

Example #1a:

   main(int argc, char **argv)
   {
     int me, size;
     ...
     MPI_Init ( &argc, argv );
     MPI_Comm_rank (MPI_COMM_WORLD, &me);
     MPI_Comm_size (MPI_COMM_WORLD, &size);
     (void)printf ("Process %d size %d\n", me, size);
     ...
     MPI_Finalize();
   }

Example #1a is a do-nothing program that initializes itself legally, and refers to the the “all” communicator, and prints a message. It terminates itself legally too. This example does not imply that MPI supports printf-like communication itself. Example #1b (supposing that size is even):

    main(int argc, char **argv)
    {
       int me, size;
       int SOME_TAG = 0;
       ...
       MPI_Init(argc, argv);
       MPI_Comm_rank(MPI_COMM_WORLD, &me);   /* local */
       MPI_Comm_size(MPI_COMM_WORLD, &size); /* local */
       if((me % 2) == 0)
       {
          /* send unless highest-numbered process */
          if((me + 1) < size)
             MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);
       }
       else
          MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

   ...
   MPI_Finalize();
}

Example #1b schematically illustrates message exchanges between “even” and “odd” processes in the “all” communicator.

5.8.2 Current Practice #2

main(int argc, char **argv)
{
 int me, count;
 void *data;
 ...
 MPI_Init(argc, argv);
 MPI_Comm_rank(MPI_COMM_WORLD, &me);
 if(me == 0)
 {
     /* get input, create buffer ``data'' */
     ...
 }
 MPI_Bcast(data, count, MPI_BYTE, 0, MPI_COMM_WORLD);
 ...
 MPI_Finalize();
}

This example illustrates the use of a collective communication.

5.8.3 (Approximate) Current Practice #3

main(int argc, char **argv)
{
int me, count, count2;
void *send_buf, *recv_buf, *send_buf2, *recv_buf2;
MPI_Group MPI_GROUP_WORLD, grp0, grprem;
MPI_Comm commslave;
static int ranks[] = {0};
...
MPI_Init(argc, argv);
MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD);
MPI_Comm_rank(MPI_COMM_WORLD, &me);  /* local */
MPI_Group_excl(MPI_GROUP_WORLD,1, ranks, &grprem);  /* local */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

    MPI_Comm_make(MPI_COMM_WORLD, grprem, &commslave);
    if(me != 0)
    {
      /* compute on slave */
      ...
      MPI_Reduce(send_buf,recv_buff,count, MPI_INT, MPI_SUM, 1, commslave);
      ...
    }
    /* zero falls through immediately to this reduce, others do later... */
    MPI_Reduce(send_buf2, recv_buff2, count2,
               MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
    MPI_Finalize();
  }

This example illustrates how a group consisting of all but the zeroth process of the “all” group is created, and then how a communicator is formed ( commslave) for that new group. The new communicator is used in a collective call, and all processes execute a collective call in the MPI COMM WORLD context. This example illustrates how the two communicators (that inherently possess distinct contexts) protect communication. That is, communication in MPI COMM WORLD is insulated from communication in commslave, and vice versa.

In summary, “group safety” is achieved via communicators because distinct contexts within communicators are enforced to be unique on any process.

5.8.4 Example #4

The following example is meant to illustrate “safety” between point-to-point and collective communication. MPI guarantees that a single communicator can do safe point-to-point and collective communication.

   #define TAG_ARBITRARY 12345
   #define SOME_COUNT       50
   main(int argc, char **argv)
   {
     int me;
     MPI_Group MPI_GROUP_WORLD, subgroup;
     int ranks[] = {2, 4, 6, 8};
     ...
     MPI_Init(argc, argv);
     MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD);
     MPI_Group_incl(MPI_GROUP_WORLD, 4, ranks, &subgroup); /* local */
     MPI_Group_rank(subgroup, &me);     /* local */
     MPI_Comm_make(MPI_COMM_WORLD, subgroup, &the_comm);
     if(me != MPI_UNDEFINED)
     {

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

    MPI_Irecv(buff1, count, MPI_DOUBLE, MPI_ANY_SOURCE, TAG_ARBITRARY,
                      the_comm, request)
    MPI_Isend(buff2, count, MPI_DOUBLE, (me+1)%4, TAG_ARBITRARY,
                      the_comm, request+1)
}
for(i = 0; i < SOME_COUNT, i++)
  MPI_Reduce(..., the_comm);
MPI_Waitall(2, request, status)
MPI_Finalize();
}

5.8.5 Library Example #1

The main program:

main(int argc, char **argv)
{
int done = 0;
user_lib_t *libh_a, *libh_b;
void *dataset1, *dataset2;
...
MPI_Init(argc, argv);
...
init_user_lib(MPI_COMM_WORLD, &libh_a);
init_user_lib(MPI_COMM_WORLD, &libh_b);
...
user_start_op(libh_a, dataset1);
user_start_op(libh_b, dataset2);
...
while(!done)
{
   /* work */
   ...
   MPI_Reduce(..., MPI_COMM_WORLD);
   ...
   /* see if done */
   ...
}
user_end_op(libh_a);
user_end_op(libh_b);
MPI_Finalize();
}

The user library initialization code:

void init_user_lib(MPI_Comm *comm, user_lib_t **handle)
{
user_lib_t *save;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

user_lib_initsave(&save); /* local */
MPI_Comm_dup(comm, &(save -> comm));
/* other inits */
...
*handle = save;
}

User start-up code:

void user_start_op(user_lib_t *handle, void *data)
{
MPI_Irecv( ..., handle->comm, &(handle -> irecv_handle) );
MPI_Isend( ..., handle->comm, &(handle -> isend_handle) );
}

User clean-up code:

void user_end_op(user_lib_t *handle)
{
MPI_Status *status;
MPI_Wait(handle -> isend_handle, status);
MPI_Wait(handle -> irecv_handle, status);
}

5.8.6 Library Example #2

The main program:

main(int argc, char **argv)
{
int ma, mb;
...
static int list_a[] = {0, 1};
#if  defined(EXAMPLE_2B) | defined(EXAMPLE_2C)
static int list_b[] = {0, 2 ,3};
#else/* EXAMPLE_2A */
static int list_b[] = {0, 2};
#endif
int size_list_a = sizeof(list_a)/sizeof(int);
int size_list_b = sizeof(list_b)/sizeof(int);
...
MPI_Init(argc, argv);
...
MPI_Group_incl(MPI_GROUP_WORLD, size_list_a, list_a, &group_a);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI_Group_incl(MPI_GROUP_WORLD, size_list_b, list_b, &group_b);
MPI_Comm_make(MPI_COMM_WORLD, group_a, &comm_a);
MPI_Comm_make(MPI_COMM_WORLD, group_b, &comm_b);
MPI_Comm_rank(comm_a, &ma);
MPI_Comm_rank(comm_b, &mb);
if(ma != MPI_UNDEFINED)
   lib_call(comm_a);
if(mb != MPI_UNDEFINED)
{
  lib_call(comm_b);
  lib_call(comm_b);
}
MPI_Finalize();
}

The library:

void lib_call(MPI_Comm comm)
{
int me, done = 0;
MPI_Comm_rank(comm, &me);
if(me == 0)
   while(!done)
   {
      MPI_Recv(..., MPI_ANY_SOURCE, MPI_ANY_TAG, comm);
      ...
   }
else
{
  /* work */
  MPI_Send(..., 0, ARBITRARY_TAG, comm);
  ....
}
#ifdef EXAMPLE_2C
/* include (resp, exclude) for safety (resp, no safety): */
MPI_Sync(comm);
#endif
}

The above example is really three examples, depending on whether or not one includes rank 3 in list b, and whether or not a synchronize is included in lib call. This example illustrates that, despite contexts, subsequent calls to lib call with the same context need not be safe from one another (colloquially, “back-masking”). Safety is realized if the MPI Sync is added. What this demonstrates is that libraries have to be written carefully, even with

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

contexts. When rank 3 is excluded, then the synchronize is not needed to get safety from back masking.

Algorithms like “combine” have strong enough source selectivity so that they are inherently okay. So are multiple calls to a typical tree-broadcast algorithm with the same root. However, multiple calls to a typical tree-broadcast algorithm — with different roots — could back-mask, producing undesirable results. Therefore, such algorithms would have to utilize the monotonically increasing tags (with communicator scope) to keep things straight. All of the foregoing is a supposition of “collective calls” implemented with point-to-point operations. MPI implementations may or may not implement collective calls using pointto-point operations. These algorithms are used to illustrate the issues of correctness and safety, independent of how MPI implements its collective calls.

5.8.7 Inter-Communication Examples

Example 1: Three-Group “Pipeline”

              +---------+         +---------+         +---------+
              |         |         |         |         |         |
              | Group 0 | <-----> | Group 1 | <-----> | Group 2 |
              |         |         |         |         |         |
              +---------+         +---------+         +---------+

Groups 0 and 1 communicate. Groups 1 and 2 communicate. Therefore, group 0 requires one inter-communicator, group 1 requires two inter-communicators, and group 2 requires 1 inter-communicator.

   main(int argc, char **argv)
   {
     MPI_Comm myComm;       /* intra-communicator of local sub-group */
     MPI_Comm myFirstComm;  /* inter-communicator */
     MPI_Comm mySecondComm; /* second inter-communicator (group 1 only) */
     MPI_Intercomm_request inter_request1, inter_request2;
     int membershipKey;
     int rank;
     MPI_Init(argc, argv);
     MPI_Comm_rank(MPI_COMM_WORLD, &rank);
     /* User code must generate membershipKey in the range [0, 1, 2] */
     membershipKey = rank % 3;
     /* Build intra-communicator for local sub-group */
     MPI_Comm_split(MPI_COMM_WORLD, membershipKey, rank, &myComm);
     /* Build inter-communicators.  Tags are hard-coded. */
     if (membershipKey == 0)
     {                     /* Group 0 communicates with group 1. */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 1,
                       1, &inter_request1 );
  MPI_Intercomm_finish( inter_request1, myFirstComm );
}
else if (membershipKey == 1)
{              /* Group 1 communicates with groups 0 and 2. */
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 0,
                       1, &inter_request1 );
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 2,
                       12, &inter_request2 );
  MPI_Intercomm_finish( inter_request1, myFirstComm );
  MPI_Intercomm_finish( inter_request2, mySecondComm );
}
else if (membershipKey == 2)
{                     /* Group 2 communicates with group 1. */
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 1,
                       12, &inter_request1 );
  MPI_Intercomm_finish( inter_request1, myFirstComm );
}
/* Do work ... */
MPI_Finalize();
}

Example 2: Three-Group “Ring”

    +-----------------------------------------------------------+
    |                                                           |
    |    +---------+         +---------+         +---------+    |
    |    |         |         |         |         |         |    |
    +--> | Group 0 | <-----> | Group 1 | <-----> | Group 2 | <--+
         |         |         |         |         |         |
         +---------+         +---------+         +---------+

Groups 0 and 1 communicate. Groups 1 and 2 communicate. Groups 0 and 2 communicate. Therefore, each requires two inter-communicators.

main(int argc, char **argv)
{
MPI_Comm  myComm;     /* intra-communicator of local sub-group */
MPI_Comm myFirstComm; /* inter-communicators */
MPI_Comm mySecondComm;
MPI_Intercomm_request inter_request1, inter_request2;
int membershipKey;
int rank;
MPI_Init(argc, argv);
MPI_Comm_rank(MPI_COMMM_WORLD, &rank);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

...
/* User code must generate membershipKey in the range [0, 1, 2] */
membershipKey = rank % 3;
/* Build intra-communicator for local sub-group */
MPI_Comm_split(MPI_COMM_WORLD, membershipKey, rank, &myComm);
/* Build inter-communicators.  Tags are hard-coded. */
if (membershipKey == 0)
{             /* Group 0 communicates with groups 1 and 2. */
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 1,
                       1, &inter_request1 );
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 2,
                       2, &inter_request2 );
}
else if (membershipKey == 1)
{         /* Group 1 communicates with groups 0 and 2. */
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 0,
                       1, &inter_request1 );
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 2,
                       12, &inter_request2 );
}
else if (membershipKey == 2)
{        /* Group 2 communicates with groups 0 and 1. */
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 0,
                       2, &inter_request1 );
  MPI_Intercomm_start( myComm, 0, MPI_COMM_WORLD, 1,
                       12, &inter_request2 );
}
/* Everyone has the same "FINISH" code... */
MPI_Intercomm_finish(inter_request1, &myFirstComm);
MPI_Intercomm_finish(inter_request2, &mySecondComm);
/* Do some work ... */
MPI_Finalize();
}

Example 3: Three-Group “Pipeline” Using Name Service

         +---------+         +---------+         +---------+
         |         |         |         |         |         |
         | Group 0 | <-----> | Group 1 | <-----> | Group 2 |
         |         |         |         |         |         |
         +---------+         +---------+         +---------+

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Groups 0 and 1 communicate. Groups 1 and 2 communicate. Therefore, group 0 requires one inter-communicator, group 1 requires two inter-communicators, and group 2 requires 1 inter-communicator.

main(int argc, char **argv)
{
MPI_Comm myComm;      /* intra-communicator of local sub-group */
MPI_Comm myFirstComm; /* inter-communicator */
MPI_Comm mySecondComm;/* second inter-communicator (group 1 only) */
MPI_Intercomm_request inter_request1, inter_request2;
int color, rank, size;
MPI_Init(argc, argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
...
/* User code must generate color in the range [0, 1, 2] */
if ( rank < ((size+1)/3) )
  color = 0;
else if ( rank < (2*(size+1)/3) )
  color = 1;
else
  color = 2;
/* Build intra-communicator for local sub-group */
MPI_Comm_split(MPI_COMM_WORLD, color, rank, &myComm);
/* Build inter-communicators.  Group membership conditions must be */
/* provided by the user. */
if (color == 0)
{       /* Group 0 communicates with group 1. */
  MPI_Intercomm_name_start(myComm, "Connect 01", &inter_request1);
  MPI_Intercomm_finish (inter_request1, &myFirstComm);
}
else if (color == 1)
{       /* Group 1 communicates with groups 0 and 2. */
  MPI_Intercomm_name_start(myComm, "Connect 01", &inter_request1);
  MPI_Intercomm_name_start(myComm, "Connect 12", &inter_request2);
  MPI_Intercomm_finish (inter_request1, &myFirstComm);
  MPI_Intercomm_finish (inter_request2, &mySecondComm);
}
else if (color == 2)
{       /* Group 2 communicates with group 1. */
  MPI_Intercomm_name_start(myComm, "Connect 12", &inter_request1);
  MPI_Intercomm_finish (inter_request1, &myFirstComm);
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

/* Do some work ... */
MPI_Finalize();
}

Example 4: Three-Group “Ring” Using Name Service

    +-----------------------------------------------------------+
    |                                                           |
    |    +---------+         +---------+         +---------+    |
    |    |         |         |         |         |         |    |
    +--> | Group 0 | <-----> | Group 1 | <-----> | Group 2 | <--+
         |         |         |         |         |         |
         +---------+         +---------+         +---------+

Groups 0 and 1 communicate. Groups 1 and 2 communicate. Groups 0 and 2 communicate. Therefore, each requires two inter-communicators.

main(int argc, char **argv)
{
MPI_Comm  myComm;   /* intra-communicator of local sub-group */
MPI_Comm  myFirstComm;                /* inter-communicators */
MPI_Comm  mySecondComm;
MPI_Intercomm_request inter_request1, inter_request2;
int color, rank, size;
MPI_Init(argc, argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
...
/* User code must generate color in the range [0, 1, 2] */
if ( rank == 0 )
  color = 0;
else if ( rank == 1 )
  color = 1;
else
  color = 2;
/* Build intra-communicator for local sub-group */
MPI_Comm_split(MPI_COMM_WORLD, color, rank, &myComm);
/* Build inter-communicators.  Group membership conditions must be */
/* provided by the user. */
if (color == 0)
{     /* Group 0 communicates with groups 1 and 2. */
  MPI_Intercomm_name_start(myComm, "Connect 01", &inter_request1);
  MPI_Intercomm_name_start(myComm, "Connect 02", &inter_request2);
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

else if (color == 1)
{     /* Group 1 communicates with groups 0 and 2. */
  MPI_Intercomm_name_start(myComm, "Connect 01", &inter_request1);
  MPI_Intercomm_name_start(myComm, "Connect 12", &inter_request2);
}
else if (<I am in group 2>)
{    /* Group 2 communicates with groups 0 and 1. */
  MPI_Intercomm_name_start(myComm, "Connect 02", &inter_request1);
  MPI_Intercomm_name_start(myComm, "Connect 12", &inter_request2);
}
/* Everyone has the same "FINISH" code... */
MPI_Intercomm_finish (inter_request1, &myFirstComm);
MPI_Intercomm_finish (inter_request2, &mySecondComm);
/* Do some work ... */
MPI_Finalize();
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 6

Process Topologies

6.1 Introduction

A process group in MPI is a collection of n processes. Each process in the group is assigned a rank between 0 and n − 1. In many parallel applications a linear process ordering does not reflect the communication pattern of the processes, which is usually determined by the underlying problem geometry and the numerical algorithm used. Often the processes are arranged in topological patterns as simple as a two– or three–dimensional grid. In the more general case the logical process arrangement is described by a graph. In this chapter we will refer to this logical process arrangement as the “virtual topology.”

A clear distinction must be made between the virtual process topology and the topology of the physical hardware processors. The application-specific, virtual topology can be exploited by the system in the assignment of processes to physical processors, if this helps to improve the communication performance on a given machine. How this mapping is done in detail, however, is outside the scope of MPI. The description of the virtual process topology, on the other hand, only depends on the application, and is, therefore, machine– independent. The functions which are proposed in this paper only deal with this side of the mapping problem.

There are well–known techniques for mapping grid/torus structures to hardware topologies such as hypercubes or grids. For more complicated graph structures good heuristics often yield near–optimal results [16]. On the other hand, if there is no way for the user to specify the logical process arrangement as a “virtual topology,” a random mapping is most likely the consequence. On some machines this will lead to unnecessary contention in the interconnect network. Therefore, though the physical mapping is not discussed, the mere existence of the virtual topology information may be used as advice by the runtime system. Some details about predicted and measured performance improvements which result from good process–to–processor mapping on modern wormhole–routing architectures can be found in [8, 7].

6.2 Virtual Topologies

In the most general case the communication pattern of a set of processes can be represented by a graph. The nodes stand for the processes, and the edges connect processes which communicate with each other. MPI provides message–passing between any pair of processes in a group. There is no requirement for opening a channel explicitly. Therefore, a “miss-

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

ing link” in the user–defined process graph does not prevent the corresponding processes from exchanging messages. It only means that this connection is neglected in the virtual topology. This implies that the topology gives no convenient way of naming this pathway of communication. Another possible consequence is that an automatic mapping tool (if one exists for the runtime environment) will not try to assign the processes to neighboring processors. Edges in the communication graph are not weighted. Processes are either neighbors or not. Experience with similar techniques in PARMACS [4, 6] show that this information is usually sufficient for a good mapping. Additionally, a more precise specification would be more difficult for the user to set up, and it would make the interface functions more complicated.

In principle, a function for the definition of such a graph structure would be sufficient for all applications. However, in many applications the graph structure is very regular, and the detailed set–up of the graph would be inconvenient for the user and resource–consuming during run time. A large fraction of all parallel applications use process topologies like rings, two– or higher–dimensional grids, or tori. These structures are completely defined by the number of dimensions and the numbers of processes in each coordinate direction. Also, the mapping of grids and tori is generally an easier problem then that of general graphs. Thus, it is desirable to separate out this case explicitly.

Process coordinates in a Cartesian structure begin their numbering at 0. Row major numbering is used for the processes in a Cartesian structure. This means that, for example, the relation between group rank and coordinates for four processes in a (2 × 2) grid is as follows.

coord (0,0): rank 0
coord (0,1): rank 1
coord (1,0): rank 2
coord (1,1): rank 3

6.3 Embedding in MPI

The support for virtual topologies as defined in this chapter is consistent with other parts of MPI, and, whenever possible, makes use of functions which are defined elsewhere. Topology information is associated with communicators. It can be maintained using the caching mechanism described in Chapter 5. The topology functions complement and interoperate with the collective communication functions described in Chapter 4.

The current design of the process topology functions assumes that MPI is free to choose any process in a group for any location in the topology. There are no special processes which have to be aligned with topology substructures. This assumption fits best to a “local SPMD model” in which all processes in the same group execute the same program.

6.4 Overview of the proposed MPI functions

The functions MPI MAKE GRAPH and MPI MAKE CART are used to create general virtual topologies and Cartesian topologies, respectively. These topology creation functions are collective. As for other collective calls, the code should be written to work correctly, whether the call synchronizes or not.

The topology definition function MPI MAKE CART can be used to describe Cartesian structures of arbitrary dimension. For each coordinate direction it is specified whether

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the process structure is periodic or not. Similar functions are contained in EXPRESS [17] and PARMACS. Note that an n–dimensional hypercube is an n–dimensional torus with 2 processes per coordinate direction. Thus, special support for hypercube structures is not necessary. The local auxiliary function MPI MAKE DIMS can be used to compute an “even” distribution of processes among a given number of dimensions.

The function MPI TOPO STATUS can be used to inquire on the topology associated with a communicator. The topological information can be extracted from the communicator using the functions MPI GET GRAPHDIMS and MPI GET GRAPH, for general graphs, and MPI GET CARTDIM and MPI GET CART, for Cartesian topologies. Several additional functions are provided to manipulate Cartesian topologies: the functions MPI CART RANK and MPI CART COORDS translate Cartesian coordinates into a group rank, and vice-versa; the function MPI CART SUB can be used to extract a Cartesian subspace; and the function MPI CART SHIFT provides the information needed to execute a shift in a Cartesian dimension. The two functions MPI GRAPH NEIGHBORS COUNT and MPI GRAPH NEIGHBORS can be used to extract the neighbors of a node in a graph. The function MPI CART SUB is collective; all other functions are local.

Two additional functions, MPI MAP GRAPH and MPI MAP CART are presented in the last section. In general these functions will not be called by the user directly. However, together with the communicator manipulation functions presented in Chapter 5, they are sufficient to implement all other topology functions. The last section outlines such an implementation.

6.5 Topology Constructors

MPI MAKE CART(comm old, ndims, dims, periods, reorder, comm cart)

IN  comm old  communicator for parent group (handle)
IN  ndims  number of dimensions of Cartesian grid (integer)
IN  dims  integer array of size ndims specifying the number of

processes in each dimension

IN  periods  logical array of size ndims specifying whether the grid

is periodic (true) or aperiodic (false) in each dimension

IN  reorder  ranking may be reordered (true) or not (false) (logical)
OUT  comm cart  communicator for new Cartesian group (handle)

int MPI Make cart(MPI comm comm old, int ndims, int *dims, int *periods,
             int reorder, MPI comm *comm cart)
MPI MAKE CART(COMM OLD, NDIMS, DIMS, PERIODS, REORDER, COMM CART, IERROR)
    INTEGER COMM OLD, NDIMS, DIMS(*), PERIODS(*), REORDER, COMM CART,
    IERROR

The function returns a handle to a new communicator to which the Cartesian topology information is attached. If reorder = false then the rank of each process in the new group is identical to its rank in the old group. Otherwise, the function may reorder the processes, so as to choose a good embedding of the virtual topology onto the physical machine. If

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the total size of the Cartesian grid is smaller than the size of the group of comm, then a process which is not a member of the grid is returned a null handle. The call is erroneous if it specifies a grid that is larger than the group size.

In a portable program the number of available processors is usually not known at compile time. In the case of a Cartesian topology the function MPI MAKE DIMS helps in setting the numbers of processes per coordinate direction, depending on the number of processes in the group and optional constraints which can be specified by the user:

MPI MAKE DIMS(nnodes, ndims, dims)

IN  nnodes  number of nodes in a grid (integer)
IN  ndims  number of Cartesian dimensions (integer)

INOUT dims integer array of size ndims specifying the number of nodes in each dimension

int MPI Make dims(int nnodes, int ndims, int *dims)
MPI MAKE DIMS(NNODES, NDIMS, DIMS, IERROR)
    INTEGER NNODES, NDIMS, DIMS(*), IERROR

The entries in the array dims are set to describe a Cartesian grid with ndims dimensions and a total of nnodes nodes. The dimensions are set to be as close to each other as possible. The caller may further constrain the operation of this routine by specifying elements of array dims. If dims[i] is set to a positive number, the routine will not modify the number of nodes in the corresponding dimension; only those entries where dims[i] = 0 will be modified by the call.

Note that an error will occur if nnodes is not a multiple of ∏_{i,dims[i]≠0} dims[i].

Negative input values of dims[i] will also cause an error. All elements of dims must be set to 0 for completely automatic process distribution.

For dims[i] set by the call, dims[i] will be ordered in non–increasing order. Array dims is suitable for use as input to routine MPI MAKE CART. The function MPI MAKE DIMS is local.

Examples:

dims  function call  dims
before call  on return
(0,0)  MPI MAKE DIMS(6, 2, dims) (3,2)
(0,0)  MPI MAKE DIMS(7, 2, dims) (7,1)
(0,3,0)  MPI MAKE DIMS(6, 3, dims) (2,3,1)
(0,3,0)  MPI MAKE DIMS(7, 3, dims) erroneous call

For a general graph structure, the constructor is

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI MAKE GRAPH(comm old, nnodes, index, edges, reorder, comm graph)

IN  comm old  communicator for parent group (handle)
IN  nnodes  number of nodes in graph (integer)
IN  index  array of integers describing node degrees (see below)
IN  edges  array of integers describing graph edges (see below)
IN  reorder  ranking may be reordered (true) or not (false) (logical)
OUT  comm graph  communicator for new graph group (handle)

int MPI Make graph(MPI comm comm old, int nnodes, int *index, int *edges,
             int reorder, MPI comm *comm graph)
MPI MAKE GRAPH(COMM OLD, NNODES, INDEX, EDGES, REORDER, COMM GRAPH, IERROR)
    INTEGER COMM OLD, NNODES, INDEX(*), EDGES(*), REORDER, COMM GRAPH,
    IERROR

The function returns a handle to a new communicator to which the graph topology information is attached. If reorder = false then the rank of each process in the new group is identical to its rank in the old group. Otherwise, the function may reorder the processes, so as to choose a good embedding of the virtual topology onto the physical machine. If the size nnodes of the graph is smaller than the size of the group of comm, then a process which is not a member of the grid is returned a null handle. The call is erroneous if it specifies a graph that is larger than the group size.

The three parameters nnodes, index and edges define the graph structure. nnodes is the number of nodes of the graph. The nodes are numbered from 0 to nnodes-1.

The i-th entry of array index stores the total number of neighbors of the first i graph nodes. The lists of neighbors of nodes 0, 1, …, nnodes-1 are stored in consecutive locations in array edges.

The definitions of the arguments nnodes, index, and edges are illustrated with the following simple example:

Assume there are four processes 0, 1, 2, 3 with the following adjacency matrix:

process neighbors
0  1, 3
1  0
2  3
3  0, 2

Then, the input arguments are:

nnodes = 4
index = 2, 3, 4, 6
edges = 1, 3, 0, 3, 0, 2

Thus, in C, index[0] is the degree of node zero, and index[i] - index[i-1] is the degree of node i, i=1, …, nnodes-1; the list of neighbors of node zero is stored in edges[j], 0 ≤ j ≤ index[0] − 1 and the list of neighbors of node i, i > 0, is stored in

edges[j], index[i − 1] ≤ j ≤ index[i] − 1.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

In Fortran, index[1] is the degree of node zero, and index[i+1] - index[i] is the degree of node i, i=1, …, nnodes-1; the list of neighbors of node zero is stored in edges[j], 1 ≤ j ≤ index[1] and the list of neighbors of node i, i > 0, is stored in

edges[j], index[i] + 1 ≤ j ≤ index[i + 1].

The total number of entries in index is nnodes and the total number of entries in edges is equal to the number of graph edges.

Implementation note: The following topology information is likely to be stored with a communicator:

1. Type of topology (Cartesian/graph)

2. For a Cartesian topology:

• ndims (number of dimensions),

• dims (numbers of processes per coordinate direction),

• periods (periodicity information),

• own position (own position in grid, could also be computed from rank and dims)

3. For a graph topology:

• index, edges (vectors defining graph structure)

For a graph structure the number of nodes is equal to the number of processes in the group. Therefore, it does not have to be stored explicitly. An additional zero entry at the start of array index will simplify access to the topology information.

6.5.1 Topology inquiry functions

If a topology has been defined with one of the above functions, the topology information can be looked up using inquiry functions. They all are local calls, and do not require any communication.

The function MPI TOPO STATUS returns the type of topology which is assigned to a communicator.

MPI TOPO STATUS(comm, status)

IN  comm  communicator (handle)
OUT  status  topology type of communicator comm (choice)

int MPI Topo status(MPI comm comm, int *status)
MPI TOPO STATUS(COMM, STATUS, IERROR)
    INTEGER COMM, STATUS, IERROR

The output value status is one of the following:

MPI GRAPH graph topology

MPI CART Cartesian topology

MPI UNDEFINED no topology

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Functions MPI GET GRAPHDIMS and MPI GET GRAPH retrieve the graph topology information that was associated with a communicator by MPI MAKE GRAPH.

MPI GET GRAPHDIMS(comm, nnodes, nedges)

IN  comm  communicator for group with graph structure (handle)
OUT  nnodes  number of nodes in graph (integer)
OUT  nedges  number of edges in graph (integer)

int MPI Get graphdims(MPI comm comm, int *nnodes, int *nedges)
MPI GET GRAPHDIMS(COMM, NNODES, NEDGES, IERROR)
    INTEGER COMM, NNODES, NEDGES, IERROR

The information provided by MPI GET GRAPHDIMS can be used to dimension the vectors index and edges long enough in the following call to MPI GET GRAPH.

MPI GET GRAPH(comm, maxindex, maxedges, index, edges)

IN  comm  communicator for group with graph structure (handle)
IN  maxindex  length of vector index in the calling program

(integer)

IN  maxedges  length of vector edges in the calling program

(integer)

OUT  index  array of integers containing the graph structure (for

details see the definition of MPI MAKE GRAPH)

OUT  edges  array of integers containing the graph structure

int MPI Get graph(MPI comm comm, int maxindex, int maxedges, int *index,
             int *edges)
MPI GET GRAPH(COMM, MAXINDEX, MAXEDGES, INDEX, EDGES, IERROR)
    INTEGER COMM, MAXINDEX, MAXEDGES, INDEX(*), EDGES(*), IERROR

The functions MPI GET CARTDIM and MPI GET CART return the Cartesian topology information that was associated with a communicator by the function MPI MAKE CART.

MPI GET CARTDIM(comm, ndims)

IN  comm  communicator for group with Cartesian structure (han-

dle)

OUT  ndims  number of dimensions of the Cartesian structure (in-

teger)

int MPI Get cartdim(MPI comm comm, int *ndims)
MPI GET CARTDIM(COMM, NDIMS, IERROR)
    INTEGER COMM, NDIMS, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GET CART(comm, maxdims, dims, periods, coords)

IN  comm  communicator for group with Cartesian structure (han-

dle)

IN  maxdims  length of vectors dims, periods, and coords in the

calling program (integer)

OUT  dims  number of processes for each Cartesian dimension (ar-

ray of integer)

OUT  periods  periodicity (true/false) for each Cartesian dimension

(array of logical)

OUT  coords  coordinates of calling process in Cartesian structure

(array of integer)

int MPI Get cart(MPI comm comm, int maxdims, int *dims, int *periods,
             int *coords)
MPI GET CART(COMM, MAXDIMS, DIMS, PERIODS, COORDS, IERROR)
    INTEGER COMM, MAXDIMS, DIMS(*), PERIODS(*), COORDS(*), IERROR

For a process group with Cartesian structure, the function MPI CART RANK translates the logical process coordinates to process ranks as they are used by the point–to–point routines:

MPI CART RANK(comm, coords, rank)

IN  comm  communicator for group with Cartesian structure (han-

dle)

IN  coords  integer array (of size ndims) specifying the Cartesian

coordinates of a process

OUT  rank  rank of specified process (integer)

int MPI Cart rank(MPI comm comm, int *coords, int *rank)
MPI CART RANK(COMM, COORDS, RANK, IERROR)
    INTEGER COMM, COORDS(*), RANK, IERROR

If for dimension i with periods(i) = true the coordinate coords(i) is out of range, i.e., coords(i) < 0 or coords(i) ≥ dims(i), it is shifted back to the interval 0 ≤ coords(i) ≤ dims(i) automatically. Out–of–range coordinates are erroneous for aperiodic dimensions.

The inverse rank-to-coordinates translation is provided by the function:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI CART COORDS(comm, rank, maxdims, coords)

IN  comm  communicator for group with Cartesian structure (han-

dle)

IN  rank  rank of a process within group of comm (integer)
IN  maxdims  length of vector coord in the calling program (integer)
OUT  coords  integer array (of size ndims) containing the Cartesian

coordinates of specified process (integer)

int MPI Cart coords(MPI comm comm, int rank, int maxdims, int *coords)
MPI CART COORDS(COMM, RANK, MAXDIMS, COORDS, IERROR)
    INTEGER COMM, RANK, MAXDIMS, COORDS(*), IERROR

The following functions provide “adjacency” information for a general graph topology.

MPI GRAPH NEIGHBORS COUNT(comm, rank, nneighbors)

IN  comm  communicator for a group with graph topology (han-

dle)

IN  rank  rank of process in group (integer)
OUT  nneighbors  number of neighbors of specified process (integer)

int MPI Graph neighbors count(MPI comm comm, int rank, int *nneighbors)
MPI GRAPH NEIGHBORS COUNT(COMM, RANK, NNEIGHBORS, IERROR)
    INTEGER COMM, RANK, NNEIGHBORS, IERROR

MPI GRAPH NEIGHBORS(comm, rank, maxneighbors, neighbors)

IN  comm  communicator for a group with graph topology (han-

dle)

IN  rank  rank of process in group (integer)
OUT  maxneighbors  size of array neighbors (integer)
OUT  neighbors  ranks of processes that are neighbors to specified pro-

cess (array of integer)

int MPI Graph neighbors(MPI comm comm, int rank, int *maxneighbors,
             int *neighbors)
MPI GRAPH NEIGHBORS(COMM, RANK, MAXNEIGHBORS, NEIGHBORS, IERROR)
    INTEGER COMM, RANK, MAXNEIGHBORS, NEIGHBORS(*), IERROR

Example: Suppose that comm is a communicator for a group with a shuffle-exchange topology: The group has 2^{n} members. Each process a_{1}, …, a_{n} (a_{i} ∈ {0, 1}) has three neighbors: exchange(a_{1}, …, a_{n}) = a_{1}, …, a_{n−1}, ā_{n} (ā = 1−a), shuffle(a_{1}, …, a_{n}) = a_{2}, …, a_{n}, a_{1}, and unshuffle(a_{1}, …, a_{n}) = a_{n}, a_{1}, …, a_{n−1}. The graph adjacency list is illustrated below for n = 3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

node  exchange  shuffle  unshuffle
neighbors(1) neighbors(2) neighbors(3)
0 (000)  1  0  0
1 (001)  0  2  4
2 (010)  3  4  1
3 (011)  2  6  5
4 (100)  5  1  2
5 (101)  4  3  6
6 (110)  7  5  3
7 (111)  6  7  7

C  assume: each process has stored a real number A.
C  extract neighborhood information
      CALL MPI_COMM_RANK(comm, myrank, ierr)
      CALL MPI_GRAPH_NEIGHBORS(comm, myrank, 3, neighbors, ierr)
C  perform exchange permutation
      CALL MPI_SENDRECV_REPLACE(A, 1, MPI_REAL, neighbors(1), 0,
     +     neighbors(1), 0, comm, status, ierr)
C  perform shuffle permutation
      CALL MPI_SENDRECV_REPLACE(A, 1, MPI_REAL, neighbors(2), 0,
     +     neighbors(3), 0, comm, status, ierr)
C  perform unshuffle permutation
      CALL MPI_SENDRECV_REPLACE(A, 1, MPI_REAL, neighbors(3), 0,
     +     neighbors(2), 0, comm, status, ierr)

6.5.2 Cartesian shift coordinates

If the process topology is a Cartesian structure, a collective shift operation is most likely to be used along a coordinate direction. In MPI shift operations are handled by the more general send–receive functions. As input they take the rank of a source process for the receive, and the rank of a destination process for the send.

If the function MPI CART SHIFT is called for a Cartesian process group, it provides the calling process with the above identifiers, which then can be passed to the send–receive function. The user specifies the coordinate direction for the shift and the step length. The function is local.

MPI CART SHIFT(comm, direction, disp, rank source, rank dest)

IN  comm  communicator for group with Cartesian structure (han-

dle)

IN  direction  coordinate dimension of shift (integer)
IN  disp  displacement (> 0: upwards shift, < 0: downwards

shift) (integer)

OUT  rank source  rank of source process (integer)
OUT  rank dest  rank of destination process (integer)

int MPI Cart shift(MPI comm comm, int direction, int disp, int *rank source,
             int *rank dest)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI CART SHIFT(COMM, DIRECTION, DISP, RANK SOURCE, RANK DEST, IERROR)
    INTEGER COMM, DIRECTION, DISP, RANK SOURCE, RANK DEST, IERROR

Depending on the periodicity of the Cartesian group in the specified coordinate direction, MPI CART SHIFT provides the identifiers for a circular or an end–off shift. In the case of an end–off shift, the value MPI PROCNULL may be returned in rank source or rank dest, indicating that the source or the destination for the shift is out of range.

Example: The communicator comm is associated with a two-dimensional periodic Cartesian topology. A two-dimensional array of REALs is stored one element per process, in variable A. One wishes to skew this array, by shifting column i by i.

....
C find process rank
      CALL MPI_COMM_RANK(comm, rank, ierr, ierr))
C find Cartesian coordinates
      CALL MPI_CART_COORD(comm, rank, coords, ierr)
C compute shift source and destination
      CALL MPI_CART_SHIFT(comm, 2, coords(2), source, dest, ierr)
C skew array
      CALL MPI_SENDRECV_REPLACE(A, 1, MPI_REAL, dest, 0, source, 0, comm,
     +                          status, ierr)

6.5.3 Partitioning of Cartesian structures

If a Cartesian topology has been created with MPI MAKE CART, the function MPI CART SUB can be used to partition the communicator group into subgroups which form lower–dimensional Cartesian subgrids, and to build for each subgroup a communicator with the associated subgrid Cartesian topology.

MPI CART SUB(comm, remain dims, comm new)

IN  comm  communicator for group with Cartesian structure (han-

dle)

IN  remain dims  the i-th entry of remain dims specifies whether the

i-th dimension is kept in the subgrid (true) or is dropped (false) (logical vector)

OUT  comm new  communicator containing the subgrid which includes

the calling process (handle)

int MPI Cart sub(MPI comm comm, int *remain dims, MPI comm *comm new)
MPI CART SUB(COMM, REMAIN DIMS, COMM NEW, IERROR)
    INTEGER COMM, REMAIN DIMS(*), COMM NEW, IERROR

Example: Assume that MPI MAKE CART(..., comm) has defined a (2×3×4) grid. Let remain dims = (true, false, true). Then a call to MPI CART SUB(comm, remain dims, comm new) will create communicators for three new groups with eight processes, each with a 2 × 4 Cartesian topology. If remain dims = (false, false, true) then the call to MPI CART SUB(comm, remain dims, comm new) will create communicators for six new groups with four processes, each with a one-dimensional Cartesian topology.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

6.5.4 Low–level topology functions

The two new functions introduced in this section can be used to implement all other topology functions. In general they will not be called by the user directly.

MPI MAP CART(comm old, ndims, dims, periods, newrank)

IN  comm old  communicator for parent group (handle)

IN  ndims  number of dimensions of Cartesian structure (integer)

IN  dims  integer array of size ndims specifying the number of

processes in each coordinate direction

IN  periods  logical array of size ndims specifying the periodicity

specification in each coordinate direction

OUT  newrank  reordered rank of the calling process; MPI UNDEFINED

if calling process does not belong to grid (integer)

int MPI Map cart(MPI comm comm old, int ndims, int *dims, int *period,
             int *newrank)
MPI MAP CART(COMM OLD, NDIMS, DIMS, PERIOD, NEWRANK, IERROR)
    INTEGER COMM OLD, NDIMS, DIMS(*), PERIOD(*), NEWRANK, IERROR

This function computes an optimal placement for the calling process on the physical machine. A possible implementation of this function is to always return the rank of the calling process, i.e., not to perform any reordering.

The function MPI MAKE CART(comm old, ndims, dims, periods, reorder, comm cart), with reorder = true can be implemented by calling MPI MAP CART(comm old, ndims, dims, periods, newrank), then calling MPI COMM SPLIT(comm, color, key, comm cart), with

color = 0 if newrank ≠ MPI UNDEFINED, color = MPI UNDEFINED otherwise, and key =
newrank.

The function MPI CART SUB(comm, remain dims, comm new) can be implemented by a call to MPI COMM SPLIT(comm, color, key, comm new), using a single number encoding of the lost dimensions as color and a single number encoding of the preserved dimensions

as key.

All other Cartesian topology functions can be implemented locally, using the topology information that is cached with the communicator.

The corresponding new function for general graph structures is

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI MAP GRAPH(comm old, nnodes, index, edges, newrank)

IN  comm old  communicator for parent group (handle)
IN  nnodes  number of graph nodes (integer)
IN  index  integer array specifying the graph structure, see

MPI MAKE GRAPH

IN  edges  integer array specifying the graph structure
OUT  newrank  reordered rank of the calling process; MPI UNDEFINED

if the calling process does not belong to graph (integer)

int MPI Map graph(MPI comm comm old, int nnodes, int *index, int *edges,
             int *newrank)
MPI MAP GRAPH(COMM OLD, NNODES, INDEX, EDGES, NEWRANK, IERROR)
    INTEGER COMM OLD, NNODES, INDEX(*), EDGES(*), NEWRANK, IERROR

The function MPI MAKE GRAPH(comm old, nnodes, index, edges, reorder, comm graph), with reorder = true can be implemented by calling MPI MAP GRAPH(comm old, nnodes, index, edges, newrank), then calling MPI COMM SPLIT(comm, color, key, comm graph), with

color = 0 if newrank ≠ MPI UNDEFINED, color = MPI UNDEFINED otherwise, and key =
newrank.

All other graph topology functions can be implemented locally, using the topology information that is cached with the communicator.

6.6 A simple example

The example in figure 6.1 shows how the grid definition and inquiry functions can be used in an application program. A partial differential equation, for example the Poisson equation, is to be solved on a rectangular domain. First the processes organize themselves in a two– dimensional structure. Each process then inquires for the ranks of its neighbors in the four directions. The numerical problem is solved by an iterative method, the details of which are hidden in the subroutine relax.

In each relaxation step each process computes new values for the solution grid function at all points owned by the process. Then the values at interprocess boundaries have to be exchanged with neighboring processes. For example, the exchange subroutine might contain a call like MPI SEND(...,neigh rank(1),...) to send updated values to the left–hand neighbor (i-1,j). Note that there is only local communication in this exchange operation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

integer ndims, num neigh
logical reorder
parameter (ndims=2, num neigh=4, reorder=.true.)
integer comm, comm cart, dims(ndims), neigh def(ndims)
integer neigh rank(num neigh), own position(ndims), i, j
logical periods(ndims)
real∗8 u(0:101,0:101), f(0:101,0:101)
data dims / ndims ∗ 0 /
comm = MPI COMM WORLD
C    Set process grid size and periodicity
call MPI MAKDIM(comm, ndims, dims)
periods(1) = .TRUE.
periods(2) = .TRUE.
C    Create a grid structure in WORLD group and inquire about own position
call MPI MAKE CART (comm, ndims, dims, periods, reorder, comm cart)
call MPI GET CART (comm cart, ndims, dims, periods, own position)
C    Look up the ranks for the neighbors.  Own process coordinates are (i,j).
C    Neighbors are (i-1,j), (i+1,j), (i,j-1), (i,j+1)
i = own position(1)
j = own position(2)
neigh def(1) = i-1
neigh def(2) = j
call MPI CART RANK (comm cart, neigh def, neigh rank(1))
neigh def(1) = i+1
neigh def(2) = j
call MPI CART RANK (comm cart, neigh def, neigh rank(2))
neigh def(1) = i
neigh def(2) = j-1
call MPI CART RANK (comm cart, neigh def, neigh rank(3))
neigh def(1) = i
neigh def(2) = j+1
call MPI CART RANK (comm cart, neigh def, neigh rank(4))
C    Initialize the grid functions and start the iteration
call init (u, f)
do 10 it=1,100
  call relax (u, f)
C    Exchange data with neighbor processes
  call exchange (u, comm cart, neigh rank, num neigh)
10   continue
call output (u)
end

Figure 6.1: Set–up of process structure for two–dimensional parallel Poisson solver.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 7

MPI Environmental Management

This chapter discusses routines for getting and, where appropriate, setting various parameters that relate to the MPI implementation and the execution environment. (such as error handling and communication buffering). The procedures for entering and leaving the MPI execution environment are also dexcribed here.

7.1 Implementation information

7.1.1 Environmental Inquiries

The following routines allow to inquire on the execution environment of an MPI program.

MPI GET VALID TAG RANGE ( low, high )

OUT  low  Smallest valid tag value (integer)
OUT  high  Largest valid tag value (integer)

int MPI Get valid tag range(int *low, int *high)
MPI GET VALID TAG RANGE(LOW, HIGH, IERROR)
    INTEGER LOW, HIGH, IERROR

This routine gets the range of valid tags; that is, the user may use any tag value from low to high inclusive.

MPI GET PROCESSOR NAME( len, name )

IN  len  Length (in characters) of the space in name
OUT  name  A unique specifier for the actual (as opposed to vir-

tual) node.

int MPI Get processor name(int len, char *name)
MPI GET PROCESSOR NAME(NAME, IERROR)
    CHARACTER*(*) NAME
    INTEGER IERROR

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The name is a character string for maximum flexibility. From this value it must be possible to identify a specific piece of hardware; possible values include “processor 9 in rack 4 of mpp.cs.org” and “231” (where 231 is the actual processor number in the running homogeneous system). A error code of MPI UNDEFINED is allowed but strongly discouraged. The value of MPI MAX PROCESSOR NAME is the maximum length required by MPI GET PROCESSOR NAME (including, in the C binding, the terminating null).

MPI GET HOST( comm, rank)

IN  comm  communicator (handle)
OUT  rank  rank of HOST in group of comm; MPI PROCNULL if

there is no HOST (integer)

int MPI Get host(MPI Comm comm, int *rank)
MPI GET HOST(COMM, RANK, IERROR)
    INTEGER COMM, RANK, IERROR

This routines gets the rank of the HOST process in the group associated with communicator comm, if there is such. MPI does not specify what it means for a process to be a HOST, nor does it requires that a HOST exists.

7.1.2 Buffering

While it is possible to design algorithms that do not require that any buffering be provided for sending and receiving “blocking” messages, many practitioners expect that the messagepassing system will provide some buffering. That is, there is often an expectation that the program fragment, executed by all tasks,

MPI_send( buf, size, datatype, nbr, tag, comm )
MPI_recv( buf, size, datatype, nbr, tag, comm )

will execute for messages whose size is not too large. The purpose of this section is to give the user a way to determine what “too large” is and to help the implementation provide enough buffering for the user’s need.

Note that nothing in this section requires an implementation to provide any buffering; that is, an implementation may define the above program fragment to be erroneous for any positive value of size.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

General buffering parameters

MPI GET BUFFER PARAMS( numsend, sizesend, numrecv, sizerecv, totalnumsend, totalsizesend, totalnumrecv, totalsizerecv)

OUT  numsend  Bound on the number of pending messages with source

at calling process and same destination

OUT  sizesend  Bound on the total size in bytes of pending messages

with source at calling process and same destination

OUT  numrecv  Bound on the number of pending messages with des-

tination at calling process and same source

OUT  sizerecv  Bound on the total size in bytes of pending messages

with destination at calling process and same source

OUT  totalnumsend  Bound on the number of pending messages with source

at the calling process and any destination

OUT  totalsizesend  Bound on the total size in bytes of pending messages

with source at the calling process and any destination

OUT  totalnumrecv  Bound on the number of pending messages from any

source with the calling process as destination

OUT  totalsizerecv  Bound on the total size in bytes of pending messages

from any source with destination at the calling process

int MPI Get buffer params(int *numsend, int *sizesend, int *numrecv,
             int *sizerecv, int *totalnumsend, int *totalsizesend,
             int *totalnumrecv, int *totalsizerecv)
MPI GET BUFFER PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND,
             TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR)
    INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND,
    TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR

A program where at any point in time no process has more then numsend pending messages sent by it to the same destination, no more than sizesend bytes of data in pending messages sent by it to the same destination, no more than totalnumsend pending messages sent by it to any destination, no more than totalsizesend bytes of data in pending messages sent by it to any destination, no more than numrecv pending messages sent to it from the same source, no more than sizerecv bytes in pending messages sent to it from the same source, no more than totalnumrecv pending messages sent to it from any source, and no more than totalsizerecv bytes in pending messages sent to it from any source is guaranteed not to deadlock because of the lack of buffer space for pending messages.

Examples:

1. Suppose that pending messages are always buffered at the sender, all buffers are shared, total buffer size at the process is B, each pending message takes h bytes of storage, in addition to the storage for its data, and no storage is lost due to fragmentation. Then possible return values for MPI GET BUFFER PARAMS are numsend = totalnumsend = k,

sizesend = totalsizesend = B-kh, and numrecv = totalnumrecv = sizerecv = totalsizerecv

= INT MAX, for some 0 < k < B/h.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

2. Suppose that messages are always buffered at the receiver, and that a separate buffer with C bytes is allocated for each pair of communicating processes; entries for pending messages are kept in a seeparate table, with room for total of m pending messages at each receiving process; the total number of processes is p, and no storage is lost due to fragmentation. Then possible return values for MPI GET BUFFER PARAMS are sendcount =

totalsendcount = sendsize = totalsendsize = INT MAX, recvcount = totalrecvcount
= m, recsize = C and totalrecvsize = (p-1)×C.

The call may return the value MPI UNDEFINED in any argument. If this is the case, then the MPI implementation makes no guarantee about buffering.

Some implementations may be able to adjust their buffering. In addition, since the parameters returned by MPI GET BUFFER PARAMS represent a guarantee about a particular choice of number of buffers, it may not guarantee buffering that it can in fact perform. Thus, in example 1 above, the implementation picks an arbitrary value for k, trading off number of messages for total message size, rather than letting the user make this tradeoff. In order to allow the MPI user to inquire about a specific buffering need, the routine MPI SUGGEST BUFFER PARAMS is provided.

MPI SUGGEST BUFFER PARAMS( numsend, sizesend, numrecv, sizerecv, totalnumsend, totalsizesend, totalnumrecv, totalsizerecv, flag)

IN  numsend  Bound on number of pending messages with source at

calling process and same destination

IN  sizesend  Bound on total size in bytes of pending messages with

source at calling process and same destination

IN  numrecv  Bound on the number of pending messages with des-

tination at calling process and same source

IN  sizerecv  Bound on total size in bytes of pending messages with

destination at calling process and same source

IN  totalnumsend  Bound on the number of pending messages with source

at the calling process and any destination

IN  totalsizesend  Bound on total size in bytes of pending messages with

source at the calling process and any destination

IN  totalnumrecv  Bound on the number of pending messages from any

source with the calling process as destination

IN  totalsizerecv  Bound on the total size in bytes of pending messages

from any source with destination at the calling process

OUT  flag  TRUE if the suggestion was accepted, FALSE, other-

wise

int MPI Suggest buffer params(int numsend, int sizesend, int numrecv,
             int sizerecv, int totalnumsend, int totalsizesen,
             int totalnumrecv, int totalsizerecv, int *flag)
MPI SUGGEST BUFFER PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV,
             TOTALNUMSEND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, FLAG,
             IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

    INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEND,
    TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR
    LOGICAL FLAG

If a value of true is returned in flag then it is as if the values provided by the user in MPI SUGGEST BUFFER PARAMS were returned by a call to MPI GET BUFFER PARAMS: the user is guaranteed that no deadlock will occur due to lack of buffers, provided it never exceeds the limitations on number and size of pending messages implied by these parameters.

This routines allows the user to tell the MPI implementation that a certain amount of buffering would be desireable, The MPI implementation is not required to provide such buffering.

It is expected that a common use of these routines will be

MPI_Get_buffer_params( &sendnum, &sendsize, ... );
if (sendsize < buffer_needed) {
    fprintf( stderr, "Program requires more buffering than is available\n" );
    MPI_Abort( 1 );
    }
else
    exploit buffering

This allows a program to fail rather than deadlock. It is also possible for a program to switch to a different algorithm that requires less or no buffering.

Another use is to inform the MPI implementation of the buffer space desires of the application:

MPI_Suggest_buffer_params( 1, buffer_needed, 1, buffer_needed, 0, 0, 0,
       0, &flag );
if (flag) {
    exploit buffering
    }
else {
    fprintf( stderr, "Program requires more buffering than is available\n" );
    MPI_abort( 1 );
    }

Application provided buffering

An application program can optionally provide space for MPI’s use in buffering message data via the following call:

MPI USER SPECIFIES BUFFER( comm, buffer, size)

IN  comm  is a communicator with which the buffer is to be as-

sociated (handle)

IN  buffer  initial buffer address (choice)
IN  size  buffer size, in bytes (integer)

int MPI User specifies buffer(MPI Comm comm, void* buffer, int size)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI USER SPECIFIES BUFFER(COMM, BUFFER, SIZE, IERROR)
    <type> BUFFER(*)
    INTEGER COMM, SIZE, IERROR

If MPI USER SPECIFIES BUFFER is called and the value returned in flag is MPI OK, then for subsequent regular sends using the specified communicator, MPI must provide as much safety as if outgoing message data were buffered by the sending process, in the specified buffer space, using a circular contiguous-space allocation policy. The buffer is managed by MPI.

Access to the provided buffer space is not inherited by communicators derived from the initially specified one.

Implementation note: One approach is for MPI to simply transform all blocking sends on the specified communicator into

• copy data to buffer space

• issue non-blocking send from the buffer copy

• return to application

• check completion on subsequent MPI call(s)

No doubt many optimizations within MPI are possible—the proposal just requires that MPI provide at least this much safety.

The reason that MPI provides the buffer rather than the user is that once the user frees a communicator, only MPI knows when all data-structures associated with that communicator can be freed.

Note that collective communications and other nonlocal operations will share suggested or user-supplied communication buffers with point to point communication.

7.2 Error handling

This section contains the suggested POSIX binding for error handling in MPI. MPI implementations that do not run in a POSIX-compliant environment may need to use alternative mechanisms.

An MPI implementation cannot or may choose not to handle some errors that occur during MPI calls. These can include errors that generate exceptions or traps, such as floating point errors or access violations. The set of errors that are not handled by MPI is implementation-dependent.

There are three possible error handling modes for those errors that MPI handles, which are identified by three constant values:

MPI ERRORSFATAL The error is fatal and causes the program to abort.

MPI ERRORSRETURN The error causes the MPI routine where it occurred to return an error code.

MPI ERRORSCALLUSERROUTINE The error causes MPI to call a user-specified error handling routine. Such a handler can be specified by a call to MPI SET HANDLER.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

A user can associate an error handling mode with a communicator. The specified error handling mode will be used for any error that occurs during a call to MPI for a communication with this communicator. MPI calls that are not related to any communicator are considered to be attached to the communicator MPI COMM WORLD.

A user may also attach an error handling routine to a communicator; This attachement is purely local: different processes may attach different error handlers to the same communicator.

A newly created communicator inherits the error mode and the error handling routine, if any, that is associated with the “parent” communicator. In particular, the user can specify a “global” error-handling mode for all communicators by associating this mode with the communicator MPI COMM WORLD immediatly after initialization.

The default error handling mode for MPI COMM WORLD is MPI ERRORSFATAL.

MPI GET ERRORMODE ( comm, mode )

IN  comm  communicator to get error mode from (handle)
OUT  mode  current error handling mode (state)

int MPI Get errormode(MPI Comm comm, MPI Errmode *mode)
MPI GET ERRORMODE(COMM, MODE, IERROR)
    INTEGER COMM, MODE, IERROR

Gets the error handling mode.

MPI SET ERRORMODE ( comm, mode )

IN  comm  communicator to set error mode for (handle)
IN  mode  new error mode (state)

int MPI Set errormode(MPI Comm comm, MPI Errmode mode)
MPI SET ERRORMODE(COMM, MODE, IERROR)
    INTEGER COMM, MODE, IERROR

Sets the error handling mode.

MPI SET ERRHANDLER( comm, handler )

IN  comm  communicator to set the error handler for (handle)
IN  handler  user-defined error handler to call

int MPI Set errhandler(MPI Comm comm, void handler)
MPI SET ERRHANDLER(COMM, HANDLER, IERROR)
    EXTERNAL HANDLER
    INTEGER COMM, IERROR

This routine returns the previous error handler as its value.

An error handler is a C function declared as

typedef void (MPI_ERRHANDLER)(MPICOMM, int *, ...);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The first argument is the communicator in use. The second is a pointer to the error code to be returned by the MPI routine. The remaining arguments are “varargs” arguments whose number and meaning is implementation-dependent. An implementation should clearly document these arguments.

There are two predefined error handlers that the user may use instead of writing their own:

MPI DEFAULTHANDLER aborts from within MPI.

MPI NOHANDLER returns the error value and has no further effect.

MPI ERROR STRING( errorcode, len, string )

IN  errorcode  Error code returned by an MPI routine
IN  len  Length of string
OUT  string  Text that corresponds to the errorcode

int MPI Error string(int errorcode, int len, char *string)
MPI ERROR STRING(ERRORCODE, STRING, IERROR)
    INTEGER ERRORCODE, IERROR
    CHARACTER*(*) STRING

7.3 Startup

One goal of MPI is to achieve source code portability. By this we mean that a program written using MPI and complying with the relevant language standards is portable as written, and must not require any source code changes when moved from one system to another. This explicitly does not say anything about how an MPI program is started or launched from the command line, nor what the user must do to set up the environment in which an MPI program will run. However, an implementation may require some setup to be performed before other MPI routines may be called. To provide for this, MPI includes an initialization routine MPI INIT.

MPI INIT()

int MPI Init(int *argc, char **argv)
MPI INIT(IERROR)
    INTEGER IERROR

This routine must be called before any other MPI routine. It must be called at most once; subsequent calls are erroneous (see MPI INITIALIZED).

All MPI programs must contain a call to MPI init; this routine must be called before any other MPI routine (apart from MPI INITIALIZED) is called. The version for ANSI C accepts the argc and argv that are provided by the arguments to main:

MPI_init( argc, argv );

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The Fortran version takes no arguments.

MPI FINALIZE()

int MPI Finalize()
MPI FINALIZE(IERROR)
    INTEGER IERROR

This routines cleans all MPI state. Once this routine is called, no MPI routine (even MPI INIT) may be called. The user must ensure that all pending communications involving a process complete before the process calls MPI FINALIZE.

MPI INITIALIZED( flag )

OUT  flag  Flag is true if MPI INIT has been called and false

otherwise.

int MPI Initialized(int *flag)
MPI INITIALIZED(FLAG, IERROR)
    LOGICAL FLAG
    INTEGER IERROR

This routine may be used to determine whether MPI INIT has been called. It is the only routine that may be called before MPI INIT is called.

MPI ABORT( comm, errorcode )

IN  comm  communicator of tasks to abort
IN  errorcode  error code to return to invoking environment

int MPI Abort(MPI Comm comm, int errorcode)
MPI ABORT(COMM, ERRORCODE, IERROR)
    INTEGER COMM, ERRORCODE, IERROR

This routine makes a “best attempt” to abort all tasks in the group of comm. This function does not require that the invoking environment take any action with the error code. However, a Unix or POSIX environment should handle this as a return errorcode from the main program or an abort(errorcode).

MPI implementations are required to define the behavior of MPI ABORT at least for a

comm of MPI COMM WORLD.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 8

Profiling Interface

8.1 Requirements

To meet the MPI profiling interface, an implementation of the MPI functions must

1. provide a mechanism through which all of the MPI defined functions may be accessed with a name shift. Thus all of the MPI functions (which normally start with the prefix “MPI ”) should also be accessable with the prefix “PMPI ”.

2. ensure that those MPI functions which are not replaced may still be linked into an executable image without causing name clashes.

3. document the implementation of different language bindings of the MPI interface if they are layered on top of each other, so that the profiler developer knows whether she must implement the profile interface for each binding, or can economise by implementing it only for the lowest level routines.

8.2 Discussion

The objective of the MPI profiling interface is to ensure that it is relatively easy for authors of profiling (and other similar) tools to interface their codes to MPI implementations on different machines.

Since MPI is a machine independent standard with many different implementations, it is unreasonable to expect that the authors of profiling tools for MPI will have access to the source code which implements MPI on any particular machine. It is therefore necessary to provide a mechanism by which the implementors of such tools can collect whatever performance information they wish without access to the underlying implementation.

We believe that having such an interface is important if MPI is to be attractive to end users, since the availability of many different tools will be a significant factor in attracting users to the MPI standard.

The profiling interface is just that, an interface. It says nothing about the way in which it is used. There is therefore no attempt to lay down what information is collected through the interface, or how the collected information is saved, filtered, or displayed.

While the initial impetus for the development of this interface arose from the desire to permit the implementation of profiling tools, it is clear that an interface like that specified

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

may also prove useful for other purposes, such as “internetworking” multiple MPI implementations. Since all that is defined is an interface, there is no objection to its being used wherever it is useful.

As the issues being addressed here are intimately tied up with the way in which executable images are built, which may differ greatly on different machines, the examples given below should be treated solely as one way of implementing the objective of the MPI profiling interface. The actual requirements made of an implementation are those detailed in the Requirements section above, the whole of the rest of this chapter is only present as justification and discussion of the logic for those requirements.

The examples below show one way in which an implementation could be constructed to meet the requirements on a Unix system (there are doubtless others which would be equally valid).

8.3 Logic of the design

Provided that an MPI implementation meets the requirements above, it is possible for the implementor of the profiling system to intercept all of the MPI calls which are made by the user program. She can then collect whatever information she requires before calling the underlying MPI implementation (through its name shifted entry points) to achieve the desired effects.

8.4 Examples

8.4.1 Profiler implementation

Suppose that the profiler wishes to accumulate the total number of times the MPI SEND function is called, along with the total elapsed time spent in the function. This could trivially be achieved as follows.

static int totalSendCount;
static int totalTime;
extern int now(void);   /* Some suitable timer function ! */
int MPI_Send(void* buf, int count, MPI_Datatype datatype, int dest,
             int tag, MPI_Comm comm)
{
   int tstart = now();
   /* Now call the real send, passing along all the arguments */
   int result = PMPI_Send(buf, count, datatype, dest, tag, comm);
   totalSendCount++;                    /* Accumulate byte count */
   totalTime  += now() - tstart;        /* and time              */
   return result;
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

8.4.2 MPI library implementation

On a Unix system, in which the MPI library is implemented in C, then there are various possible options, of which two of the most obvious are presented here. Which is better depends on whether the linker and compiler support weak symbols.

Systems with weak symbols

If the compiler and linker support weak external symbols (e.g. Solaris 2.x, other system V.4 machines), then only a single library is required through the use of #pragma weak thus :–

#pragma weak MPI_EXAMPLE = PMPI_EXAMPLE
int PMPI_EXAMPLE(/* appropriate args */)
{
    /* Useful content */
}

The effect of this #pragma is to define the external symbol MPI EXAMPLE as a weak definition, this means that the linker will not complain if there is another definition of the symbol (for instance in the profiling library), however if no other definition exists, then the linker will use the weak definition.

Systems without weak symbols

In the absence of weak symbols then one possible solution would be to use the C macro pre-processor thus :–

#ifdef PROFILELIB
#    ifdef __STDC__
#        define FUNCTION(name) P##name
#    else
#        define FUNCTION(name) P/**/name
#    endif
#else
#    define FUNCTION(name) name
#endif

Each of the user visible functions in the library would then be declared thus

int FUNCTION(MPI_EXAMPLE)(/* appropriate args */)
{
    /* Useful content */
}

The same source file can then be compiled to produce both versions of the library, depending on the state of the PROFILELIB macro symbol.

It is required that the standard MPI library be built in such a way that the inclusion of MPI functions can be achieved one at a time. This is a somewhat unpleasant requirement,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

since it may mean that each external function has to be compiled from a separate file. However this is necessary so that the author of the profiling library need only define those MPI functions which she wishes to intercept, references to any others being fulfilled by the normal MPI library. Therefore the link step can look something like this :–

% cc ... -lmyprof -lpmpi -lmpi

Here libmyprof.a contains the profiler functions which intercept some of the MPI functions. libpmpi.a contains the “name shifted” MPI functions, and libmpi.a contains the normal definitions of the MPI functions.

8.5 Multiple levels of interception

The scheme given here does not directly support the nesting of profiling functions, since it provides only a single alternative name for each MPI function. Consideration was given to an implementation which would allow multiple levels of call interception, however we were unable to construct an implementation of this which did not have the following disadvantages

• assuming a particular implementation language.

• imposing a run time cost even when no profiling was taking place.

Since one of the objectives of MPI is to permit efficient, low latency implementations, and it is not the business of a standard to require a particular implementation language, we decided to accept the scheme outlined above.

Note, however, that it is possible to use the scheme above to implement a multi-level system, since the function called by the user may call many different profiling functions before calling the underlying MPI function.

Unfortunately such an implementation may require more co-operation between the different profiling libraries than is required for the single level implementation detailed above.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Chapter 9

Initial Implementation Subset

9.1 Introduction

This chapter defines the minimal subset of MPI for initial implementation. This subset is being defined so that consistent implementations can appear more rapidly. It was recognized early in the process that MPI needed to appear as quickly as possible and practical. The creation of a subset will hopefully allow users earlier access to the standard and still allow for the writing of portable message passing codes.

This subset should not in any way be construed as a limitation on MPI implementations. It is strictly the minimum necessary to have an initial MPI subset conforming implementation. It is hoped that an officially sanctioned subset will encourage and allow implementors to introduce MPI in a more timely fashion. It should be noted, however, that implementation of the subset does not make an implementation conform with MPI. The subset is a potential first step in the process. All implementations shall conform to the entire standard; implementations are encouraged to do the full standard as rapidly as possible.

The subset presented is consistent with the complete MPI standard. This was an important goal so the additional MPI features could be added without changing any previous functionality from the user’s perspective. Thus, users can be assured that programs written now for the subset will run without modification under the full MPI standard. Furthermore, using additional features of the full MPI standard in the future will not require changes to code already written. This requires that the MPI functions have their full calling arguments in the subset. To achieve this goal, the subset will generally limit access to full MPI functionality through complete elimination of functions or limitations on the creation mechanisms for certain MPI input arguments.

9.2 Criteria and Rationale

Having the subset be consistent with the entire MPI standard was considered critical to the effort. In addition to this, there were many possible criteria to use in determining the elements of MPI to include in the subset. The main criteria used were that the MPI subset should:

1. Contain routines that are as close as possible to current standard practice to minimize

the effort to port codes.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

2. Contain new and important MPI features.

3. Allow developers to be able to have the subset show up as rapidly as possible while still meeting the other criteria.

There were many rationales for these criteria. It was recognized that it is impossible to come up with an ideal subset, and compromises were necessary. It was felt that current users should be comfortable in migrating to MPI. Thus, the subset should contain the message passing elements that represent common practice today. Furthermore, it was felt that the initial (and later) version of MPI should maintain efficiency; otherwise, users could suffer from disenchantment with MPI on their first usage and might never give the standard a second try. Having the common practice routines in the subset should encourage developers to optimize these routines in the initial version of MPI. Also, although some felt that the portability offered by the MPI subset was sufficient to entice users, many felt that the subset should contain some new features over and above current common practice as an added incentive. The additional items selected were meant to represent some of the significant new features of MPI. Balanced against these first two goals is the need for the subset to show up in a timely fashion. Thus, each feature chosen for inclusion in the subset was deemed of sufficient importance to outweigh its added complexity in implementing the subset. Although some functions seem easy to implement, there are often overlooked costs in testing, documentation, and development that were considered before a feature was added to the MPI subset. Inclusion of too many routines might lead to a moderately efficient implementation of them all instead of a very efficient and possibly more useful implementation of a smaller subset.

9.3 Subset Functionality

The following sections contain a general summary and specific routines of what is included in/excluded from the subset for each chapter. The general requirements of each chapter also hold for the subset. As examples, the message envelope in subsection 3.2.2, the semantics in section 3.4, the type matching in section 3.5, and the communication objects in subsection 3.8.1 are all part of the subset.

9.3.1 Point to Point Functionality

The point to point routines are not only central to MPI but represent common practice today. As such, the majority of the functionality in this chapter is included in the subset. Derived datatypes represent a significant new feature in MPI and is included in the subset. However, only basic datatypes of section 3.2.1 will be allowed as input to the datatype construction routines described in section 3.13.1.

To reduce the difficulty of implementation, several features were left out of the subset. The ability to test for multiple completions of messages is removed. Also, the ability to cancel a message and use persistent communication objects is removed. Finally, the routines associated with ready and synchronous communication modes are excluded.

The table below contains a listing by section of which routines from the point to point communication chapter are included in/excluded from the subset.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Section Title  Included  Excluded
3.2  Basic Send Operation MPI SEND  ——
3.2.1  Message Data  All types  ——
3.3  Basic Recv Operation MPI RECV  ——
3.3.1  Return Status  MPI GET SOURCE  ——

MPI GET TAG MPI GET COUNT

3.7 Comm. Modes Standard MPI RSEND

MPI SSEND

3.8.2 Comm. Initiation MPI ISEND MPI IRSEND
MPI IRECV  MPI ISSEND
3.8.3 Comm. Completion MPI WAIT ——

MPI TEST

3.8.5 Multiple Completion —— MPI WAITANY

MPI TESTANY MPI WAITALL MPI TESTALL

3.9 Probe & Cancel MPI PROBE MPI CANCEL
MPI IPROBE  MPI TEST CANCELLED
3.10 Persistent Comm. —— MPI CREATE SEND
Objects  MPI CREATE RSEND

MPI CREATE SSEND MPI CREATE RECV MPI START MPI STARTALL MPI REQUEST FREE

3.11  Send-receive  MPI SENDRECV  MPI SENDRECV REPLACE
3.12  Null process  MPI PROCNULL  ——
3.13.1 Datetype const.  MPI TYPE CONTIGUOUS See text above

MPI TYPE VECTOR MPI TYPE HVECTOR MPI TYPE INDEXED MPI TYPE HINDEXED MPI TYPE STRUCT

3.13.2 Address and extent  MPI ADDRESS
functions  MPI TYPE EXTENT

MPI TYPE SIZE MPI TYPE COUNT

3.13.3 Lower-bound and  MPI LB
upper-bound  MPI UB
3.13.4 Commit and free MPI TYPE COMMIT ——

MPI TYPE FREE

3.13.5 Use of General...  MPI GET ELEMENTS  ——

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

9.3.2 Collective Communication Functionality

As with point to point functionality, collective communication is common practice today, and the functionality is included in the subset. Furthermore, the ability to define arbitrary groups in the MPI subset, which are then associated with a communicator, allows the user to perform collective communication operations over an arbitrary set of processes. This increases the functionality of the collective communication routines over what is commonly available today.

To ease initial implementation, the user defined reduce functions are excluded from the subset. Note that all of the system provided reduce functions listed in section 4.5.1 are include in the subset. Also, the scan functions are excluded from the subset. Finally, the generalized “v” versions of routines are excluded from the subset as well as MPI REDUCE SCATTER.

The table below contains a listing by section of which routines from the collective communication chapter are included in/excluded from the subset.

Section Title  Included  Excluded
4.3  Barrier Synch.  MPI BARRIER  ——
4.4.1  Broadcast  MPI BCAST  ——
4.4.2  Gather  MPI GATHER  MPI GATHERV
4.4.4  Scatter  MPI SCATTER  MPI SCATTERV
4.4.6  Gather-to-all  MPI ALLGATHER MPI ALLGATHERV
4.4.7  All-to-All Scatter/Gather MPI ALLTOALL  MPI ALLTOALLV
4.5.1  Reduce  MPI REDUCE
4.5.2  User-Reduce  MPI USER REDUCE

MPI USER REDUCEA

4.5.3 All-Reduce MPI ALLREDUCE MPI USER ALLREDUCE

MPI USER ALLREDUCEA

4.5.4 Reduce-Scatter MPI REDUCE SCATTER

MPI USER REDUCE SCATTER MPI USER REDUCE SCATTERA

4.5.5 Scan —— MPI SCAN

MPI USER SCAN MPI USER SCANA

9.3.3 Groups, Contexts, Communicators, and Cacheing

Groups, contexts, and communicators are used throughout the MPI standard. Though these concepts are used in some packages today, their inclusion in MPI represents a significant new feature over what is generally available in common practice today. For these reasons, this functionality is included in the subset. However, implementation of groups, contexts, and communicators is also a significant effort. As a compromise, the MPI intracommunicator functionality is included, but the inter-communicator functionality is excluded. The cacheing facility was included to support the topology functionality and extensions to MPI.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The table below contains a listing by section of which routines from the Groups, Contexts, Communicators, and Cacheing chapter are included in/excluded from the subset.

Section Title  Included  Excluded
5.2.4 Predefined Comm. MPI COMM WORLD  ——
5.3.1 Group Accessors MPI GROUP SIZE  ——

MPI GROUP RANK

MPI GROUP TRANSLATE RANKS

5.3.2 Group Const. MPI GROUP UNION ——

MPI GROUP INTERSECTION MPI GROUP DIFFERENCE MPI GROUP INCL MPI GROUP EXCL MPI GROUP RANGE INCL MPI GROUP RANGE EXCL

5.3.3 Group Destruct. MPI GROUP FREE  ——
5.4.1 Comm. Access. MPI COMM SIZE  ——

MPI COMM RANK MPI COMM GROUP

5.4.2 Comm. Const. MPI COMM DUP ——

MPI COMM MAKE MPI COMM SPLIT

5.4.3 Comm. Destruct. MPI COMM FREE  ——
5.5.1 Comm. Access. ——  MPI COMM TEST INTER
5.5.2 Intercomm. Const. ——  MPI INTERCOMM START
& Destr.  MPI INTERCOMM FINISH

MPI INTER COMM MERGE

5.5.3 Name Service  ——  MPI INTERCOMM NAME START
5.6.1 Functionality  MPI ATTR GET KEY  ——

MPI ATTR FREE KEY MPI ATTR PUT VALUE MPI ATTR GET VALUE MPI ATTR DELETE VALUE

9.3.4 Topology Functionality

The capability presented in the process topology chapter represents a new functionality in MPI and is not generally common practice today. To make this available in the subset, the functionality associated with Cartesian topologies is included, but the general graph topologies are excluded.

The table below contains a listing by section of which routines from the topology chapter are included in/excluded from the subset.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Section Title  Included  Excluded
6.5  Topology Const.  MPI MAKE CART MPI MAKE GRAPH

MPI MAKE DIMS

6.5.1 Topol. inquiry func. MPI TOPO STATUS
MPI GET CARTDIM MPI GET GRAPHDIMS
MPI GET CART  MPI GET GRAPH
MPI CART RANK  MPI GRAPH NEIGHBOR COUNT
MPI CART COORD MPI GRAPH NEIGHBORS
6.5.2 Cart. Shift Coord. MPI CART SHIFT
6.5.3  Part. of Cart. Struct. MPI CART SUB
6.5.4  Low–level topol. func. MPI MAP CART  MPI MAP GRAPH

9.3.5 Language Binding

The subset shall contain the Fortran 77 and C language bindings. All routines included in the subset will conform to the rules stated in chapter 2 and annex A.

9.3.6 MPI Environmental Management

The ability for the user to make inquiries of the MPI system is an important feature in portable programming. For this reason, most of the functions are included in the subset. However, the ability for the user to specify an error handler is not included in the subset. This is consistent with not including user called functions in the subset.

The table below contains a listing by section of which routines from the MPI Environmental Management chapter are included in/excluded from the subset.

Section Title  Included  Excluded
7.1.1 Envir. Inquires MPI GET VALID TAG RANGE ——

MPI GET PROCESSOR NAME

MPI GET HOST

7.1.2 Buffering MPI GET BUFFER PARAMS MPI USER SPECIFIES BUFFER

MPI SUGGEST BUFFER PARAMS

7.2  Error handling MPI GET ERRORMODE
MPI SET ERRORMODE  MPI SET ERRHANDLER

MPI ERROR STRING

7.3 Startup MPI INIT ——

MPI FINALIZE MPI INITIALIZED MPI ABORT

9.3.7 Profiling Functionality

Profiling is a capability that will be of great utility in MPI for determining the performance of each implementation. Furthermore, although implementation of the profiling capabilities doubles the number of routines supplied, these routines can easily be generated from the routines required elsewhere in the subset. Given these considerations, the subset will contain the shadow routines “PMPI ” for each routine included in the subset.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

9.4 Subset Testing

The test suite is expected to include a subset that will verify the functionality of this subset.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Bibliography

[1] V. Bala and S. Kipnis. Process groups: a mechanism for the coordination of and communication among processes in the Venus collective communication library. Technical report, IBM T. J. Watson Research Center, October 1992. Preprint.

[2] V. Bala, S. Kipnis, L. Rudolph, and Marc Snir. Designing efficient, scalable, and portable collective communication libraries. Technical report, IBM T. J. Watson Research Center, October 1992. Preprint.

[3] A. Beguelin, J. Dongarra, A. Geist, R. Manchek, and V. Sunderam. Visualization and debugging in a heterogeneous environment. IEEE Computer, 26(6):88–95, June 1993.

[4] Luc Bomans and Rolf Hempel. The Argonne/GMD macros in FORTRAN for portable parallel programming and their implementation on the Intel iPSC/2. Parallel Computing, 15:119–132, 1990.

[5] R. Butler and E. Lusk. User’s guide to the P4 programming system. Technical Report TM-ANL–92/17, Argonne National Laboratory, 1992.

[6] Robert Calkin, Rolf Hempel, Hans-Christian Hoppe, and Peter Wypior. Portable programming with the parmacs message–passing library. Parallel Computing, special issue on message–passing interfaces, to appear.

[7] S. Chittor and R. J. Enbody. Performance evaluation of mesh–connected wormhole– routed networks for interprocessor communication in multicomputers. In Proceedings of the 1990 Supercomputing Conference, pages 647–656, 1990.

[8] S. Chittor and R. J. Enbody. Predicting the effect of mapping on the communication performance of large multicomputers. In Proceedings of the 1991 International Conference on Parallel Processing, vol. II (Software), pages II–1 – II–4, 1991.

[9] J. Dongarra, A. Geist, R. Manchek, and V. Sunderam. Integrated PVM framework supports heterogeneous network computing. Computers in Physics, 7(2):166–75, April 1993.

[10] J. J. Dongarra, R. Hempel, A. J. G. Hey, and D. W. Walker. A proposal for a userlevel, message passing interface in a distributed memory environment. Technical Report TM-12231, Oak Ridge National Laboratory, February 1993.

[11] Edinburgh Parallel Computing Centre, University of Edinburgh. CHIMP Concepts,

June 1991.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

[12] Edinburgh Parallel Computing Centre, University of Edinburgh. CHIMP Version 1.0 Interface, May 1992.

[13] D. Feitelson. Communicators: Object-based multiparty interactions for parallel programming. Technical Report 91-12, Dept. Computer Science, The Hebrew University of Jerusalem, November 1991.

[14] G. A. Geist, M. T. Heath, B. W. Peyton, and P. H. Worley. A user’s guide to PICL: a portable instrumented communication library. Technical Report TM-11616, Oak Ridge National Laboratory, October 1990.

[15] nCUBE Corporation. nCUBE 2 Programmers Guide, r2.0, December 1990.

[16] H. Mühlenbein O. Krämer. Mapping strategies in message–based multiprocessor systems. Parallel Computing, 9:213–225, 1989.

[17] Parasoft Corporation. Express Version 1.0: A Communication Environment for Parallel Computers, 1988.

[18] Paul Pierce. The NX/2 operating system. In Proceedings of the Third Conference on Hypercube Concurrent Computers and Applications, pages 384–390. ACM Press, 1988.

[19] A. Skjellum and A. Leung. Zipcode: a portable multicomputer communication library atop the reactive kernel. In D. W. Walker and Q. F. Stout, editors, Proceedings of the Fifth Distributed Memory Concurrent Computing Conference, pages 767–776. IEEE Press, 1990.

[20] A. Skjellum, S. Smith, C. Still, A. Leung, and M. Morari. The Zipcode message passing system. Technical report, Lawrence Livermore National Laboratory, September 1992.

[21] D. Walker. Standards for message passing in a distributed memory environment. Technical Report TM-12147, Oak Ridge National Laboratory, August 1992.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Annex A

Language Binding

A.1 Introduction

In this section we summarize the specific bindings for both Fortran and C. We present first the C bindings, then the Fortran bindings. Listings are alphabetical within chapter.

A.2 Defined Constants for C and Fortran

These are required defined constants, to be defined in the files mpi.h (for C) and mpif.h (for Fortran).

/* return codes */
MPI_SUCCESS
/* assorted constants (both C and Fortran) */
MPI_BOTTOM
MPI_PROCNULL
MPI_ANY_SOURCE
MPI_ANY_TAG
MPI_UNDEFINED
MPI_UNDEFINED_RANK
MPI_UB
MPI_LB
/* constant array sizes (Fortran) */
MPI_STATUS_SIZE
/* attributes (C and Fortran) */
MPI_UNDEFINED_ATTR
MPI_FOUND
MPI_NOT_FOUND
MPI_ATTR_COPY_FREELY
MPI_ATTR_COPY_NEVER
MPI_ATTRCOPY_CALLBACK

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

/* Error-handling specifiers (C and Fortran) */
MPI_ERRORSFATAL
MPI_ERRORSRETURN
MPI_ERRORSCALLUSERROUTINE
/* Pre-defined error handlers (C and Fortran) */
MPI_NOHANDLER
MPI_DEFAULTHANDLER
/* Maximum sizes for strings */
MPI_MAX_INTERCOMM_NAME_SIZE
/* elementary datatypes (C) */
MPI_CHAR
MPI_SHORT
MPI_INT
MPI_LONG
MPI_FLOAT
MPI_DOUBLE
MPI_BYTE
MPI_UNSIGNED_CHAR
MPI_UNSIGNED_SHORT
MPI_UNSIGNED_LONG
MPI_UNSIGNED_INT
/* elementary datatypes (Fortran) */
MPI_INTEGER
MPI_REAL
MPI_DOUBLE_PRECISION
MPI_COMPLEX
MPI_DOUBLE_COMPLEX
MPI_LOGICAL
MPI_CHARACTER
MPI_BYTE
/* reserved communicators (C and Fortran) */
MPI_COMM_WORLD
/* collective operations (C and Fortran) */
MPI_MAX
MPI_MIN
MPI_SUM
MPI_PROD
MPI_AND
MPI_OR
MPI_XOR
MPI_MAXLOC
MPI_MINLOC

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

/* topologies (C and Fortran) */
MPI_REORDER
MPI_NOREORDER
MPI_GRAPHTOP
MPI_CARTTOP
MPI_NOTOP
MPI_PER
MPI_NOPER

The following are defined C type definitions, also included in the file mpi.h.

/* opaque types (C) */
MPI_Persistence
MPI_Mode
MPI_Op
MPI_Comm_name
MPI_Aint
MPI_Errmode
/* assorted opaque structures (C) */
MPI_Status
MPI_Group
MPI_Communicator
MPI_Datatype
MPI_Comm_request
MPI_Attribute
MPI_Intercomm_request

A.3 C bindings for Point-to-Point Communication

These are presented here in the order of their appearance in the chapter.

int MPI Send(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
int MPI Recv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Status *status)
int MPI Get source(MPI Status status, int *source)
int MPI Get tag(MPI Status status, int *tag)
int MPI Get count(MPI Status status, MPI Datatype datatype, int *count)
int MPI Rsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
int MPI Ssend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

int MPI Isend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Irsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Issend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Irecv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Wait(MPI Comm request *request, MPI Status *status)
int MPI Test(MPI Comm request *request, int flag, MPI Status *status)
int MPI Waitany(int count, MPI Comm request *array of requests, int *index,
             MPI Status *status)
int MPI Testany(int count, MPI Comm request *array of requests, int *index,
             int *flag, MPI Status *status)
int MPI Waitall(int count, MPI Comm request *array of requests,
             MPI Status *array of statuses)
int MPI Testall(int count, MPI Comm request *array of requests, int *flag,
             MPI Status *array of statuses)
int MPI Iprobe(int source, int tag, MPI Comm comm, int *flag,
             MPI Status *status)
int MPI Probe(int source, int tag, MPI Comm comm, MPI Status *status)
int MPI Cancel(MPI Comm request request)
int MPI Test cancelled(MPI Status status, int *flag)
int MPI Create send(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Create rsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Create ssend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Create recv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Comm request request)
int MPI Start(MPI Comm request request)
int MPI Startall(int count, MPI Comm request *array of requests)
int MPI Request free(MPI Comm request request)
int MPI Sendrecv(void *sendbuf, int sendcount, MPI Datatype sendtype,
             int dest, int sendtag, void *recvbuf, int recvcount,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

             MPI Datatype recvtype, int source, MPI Datatype recvtag,
             MPI Comm comm, MPI Status *status)
int MPI Sendrecv replace(void* buf, int count, MPI Datatype datatype,
             int dest, int sendtag, int source, int recvtag, MPI Comm comm,
             MPI Status *status)
int MPI Type contiguous(int count, MPI Datatype oldtype,
             MPI Datatype *newtype)
int MPI Type vector(int count, int blocklength, int stride,
             MPI Datatype oldtype, MPI Datatype *newtype)
int MPI Type hvector(int count, int blocklength, MPI Aint stride,
             MPI Datatype oldtype, MPI Datatype *newtype)
int MPI Type indexed(int count, int *array of blocklengths,
             int *array of displacements, MPI Datatype oldtype,
             MPI Datatype *newtype)
int MPI Type hindexed(int count, int *array of blocklengths,
             MPI Aint *array of displacements, MPI Datatype oldtype,
             MPI Datatype *newtype)
int MPI Type struct(int count, int *array of blocklengths,
             MPI Aint *array of displacements, MPI Datatype *array of types,
             MPI Datatype *newtype)
int MPI Address(void* location, MPI Aint address)
int MPI Type extent(MPI Datatype datatype, MPI Aint *extent)
int MPI Type size(MPI Datatype datatype, MPI Aint *size)
int MPI Type count(MPI Datatype datatype, int *count)
int MPI Type commit(MPI Datatype *datatype)
int MPI Type free(MPI Datatype *datatype)
int MPI Get elements(MPI Status status, MPI Datatype datatype, int *count)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

A.4 C Bindings for Collective Communication

int MPI Barrier(MPI Comm comm )
int  MPI Bcast(void* buf, int count, MPI Datatype datatype, int root,
MPI Comm comm )
int MPI Gather(void* sendbuf, int sendcount, MPI Datatype sendtype,
void* recvbuf, int recvcount, MPI Datatype recvtype, int root,
MPI Comm comm)
int MPI Gatherv(void* sendbuf, int sendcount, MPI Datatype sendtype,
void* recvbuf, int *recvcounts, int *displs,
MPI Datatype recvtype, int root, MPI Comm comm)
int MPI Scatter(void* sendbuf, int sendcount, MPI Datatype sendtype,
void* recvbuf, int recvcount, MPI Datatype recvtype, int root,
MPI Comm comm)
int MPI Scatterv(void* sendbuf, int *sendcounts, int *displs,
MPI Datatype sendtype, void* recvbuf, int recvcount,
MPI Datatype recvtype, int root, MPI Comm comm)
int MPI Allgather(void* sendbuf, int sendcount, MPI Datatype sendtype,
void* recvbuf, int recvcount, MPI Datatype recvtype,
MPI Comm comm)
int MPI Allgatherv(void* sendbuf, int sendcount, MPI Datatype sendtype,
void* recvbuf, int *recvcounts, int *displs,
MPI Datatype recvtype, MPI Comm comm)
int MPI Alltoall(void* sendbuf, int sendcount, MPI Datatype sendtype,
void* recvbuf, int recvcount, MPI Datatype recvtype,
MPI Comm comm)
int MPI Alltoallv(void* sendbuf, int *sendcounts, int *sdispls,
MPI Datatype sendtype, void* recvbuf, int *recvcounts,
int *rdispls, MPI Datatype recvtype, MPI Comm comm)
int MPI Reduce(void* sendbuf, void* recvbuf, int count,
MPI Datatype datatype, MPI Op op, int root, MPI Comm comm)
int MPI User reduce(void* sendbuf, void* recvbuf, int count,
MPI Datatype datatype, void function, int root, MPI Comm comm)
int MPI User reducea(void* sendbuf, void* recvbuf, int count,
MPI Datatype datatype, void function, int root, MPI Comm comm)
int MPI Allreduce(void* sendbuf, void* recvbuf, int count,
MPI Datatype datatype, MPI Op op, MPI Comm comm)
int MPI User allreduce(void* sendbuf, void* recvbuf, int count,
MPI Datatype datatype, void function, MPI Comm comm)
int MPI User allreducea(void* sendbuf, void* recvbuf, int count,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

             MPI Datatype datatype, void function, MPI Comm comm)
int MPI Reduce scatter(void* sendbuf, void* recvbuf, int *recvcounts,
             MPI Datatype datatype, MPI Op op, MPI Comm comm)
int MPI Scan(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, MPI Op op, MPI Comm comm )
int MPI User scan(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, void function, MPI Comm comm)
int MPI User scana(void* sendbuf, void* recvbuf, int count,
             MPI Datatype datatype, void function, MPI Comm comm)

A.5 C Bindings for Groups, Contexts, and Communicators

int MPI Group size(MPI Group group, int *size)
int MPI Group rank(MPI Group group, int *rank)
int MPI Group translate ranks (MPI Group group1, int n, int *ranks1,
             MPI Group group2, int *ranks2)
int MPI Group union(MPI Group group1, MPI Group group2, MPI Group *newgroup)
int MPI Group intersection(MPI Group group1, MPI Group group2,
             MPI Group *newgroup)
int MPI Group difference(MPI Group group1, MPI Group group2,
             MPI Group *newgroup)
int MPI Group incl(MPI Group group, int n, int *ranks, MPI Group *newgroup)
int MPI Group excl(MPI Group group, int n, int *ranks, MPI Group *newgroup)
int MPI Group range incl(MPI Group group, int n, int **ranges,
             MPI Group *newgroup)
int MPI Group range excl(MPI Group group, int n, int **ranges,
             MPI Group *newgroup)
int MPI Group free(MPI Group *group)
int MPI Comm size(MPI Comm comm, int *size)
int MPI Comm rank(MPI Comm comm, int *rank)
int MPI Comm group(MPI Comm comm, MPI Group *group)
int MPI Comm dup(MPI Comm comm, MPI Comm *newcomm)
int MPI Comm make(MPI Comm comm, MPI Group group, MPI Comm *newcomm)
int MPI Comm split(MPI Comm comm, int color, int key, MPI Comm *newcomm)
int MPI Comm free(MPI Comm *comm)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

int MPI Comm test inter(MPI Comm comm, int *flag)
int MPI Intercomm start(MPI Comm local comm, int local leader,
             MPI Comm peer comm, int remote leader, int tag,
             MPI Intercomm request *inter request)
int MPI Intercomm finish(MPI Intercomm request *inter request,
             MPI Comm *newcomm)
int MPI Intercomm merge(MPI Comm intercomm, int high,
             MPI Comm *newintracomm)
int MPI Intercomm name start(MPI Comm comm, char *name,
             MPI Intercomm request *inter request)
int MPI Attr create key(int copy fn(), int delete fn(), int keyval,
             void* extra state)
int MPI Attr free key(int keyval)
int MPI Put attr(MPI Comm comm, int keyval, void* attribute val)
int MPI Get attr(MPI Comm comm, int keyval, void* attribute val, int *found)
int MPI Delete Attr(MPI Comm comm, int keyval)

A.6 C Bindings for Process Topologies

int MPI Make cart(MPI comm comm old, int ndims, int *dims, int *periods,
             int reorder, MPI comm *comm cart)
int MPI Make dims(int nnodes, int ndims, int *dims)
int MPI Make graph(MPI comm comm old, int nnodes, int *index, int *edges,
             int reorder, MPI comm *comm graph)
int MPI Topo status(MPI comm comm, int *status)
int MPI Get graphdims(MPI comm comm, int *nnodes, int *nedges)
int MPI Get graph(MPI comm comm, int maxindex, int maxedges, int *index,
             int *edges)
int MPI Get cartdim(MPI comm comm, int *ndims)
int MPI Get cart(MPI comm comm, int maxdims, int *dims, int *periods,
             int *coords)
int MPI Cart rank(MPI comm comm, int *coords, int *rank)
int MPI Cart coords(MPI comm comm, int rank, int maxdims, int *coords)
int MPI Graph neighbors count(MPI comm comm, int rank, int *nneighbors)
int MPI Graph neighbors(MPI comm comm, int rank, int *maxneighbors,
             int *neighbors)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

int MPI Cart shift(MPI comm comm, int direction, int disp, int *rank source,
             int *rank dest)
int MPI Cart sub(MPI comm comm, int *remain dims, MPI comm *comm new)
int MPI Map cart(MPI comm comm old, int ndims, int *dims, int *period,
             int *newrank)
int MPI Map graph(MPI comm comm old, int nnodes, int *index, int *edges,
             int *newrank)

A.7 C bindings for Environmental Inquiry

int MPI Get valid tag range(int *low, int *high)
int MPI Get processor name(int len, char *name)
int MPI Get host(MPI Comm comm, int *rank)
int MPI Get buffer params(int *numsend, int *sizesend, int *numrecv,
             int *sizerecv, int *totalnumsend, int *totalsizesend,
             int *totalnumrecv, int *totalsizerecv)
int MPI Suggest buffer params(int numsend, int sizesend, int numrecv,
             int sizerecv, int totalnumsend, int totalsizesen,
             int totalnumrecv, int totalsizerecv, int *flag)
int MPI User specifies buffer(MPI Comm comm, void* buffer, int size)
int MPI Get errormode(MPI Comm comm, MPI Errmode *mode)
int MPI Set errormode(MPI Comm comm, MPI Errmode mode)
int MPI Set errhandler(MPI Comm comm, void handler)
int MPI Error string(int errorcode, int len, char *string)
int MPI Init(int *argc, char **argv)
int MPI Finalize()
int MPI Initialized(int *flag)
int MPI Abort(MPI Comm comm, int errorcode)

A.8 Fortran Bindings for Point-to-Point Communication

MPI SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR
MPI RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)
    <type> BUF(*)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE),
IERROR
MPI GET SOURCE(STATUS, SOURCE, IERROR)
INTEGER STATUS(MPI STATUS SIZE), SOURCE, IERROR
MPI GET TAG(STATUS, TAG, IERROR)
INTEGER STATUS(MPI STATUS SIZE), TAG, IERROR
MPI GET COUNT(STATUS, DATATYPE, COUNT, IERROR)
INTEGER STATUS(MPI STATUS SIZE), DATATYPE, COUNT, IERROR
MPI RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR
MPI SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR
MPI ISEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR
MPI IRSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR
MPI ISSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR
MPI IRECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR
MPI WAIT(REQUEST, STATUS, IERROR)
INTEGER REQUEST, STATUS(MPI STATUS SIZE), IERROR
MPI TEST(REQUEST, FLAG, STATUS, IERROR)
LOGICAL FLAG
INTEGER REQUEST, STATUS(MPI STATUS SIZE), IERROR
MPI WAITANY(COUNT, ARRAY OF REQUESTS, INDEX, STATUS, IERROR)
INTEGER COUNT, ARRAY OF REQUESTS(*), INDEX, STATUS(MPI STATUS SIZE),
IERROR
MPI TESTANY(COUNT, ARRAY OF REQUESTS, INDEX, FLAG, STATUS, IERROR)
LOGICAL FLAG
INTEGER COUNT, ARRAY OF REQUESTS(*), INDEX, STATUS(MPI STATUS SIZE),
IERROR
MPI WAITALL(COUNT, ARRAY OF REQUESTS, ARRAY OF STATUSES, IERROR)
INTEGER COUNT, ARRAY OF REQUESTS(*),

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR
MPI TESTALL(COUNT, ARRAY OF REQUESTS, FLAG, ARRAY OF STATUSES, IERROR)
LOGICAL FLAG
INTEGER COUNT, ARRAY OF REQUESTS(*),
ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR
MPI IPROBE(SOURCE, TAG, COMM, FLAG, STATUS, IERROR)
LOGICAL FLAG
INTEGER SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE), IERROR
MPI PROBE(SOURCE, TAG, COMM, STATUS, IERROR)
INTEGER SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE), IERROR
MPI CANCEL(REQUEST, IERROR)
INTEGER REQUEST, IERROR
MPI TEST CANCELLED(STATUS, FLAG, IERROR)
LOGICAL FLAG
INTEGER STATUS(MPI STATUS SIZE), IERROR
MPI CREATE SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER REQUEST, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR
MPI CREATE RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR
MPI CREATE SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR
MPI CREATE RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR
MPI START(REQUEST, IERROR)
INTEGER REQUEST, IERROR
MPI STARTALL(COUNT, ARRAY OF REQUESTS, IERROR)
INTEGER COUNT, ARRAY OF REQUESTS(*), IERROR
MPI REQUEST FREE(REQUEST, IERROR)
INTEGER REQUEST, IERROR
MPI SENDRECV(SENDBUF, SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVBUF,
         RECVCOUNT, RECVTYPE, SOURCE, RECVTAG, COMM, STATUS, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVCOUNT, RECVTYPE,
SOURCE, RECVTAG, COMM, STATUS(MPI STATUS SIZE), IERROR
MPI SENDRECV REPLACE(BUF, COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG,
         COMM, STATUS, IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG, COMM,
STATUS(MPI STATUS SIZE), IERROR
MPI TYPE CONTIGUOUS(COUNT, OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, OLDTYPE, NEWTYPE, IERROR
MPI TYPE VECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR
MPI TYPE HVECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR
MPI TYPE INDEXED(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
         OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
OLDTYPE, NEWTYPE, IERROR
MPI TYPE HINDEXED(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
         OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
OLDTYPE, NEWTYPE, IERROR
MPI TYPE STRUCT(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
         ARRAY OF TYPES, NEWTYPE, IERROR)
INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
ARRAY OF TYPES(*), NEWTYPE, IERROR
MPI ADDRESS(LOCATION, ADDRESS, IERROR)
<type> LOCATION(*)
INTEGER ADDRESS, IERROR
MPI TYPE EXTENT(DATATYPE, EXTENT, IERROR)
INTEGER DATATYPE, EXTENT, IERROR
MPI TYPE SIZE(DATATYPE, SIZE, IERROR)
INTEGER DATATYPE, SIZE, IERROR
MPI TYPE COUNT(DATATYPE, COUNT, IERROR)
INTEGER DATATYPE, COUNT, IERROR
MPI TYPE COMMIT(DATATYPE, IERROR)
INTEGER DATATYPE, IERROR
MPI TYPE FREE(DATATYPE, IERROR)
INTEGER DATATYPE, IERROR
MPI GET ELEMENTS(STATUS, DATATYPE, COUNT, IERROR)
INTEGER STATUS(MPI STATUS SIZE), DATATYPE, COUNT, IERROR

A.9 Fortran Bindings for Collective Communication

MPI BARRIER(COMM, IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

INTEGER COMM, IERROR
MPI BCAST(BUFFER, COUNT, DATATYPE, ROOT, COMM, IERROR)
<type> BUFFER(*)
INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR
MPI GATHER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
         ROOT, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, ROOT, COMM, IERROR
MPI GATHERV(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNTS, DISPLS,
         RECVTYPE, ROOT, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, RECVCOUNTS(*), DISPLS(*), RECVTYPE, ROOT,
COMM, IERROR
MPI SCATTER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
         ROOT, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, ROOT, COMM, IERROR
MPI SCATTERV(SENDBUF, SENDCOUNTS, DISPLS, SENDTYPE, RECVBUF, RECVCOUNT,
         RECVTYPE, ROOT, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNTS(*), DISPLS(*), SENDTYPE, RECVCOUNT, RECVTYPE, ROOT,
COMM, IERROR
MPI ALLGATHER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
         COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, COMM, IERROR
MPI ALLGATHERV(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNTS, DISPLS,
         RECVTYPE, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, RECVCOUNTS(*), DISPLS(*), RECVTYPE, COMM,
IERROR
MPI ALLTOALL(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,
         COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE, RECVCOUNT, RECVTYPE, COMM, IERROR
MPI ALLTOALLV(SENDBUF, SENDCOUNTS, SDISPLS, SENDTYPE, RECVBUF, RECVCOUNTS,
         RDISPLS, RECVTYPE, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNTS(*), SDISPLS(*), SENDTYPE, RECVCOUNTS(*), RDISPLS(*),
RECVTYPE, COMM, IERROR
MPI REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

INTEGER COUNT, DATATYPE, OP, ROOT, COMM, IERROR
MPI USER REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, ROOT, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR
MPI USER REDUCEA(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, ROOT, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR
MPI ALLREDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER COUNT, DATATYPE, OP, COMM, IERROR
MPI USER ALLREDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, COMM, IERROR
MPI USER ALLREDUCEA(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, COMM, IERROR
MPI REDUCE SCATTER(SENDBUF, RECVBUF, RECVCOUNTS, DATATYPE, OP, COMM,
         IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER RECVCOUNTS, DATATYPE, OP, COMM, IERROR
MPI SCAN(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
INTEGER COUNT, DATATYPE, OP, COMM, IERROR
MPI USER SCAN(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, COMM, IERROR
MPI USER SCANA(SENDBUF, RECVBUF, COUNT, DATATYPE, FUNCTION, COMM, IERROR)
<type> SENDBUF(*), RECVBUF(*)
EXTERNAL FUNCTION
INTEGER COUNT, DATATYPE, COMM, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

A.10. FORTRAN BINDINGS FOR GROUPS, CONTEXTS, AND COMMUNICATORS189

A.10 Fortran Bindings for Groups, Contexts, and Communicators

MPI GROUP SIZE(GROUP, SIZE, IERROR)
    INTEGER GROUP, SIZE, IERROR
MPI GROUP RANK(GROUP, RANK, IERROR)
    INTEGER GROUP, RANK, IERROR
MPI GROUP TRANSLATE RANKS(GROUP1, N, RANKS1, GROUP2, RANKS2, IERROR)
    INTEGER GROUP1, N, RANKS1(*), GROUP2, RANKS2(*), IERROR
MPI GROUP UNION(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR
MPI GROUP INTERSECTION(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR
MPI GROUP DIFFERENCE(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR
MPI GROUP INCL(GROUP, N, RANKS, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR
MPI GROUP EXCL(GROUP, N, RANKS, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR
MPI GROUP RANGE INCL(GROUP, N, RANGES, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANGES(*,*), NEWGROUP, IERROR
MPI GROUP RANGE EXCL(GROUP, N, RANGES, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANGES(*,*), NEWGROUP, IERROR
MPI GROUP FREE(GROUP, IERROR)
    INTEGER GROUP, IERROR
MPI COMM SIZE(COMM, SIZE, IERROR)
    INTEGER COMM, SIZE, IERROR
MPI COMM RANK(COMM, RANK, IERROR)
    INTEGER COMM, RANK, IERROR
MPI COMM GROUP(COMM, GROUP, IERROR)
    INTEGER COMM, GROUP, IERROR
MPI COMM DUP(COMM, NEWCOMM, IERROR)
    INTEGER COMM, NEWCOMM, IERROR
MPI COMM MAKE(COMM, GROUP, NEWCOMM, IERROR)
    INTEGER COMM, GROUP, NEWCOMM, IERROR
MPI COMM SPLIT(COMM, COLOR, KEY, NEWCOMM, IERROR)
    INTEGER COMM, COLOR, KEY, NEWCOMM, IERROR
MPI COMM FREE(COMM, IERROR)
    INTEGER COMM, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI COMM TEST INTER(COMM, FLAG, IERROR)
INTEGER COMM, IERROR
LOGICAL FLAG
MPI INTERCOMM START(LOCAL COMM, LOCAL LEADER, PEER COMM, REMOTE LEADER, TAG,
         INTER REQUEST, IERROR)
INTEGER LOCAL COMM, LOCAL LEADER, PEER COMM, REMOTE LEADER, TAG,
INTER REQUEST, IERROR
MPI INTERCOMM FINISH(INTER REQUEST, IERROR)
INTEGER INTER REQUEST, NEWCOMM, IERROR
MPI INTERCOMM MERGE(INTERCOMM, HIGH, NEWINTERCOMM, IERROR)
INTEGER INTERCOMM, NEWINTRACOMM, IERROR
LOGICAL HIGH
MPI INTERCOMM NAME START(COMM, NAME, NEW COMM, INTER REQUEST, IERROR)
CHARACTER *(*) NAME
INTEGER COMM, NEW COMM, INTER REQUEST, IERROR
MPI ATTR CREATE KEY(COPY FN, DELETE FN, KEYVAL, EXTRA STATE, IERROR)
EXTERNAL COPY FN, DELETE FN
INTEGER KEYVAL, EXTRA STATE, IERROR
MPI ATTR FREE KEY(KEYVAL, IERROR)
INTEGER KEYVAL, IERROR
MPI PUT ATTR(COMM, KEYVAL, ATTRIBUTE VAL, IERROR)
INTEGER COMM, KEYVAL, ATTRIBUTE VAL, IERROR
MPI GET ATTR(COMM, KEYVAL, ATTRIBUTE VAL, FOUND, IERROR)
INTEGER COMM, KEYVAL, ATTRIBUTE VAL, IERROR
LOGICAL FOUND
MPI DELETE ATTR(COMM, KEYVAL, IERROR)
INTEGER COMM, KEYVAL, IERROR

A.11 Fortran Bindings for Process Topologies

MPI MAKE CART(COMM OLD, NDIMS, DIMS, PERIODS, REORDER, COMM CART, IERROR)
INTEGER COMM OLD, NDIMS, DIMS(*), PERIODS(*), REORDER, COMM CART,
IERROR
MPI MAKE DIMS(NNODES, NDIMS, DIMS, IERROR)
INTEGER NNODES, NDIMS, DIMS(*), IERROR
MPI MAKE GRAPH(COMM OLD, NNODES, INDEX, EDGES, REORDER, COMM GRAPH, IERROR)
INTEGER COMM OLD, NNODES, INDEX(*), EDGES(*), REORDER, COMM GRAPH,
IERROR
MPI TOPO STATUS(COMM, STATUS, IERROR)
INTEGER COMM, STATUS, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI GET GRAPHDIMS(COMM, NNODES, NEDGES, IERROR)
INTEGER COMM, NNODES, NEDGES, IERROR
MPI GET GRAPH(COMM, MAXINDEX, MAXEDGES, INDEX, EDGES, IERROR)
INTEGER COMM, MAXINDEX, MAXEDGES, INDEX(*), EDGES(*), IERROR
MPI GET CARTDIM(COMM, NDIMS, IERROR)
INTEGER COMM, NDIMS, IERROR
MPI GET CART(COMM, MAXDIMS, DIMS, PERIODS, COORDS, IERROR)
INTEGER COMM, MAXDIMS, DIMS(*), PERIODS(*), COORDS(*), IERROR
MPI CART RANK(COMM, COORDS, RANK, IERROR)
INTEGER COMM, COORDS(*), RANK, IERROR
MPI CART COORDS(COMM, RANK, MAXDIMS, COORDS, IERROR)
INTEGER COMM, RANK, MAXDIMS, COORDS(*), IERROR
MPI GRAPH NEIGHBORS COUNT(COMM, RANK, NNEIGHBORS, IERROR)
INTEGER COMM, RANK, NNEIGHBORS, IERROR
MPI GRAPH NEIGHBORS(COMM, RANK, MAXNEIGHBORS, NEIGHBORS, IERROR)
INTEGER COMM, RANK, MAXNEIGHBORS, NEIGHBORS(*), IERROR
MPI CART SHIFT(COMM, DIRECTION, DISP, RANK SOURCE, RANK DEST, IERROR)
INTEGER COMM, DIRECTION, DISP, RANK SOURCE, RANK DEST, IERROR
MPI CART SUB(COMM, REMAIN DIMS, COMM NEW, IERROR)
INTEGER COMM, REMAIN DIMS(*), COMM NEW, IERROR
MPI MAP CART(COMM OLD, NDIMS, DIMS, PERIOD, NEWRANK, IERROR)
INTEGER COMM OLD, NDIMS, DIMS(*), PERIOD(*), NEWRANK, IERROR
MPI MAP GRAPH(COMM OLD, NNODES, INDEX, EDGES, NEWRANK, IERROR)
INTEGER COMM OLD, NNODES, INDEX(*), EDGES(*), NEWRANK, IERROR

A.12 Fortran Bindings for Environmental Inquiry

MPI GET VALID TAG RANGE(LOW, HIGH, IERROR)
INTEGER LOW, HIGH, IERROR
MPI GET PROCESSOR NAME(NAME, IERROR)
CHARACTER*(*) NAME
INTEGER IERROR
MPI GET HOST(COMM, RANK, IERROR)
INTEGER COMM, RANK, IERROR
MPI GET BUFFER PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND,
         TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR)
INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND,
TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI SUGGEST BUFFER PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV,
         TOTALNUMSEND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, FLAG,
         IERROR)
INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEND,
TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR
LOGICAL FLAG
MPI USER SPECIFIES BUFFER(COMM, BUFFER, SIZE, IERROR)
<type> BUFFER(*)
INTEGER COMM, SIZE, IERROR
MPI GET ERRORMODE(COMM, MODE, IERROR)
INTEGER COMM, MODE, IERROR
MPI SET ERRORMODE(COMM, MODE, IERROR)
INTEGER COMM, MODE, IERROR
MPI SET ERRHANDLER(COMM, HANDLER, IERROR)
EXTERNAL HANDLER
INTEGER COMM, IERROR
MPI ERROR STRING(ERRORCODE, STRING, IERROR)
INTEGER ERRORCODE, IERROR
CHARACTER*(*) STRING
MPI INIT(IERROR)
INTEGER IERROR
MPI FINALIZE(IERROR)
INTEGER IERROR
MPI INITIALIZED(FLAG, IERROR)
LOGICAL FLAG
INTEGER IERROR
MPI ABORT(COMM, ERRORCODE, IERROR)
INTEGER COMM, ERRORCODE, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI Function Index

MPI ABORT, 161 MPI ADDRESS, 52 MPI ALLGATHER, 86 MPI ALLGATHERV, 87 MPI ALLREDUCE, 96 MPI ALLTOALL, 88 MPI ALLTOALLV, 89 MPI ATTR CREATE KEY, 121 MPI ATTR FREE KEY, 122

MPI BARRIER, 71 MPI BCAST, 73

MPI CANCEL, 37 MPI CART COORDS, 147 MPI CART RANK, 146 MPI CART SHIFT, 148 MPI CART SUB, 149 MPI COMM DUP, 113 MPI COMM FREE, 115 MPI COMM GROUP, 113 MPI COMM INIT, 67 MPI COMM MAKE, 114 MPI COMM RANK, 113 MPI COMM SIZE, 113 MPI COMM SPLIT, 114 MPI COMM TEST INTER, 117 MPI CREATE RECV, 40 MPI CREATE RSEND, 39 MPI CREATE SEND, 39 MPI CREATE SSEND, 40

MPI DELETE ATTR, 123

MPI ERROR STRING, 160

MPI FINALIZE, 161

MPI GATHER, 73 MPI GATHERV, 74 MPI GET ATTR, 122 MPI GET BUFFER PARAMS, 155 MPI GET CART, 146 MPI GET CARTDIM, 145 MPI GET COUNT, 19 MPI GET ELEMENTS, 57 MPI GET ERRORMODE, 159 MPI GET GRAPH, 145 MPI GET GRAPHDIMS, 145 MPI GET HOST, 154 MPI GET PROCESSOR NAME, 153 MPI GET SOURCE, 18 MPI GET TAG, 18 MPI GET VALID TAG RANGE, 153 MPI GRAPH NEIGHBORS, 147 MPI GRAPH NEIGHBORS COUNT, 147 MPI GROUP DIFFERENCE, 109 MPI GROUP EXCL, 110 MPI GROUP FREE, 112 MPI GROUP INCL, 110 MPI GROUP INTERSECTION, 109 MPI GROUP RANGE EXCL, 111 MPI GROUP RANGE INCL, 111 MPI GROUP RANK, 108 MPI GROUP SIZE, 107 MPI GROUP TRANSLATE RANKS, 108 MPI GROUP UNION, 108

MPI INIT, 160 MPI INITIALIZED, 161 MPI INTERCOMM FINISH, 119 MPI INTERCOMM MERGE, 119 MPI INTERCOMM NAME START, 120 MPI INTERCOMM START, 118 MPI IPROBE, 36 MPI IRECV, 29 MPI IRSEND, 28 MPI ISEND, 28 MPI ISSEND, 29

MPI MAKE CART, 141 MPI MAKE DIMS, 142 MPI MAKE GRAPH, 143 MPI MAP CART, 150 MPI MAP GRAPH, 151

MPI PROBE, 37 MPI PUT ATTR, 122

MPI RECV, 17 MPI REDUCE, 90 MPI REDUCE SCATTER, 97 MPI REQUEST FREE, 41 MPI RSEND, 25

MPI SCAN, 99 MPI SCATTER, 82 MPI SCATTERV, 83 MPI SEND, 15 MPI SENDRECV, 42 MPI SENDRECV REPLACE, 43 MPI SET ERRHANDLER, 159 MPI SET ERRORMODE, 159 MPI SSEND, 26 MPI START, 40 MPI STARTALL, 41 MPI SUGGEST BUFFER PARAMS, 156

MPI TEST, 30 MPI TEST CANCELLED, 38 MPI TESTALL, 34 MPI TESTANY, 33 MPI TOPO STATUS, 144 MPI TYPE COMMIT, 54 MPI TYPE CONTIGUOUS, 46 MPI TYPE COUNT, 53 MPI TYPE EXTENT, 52 MPI TYPE FREE, 54 MPI TYPE HINDEXED, 50 MPI TYPE HVECTOR, 48 MPI TYPE INDEXED, 49 MPI TYPE SIZE, 53 MPI TYPE STRUCT, 51 MPI TYPE VECTOR, 47

MPI USER ALLREDUCE, 96 MPI USER ALLREDUCEA, 97 MPI USER REDUCE, 93 MPI USER REDUCE SCATTER, 98 MPI USER REDUCE SCATTERA, 98 MPI USER REDUCEA, 95 MPI USER SCAN, 99 MPI USER SCANA, 100 MPI USER SPECIFIES BUFFER, 157

MPI WAIT, 30 MPI WAITALL, 34 MPI WAITANY, 33

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48