Message Passing Programming Based on MPI Derived Data Types
Bora AKAYDIN
15.06.2012
Outline
Derived Data Types
•
Derived Datatypes
•
Packing/Unpacking Datatypes
15.06.2012
Derived Datatypes
•
How to send only the red elements of V in a single communication?
V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9)
T(0) T(1) T(2)
MPI_Send
This method requires inefficient copy of noncontiguous data
One possibility, copy these elements to a temporary array before sending.
Derived Data Types
15.06.2012
Non-struct Derived Data Types
•
There are routines available in MPI library that are more suitable for an array or vector like data structures: • MPI_Type_contiguous • MPI_Type_vector • MPI_Type_indexed
•
Derived Data Types
All above functions work with a single data type!
15.06.2012
MPI_Type_contiguous (C)
•
Constructs a type consisting of the replication of a data type into continuous locations.
int MPI_Type_contiguous( int count, MPI_Datatype old_type, MPI_Datatype *newtype)
old type
x count (4)
= new type
Data constructor with MPI_Type_contiguous
Derived Data Types
15.06.2012
mpi_type_contiguous (Fortran)
•
Constructs a type consisting of the replication of a data type into continuous locations.
MPI_TYPE_CONTIGUOUS( count, MPI_Datatype old_type, MPI_Datatype newtype,ierr)
old type
x count (4)
= new type
Data constructor with mpi_type_contiguous
Derived Data Types
15.06.2012
MPI_Type_contiguous
•
In C, if we create a matrix with static memory allocation, we can say that data of the matrix will be in contiguous memory.
double A[3][3]; A A A (0,0) (0,1) (0,2) A A A (1,0) (1,1) (1,2) A A A (2,0) (2,1) (2,2)
Derived Data Types
in the memory (C) A A A A A A A A A (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2)
in the memory (fortran) A A A A A A A A A (0,0) (1,0)(2,0)(0,1) (1,1)(2,1) (0,2) (1,2) (2,2)
15.06.2012
MPI_Type_contiguous (C) double A[3][3]; A A A (0,0) (0,1) (0,2) A A A (1,0) (1,1) (1,2) A A A (2,0) (2,1) (2,2)
in the memory A A A A A A A A A (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2)
count =3 old_type =MPI_DOUBLE new_type =rowtype
MPI_Type_contiguous(int count, MPI_Datatype old_type, MPI_Datatype *newtype);
Derived Data Types
15.06.2012
mpi_type_contiguous (Fortran)
A A A (0,0) (1,0)(2,0) A A A (0,1) (1,1) (2,1) A A A (0,2) (1,2)(2,2)
in the memory (fortran) A A A A A A A A A (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2)
count =3 old_type =MPI_REAL new_type =columntype
call mpi_type_contiguous(count, MPI_Datatype old_type, MPI_Datatype newtype,ierr);
Derived Data Types
15.06.2012
Handling Non-Contiguous Data
•
How to send only the red elements of V, while avoiding copying non-contiguous data to a temporary array? V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9)
•
Derived Data Types
Define a new data type, in this case a vector with stride of two from original. vType
15.06.2012
MPI_Type_vector
•
old type
Similar to contiguous, but allows for regular gaps (stride) in the displacements. (MPI_INT)
new type blocklength=3 stride=5 count=2
Data constructor with MPI_Type_vector.
Derived Data Types
15.06.2012
MPI_Type_vector
•
In C, if we create a matrix with static memory allocation, we can say that data of the matrix will be in contiguous memory. • Suppose that, we want to send columns to the each task, instead of rows. double A[3][3]; A A A (0,0) (0,1) (0,2) A A A (1,0) (1,1) (1,2) A A A (2,0) (2,1) (2,2)
Derived Data Types
in the memory A A A A A A A A A (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2)
We can use, MPI_Type_vector to create vector (strided) data type.
15.06.2012
MPI_Type_vector double A[3][3]; A A A (0,0) (0,1) (0,2) A A A (1,0) (1,1) (1,2) A A A (2,0) (2,1) (2,2)
in the memory A A A A A A A A A (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2)
blocklength=1 stride=3 count=3
Derived Data Types
15.06.2012
MPI_Type_vector (C)
blocklength=1 stride=3 count=3
MPI_Type_vector(int count, int blocklength, int stride, MPI_Datatype old_type, MPI_Datatype *new_type);
Derived Data Types
15.06.2012
mpi_type_vector (Fortran)
blocklength=1 stride=3 count=3
mpi_type_vector(count, blocklength, stride, MPI_Datatype old_type, MPI_Datatype new_type,ierr);
Derived Data Types
15.06.2012
MPI_Type_vector (C) V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9)
•
Sending reds
vType
blocklength = 1 blocklength= ?1 stride= ?2 count= ?3
stride
=2
count
=3
old_type
= MPI_DOUBLE
new_type
= vtype
MPI_Type_vector( count, blocklength, stride, old_type, &vType); MPI_Send(&V[2], 1, vType, dest, tag, MPI_COMM_WORLD);
Derived Data Types
15.06.2012
mpi_type_vector (Fortran) V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9)
•
Sending reds
type
blocklength = 1 blocklength= ?1 stride= ?2 count= ?3
stride
=2
count
=3
old_type
= MPI_INTEGER
new_type
= type
call mpi_type_vector(count,blocklength,stride,old_type,type,ierr); call mpi_send(V(2), 1, type, dest, tag, MPI_COMM_WORLD,ierr);
Derived Data Types
15.06.2012
MPI_Type_indexed
•
Indexed constructor allows one to specify a noncontiguous data layout where displacements between successive blocks need not be equal.
double A[4][4];
Derived Data Types
15.06.2012
A A A A (0,0) (0,1) (0,2)(0,3) A A A A (1,0) (1,1) (1,2)(1,3) A A A A (2,0) (2,1) (2,2)(2,3) A A A A (3,0) (3,1) (3,2)(3,3)
MPI_Type_indexed
•
Derived Data Types
This allows: • Gathering of arbitrary entries from an array and sending them in one message, or • Receiving one message and scattering the received message entries into arbitrary locations in an array.
15.06.2012
MPI_Type_indexed (C)
int MPI_Type_indexed( int count, int blocklength[], int indices[], MPI_Datatype old_type, MPI_Datatype *newtype ) • • •
Derived Data Types
count : number of blocks blocklength : number of elements in each block indices : displacement for each block, measured as number of elements 15.06.2012
mpi_type_indexed (Fortran)
mpi_type_indexed( count, blocklength(), indices(), MPI_Datatype old_type, MPI_Datatype newtype,ierr ) • • •
Derived Data Types
count : number of blocks blocklength : number of elements in each block indices : displacement for each block, measured as number of elements 15.06.2012
MPI_Type_indexed
old type blen[0]= 2
blen[1]= 3
blen[2]= 1
new type indices[0]=0
indices[1]=3 count= 3
Derived Data Types
15.06.2012
indices[2]=8
MPI_Type_indexed
• •
Suppose that, we have a matrix A(4x4) We want to send upper triangular matrix
double A[4][4]; A A A A (0,0) (0,1) (0,2)(0,3) A A A A (1,0) (1,1) (1,2)(1,3) A A A A (2,0) (2,1) (2,2)(2,3) A A A A (3,0) (3,1) (3,2)(3,3)
old type
= MPI_DOUBLE
new type = upper count
= 4
blocklen[ ] = (4, 3, 2, 1) indices[ ] = (0, 5, 10, 15)
MPI_Type_indexed(count, blocklen, indices, MPI_DOUBLE, upper )
Derived Data Types
15.06.2012
MPI_Type_commit
• •
Commits new datatype to the system. Required for all user constructed (derived) data types.
int MPI_Type_commit( MPI_Datatype *datatype ) MPI_TYPE_COMMIT( MPI_Datatype datatype,ierr )
Derived Data Types
15.06.2012
MPI_Type_free
• •
Deallocates the specified data type object. Use of this routine is especially important to prevent memory exhaustion if many data type objects are created, as in a loop.
int MPI_Type_free( MPI_Datatype *datatype ) MPI_TYPE_FREE( MPI_Datatype datatype,ierr )
Derived Data Types
15.06.2012
Packing Different Data
Derived Data Types
•
Sometimes, users need to send non-contiguous data in a single package.
•
MPI allows them to • explicitly pack data into a contiguous buffer before sending it, and • unpack it from a contiguous buffer after receiving.
•
Several messages can be successively packed into one packing unit.
15.06.2012
MPI_Pack (C) data to be buffered int MPI_Pack ( void *packdata, number of input data items datatype of each input data item output buffer start size of buffer, in bytes current position in buffer, in bytes communicator for packed message
Derived Data Types
int count, MPI_Datatype datatype, void *buffer, int size, int *position, MPI_Comm comm )
15.06.2012
mpi_pack (Fortran) data to be buffered MPI_PACK ( number of input data items datatype of each input data item output buffer start size of buffer, in bytes current position in buffer, in bytes communicator for packed message
Derived Data Types
packdata, count, MPI_Datatype datatype, buffer, size, position, MPI_Comm comm,ierr )
15.06.2012
Packing Data
char: (1 Byte)
integer: (4 Byte)
char c[25]: T o d a y int date[3]:
18
i s 7
a 2007
w o n d e r f u l integer array with 3 elements
buffer
Derived Data Types
15.06.2012
d a y !
char array with 25 element
Packing Data
buffer T o d a y
i s
a
w o n d e r f u l
d a y !
18
7
• At the beginning position=0 MPI_Pack(c, 25, MPI_CHAR, buffer, 37, 37 &position, MPI_Comm comm ) • position value is updated by MPI_Pack as position=25 MPI_Pack(date, 3, MPI_INT, buffer, 37, &position, MPI_Comm comm ) • position value is updated by MPI_Pack as position=37
Derived Data Types
15.06.2012
2007
Packing Data
buffer T o d a y
i s
a
w o n d e r f u l
d a y !
18
7
2007
• At the beginning position=0 call MPI_PACK(c, 25, MPI_CHARACTER, buffer, 37, 37 &position, MPI_Comm comm,ierr ) • position value is updated by MPI_Pack as position=25 MPI_Pack(date, 3, MPI_INTEGER, buffer, 37, &position, MPI_Comm comm,ierr ) • position value is updated by MPI_Pack as position=37
Derived Data Types
15.06.2012
Sending Packed Data
Derived Data Types
•
MPI_Send function used to send packed data,
•
MPI_PACKED type is used as datatype.
•
Size of the data must be specified in Bytes.
•
Now, buffer size is 37 Bytes
15.06.2012
Sending Packed Data
char: (1 Byte)
buffer T o d a y
i s
a
integer: (4 Byte)
w o n d e r f u l
MPI_Send( buffer, position, MPI_PACKED, dest, tag, MPI_Comm comm);
Derived Data Types
15.06.2012
d a y !
18
7
2007
Sending Packed Data
char: (1 Byte)
buffer T o d a y
i s
a
integer: (4 Byte)
w o n d e r f u l
d a y !
call MPI_SEND( buffer, position, MPI_PACKED, dest, tag, MPI_Comm comm,ierr)
Derived Data Types
15.06.2012
18
7
2007
Receiving Packed Data
Derived Data Types
•
MPI_Recv function used to receive packed data,
•
MPI_PACKED type is used as datatype.
•
Size of the data must be specified in Bytes.
15.06.2012
Receiving Packed Data
MPI_Recv( Rbuffer, 37, MPI_PACKED, source, tag, comm, &status);
Rbuffer T o d a y
Derived Data Types
i s
a
w o n d e r f u l
15.06.2012
d a y !
18
7
2007
Receiving Packed Data
MPI_RECV( Rbuffer, 37, MPI_PACKED, source, tag, comm, status,ierr);
Rbuffer T o d a y
Derived Data Types
i s
a
w o n d e r f u l
15.06.2012
d a y !
18
7
2007
MPI_Unpack (C) input buffer start int MPI_Unpack(void *buffer, size of buffer, in bytes current position in buffer, in bytes output buffer start number of items to be unpacked datatype of each output data item communicator for packed message
Derived Data Types
int size, int *position, void *packdata, int count, MPI_Datatype datatype, MPI_Comm comm )
15.06.2012
mpi_unpack (Fortran) input buffer start int MPI_UNPACK( buffer, size of buffer, in bytes
size,
current position in buffer, in bytes
position,
output buffer start
packdata,
number of items to be unpacked datatype of each output data item communicator for packed message
Derived Data Types
count, MPI_Datatype datatype, MPI_Comm comm,ierr )
15.06.2012
MPI_Unpack
• •
• •
Derived Data Types
The output buffer can be any communication buffer allowed in MPI_Recv. The buffer is a contiguous storage area containing size bytes, starting at address buffer. The value of position is the first location in the buffer occupied by the packed message. position is incremented by the size of the packed message, so that the output value of position is the first location in the buffer after the locations occupied by the message that was unpacked.
15.06.2012
Unpacking Packed Data (C)
MPI_Unpack( buffer, 37, &position, First value position=0 Updated to position=25 &Rc, 25, MPI_CHAR, MPI_Comm comm);
char Rc[25]: T o d a y
Derived Data Types
i s
a
15.06.2012
w o n d e r f u l
d a y !
Unpacking Packed Data (Fortran)
MPI_UNPACK( buffer, 37, position, First value position=0 Updated to position=25 Rc, 25, MPI_CHARACTER, MPI_Comm comm,ierr);
char Rc[25]: T o d a y
Derived Data Types
i s
a
15.06.2012
w o n d e r f u l
d a y !
Unpacking Packed Data (C)
MPI_Unpack( buffer, 37, &position, First value position=25 Updated to position=37 &Rdate, 3, MPI_INT, MPI_Comm comm);
int Rdate[3]:
Derived Data Types
18
15.06.2012
7
2007
Unpacking Packed Data (Fortran)
MPI_UNPACK( buffer, 37, position, First value position=25 Updated to position=37 Rdate, 3, MPI_INTEGER, MPI_Comm comm,ierr);
int Rdate[3]:
Derived Data Types
18
15.06.2012
7
2007
Programming Activities
Writing parallel MPI codes using following routines – Contiguous – Indexed – Vector – Pack-unpack
Derived Data Types
15.06.2012