CSc-387 Parallel Processing 
	
CHAPTER 2 : MESSAGE-PASSING COMPUTING

----------------------------------------------------------------------
2.1. BASICS
----------------------------------------------------------------------

3 ways to design a Parallel Programming Language:
   1. design from scratch     (e.g. Occam for transputers)
   2. extend the syntax/reserved words of an existing language (e.g. iPSC-C)
   3. add parallel libraries to an existing seq. language  (e.g. MPI)


Process Creation:
------------------
 Static: all the processes are specified before execution and the
         system will execute a fixed number of processes      (MPI)
  
         Typically, MPMD:    Master + Slave processes
                    SPMD:   all processes are represented with one program

         Note that If processors are of different type, the source code 
         is compiled into executable code for each processor type

 Dynamic process creation:                   spawn(name_of_process)

                         (* T Figure 2.2 *)
         Processes can be created and their execution initiated during
         the execution of other processes. They can also be destroyed.
         Obviously, the code for each process must be written beforehand.

          MPMD: Slave processes are dynamically created by a Master process 
 
- While PVM has dynamic process creation capabilities, MPI does not have it.
          
----------------------------------------------------
2.1.3 Message-Passing Routines
----------------------------------------------------

   send();     recv();     
   --------------------
   synchronous: (guarantees the receipt of message before continues)

                           (* T Figure 2.4 *)

   blocking: similar to synchronous, but send() continues after the
             message is buffered in the system buffer, does not
             guarantee the receipt of the message by destination PE)  

                           (* T Figure 2.5 *)

   non-blocking: be careful while using this one. It is programmer's
                 responsibility to monitor the correct delivery of messages 

Message Selection: there may be multiple messages from the same PE 
------------------ in the recv buffer. Use "message tags" to ensure that the 
                   correct message is read. 
                   If "wild card" is used, any message could be received.


    Broadcast, Gather/Scatter, Reduce operations:
    ----------------------------------------------
MPI_Bcast - Broadcasts a message from the process with rank root
            to all other processes of the group. 

#include "mpi.h"
int MPI_Bcast ( buffer, count, datatype, root, comm ) 
void *buffer;  int count;  MPI_Datatype datatype;  int root;  MPI_Comm comm; 

                           (* T Figure 2.6 *)

MPI_Gather - Gathers together values from a group of processes 

int MPI_Gather (sendbuf, sendcnt, sendtype, recvbuf, recvcount, recvtype, root, comm )
void *sendbuf;  int sendcnt;  MPI_Datatype sendtype; void *recvbuf; 
int recvcount;  MPI_Datatype recvtype;  int root;  MPI_Comm comm; 


MPI_Scatter - Sends data from one task to all other tasks in a group 

int MPI_Scatter (sendbuf, sendcnt, sendtype, recvbuf, recvcount, recvtype, root, comm )

                           (* T Figure 2.7 *)
                           (* T Figure 2.8 *)

MPI_Reduce - Reduces values on all processes to a single value 

int MPI_Reduce ( sendbuf, recvbuf, count, datatype, op, root, comm ) 

void *sendbuf;  void *recvbuf;  int count;  MPI_Datatype datatype; 
MPI_Op op;  int root;  MPI_Comm comm; 

                           (* T Figure 2.9 *)