Distributed Arrays

Many parallel applications operate on arrays that are distributed across the parallel program's processes. In these applications, each process operates on a local component of a larger array. While each process has parts of the data, you sometimes need to integrate all this data into a common Variable Window to examine the data as it is being manipulated in all processes.

If the data is a standard C or Fortran array, you can use the Variable Window's View > Laminate command. However, if the data's type is more complicated, you will need to use the type mapping's distributed_callback property. This property lets you see the distributed array in its own global index space.

This section uses the mandel.c program that is listed in "Distributed Array Type Mapping". simple MPI program that distributes an array cyclically in one dimension over all processors. Each processor calculates one part of the Mandelbrot set. Of course, is not the best way to compute the Mandelbrot set, but the point here is not the Mandelbrot set, but the distributed array.

This program distributes an array of structures cyclically across all of the processes in the MPI job in the second dimension. On each processor, however, individual columns are held contiguously. The Mandlebrot program's local_column() and owner_of() procedures provide the transformation from a global array index to the local index and node. The following figure shows how TotalView displays the structure's array data on one node.

Standard Display of struct "a" as an Array

However, type transformation lets TotalView display the entire array in its global index space. The next figure shows is a Variable Window that shows this reassembled data.

Reassembled Display

Since TotalView knows that the array is distributed, you can even show individual node information. For example, the next figure slices out the first element of each column, and displays which node owns it.

A Slice of the Reassembled Array