Organizing Chaos

While it is possible to attack the kinds of programs that are running thousands of processes across hundreds of computers one-at-a-time, it is not very practical. What TotalView does is structure these processes for you and then let you reorganize this information into "groups". Here are quick definitions of the four kinds of groups:

• Control Group: All the processes created by a program running on all processors. That is, the control group includes all processes that are running on multiple processors. If your program uses processes that it did not create, these other processes are in another control group.
• Share Group: All the processes within a control group that share the same code. In most cases, your program will have more than one share group. Share groups, like control groups, can have processes that execute on more than one processor.
• Workers Group: All the worker threads within a control group. These threads can be drawn from more than one share group.
• Lockstep Group: All threads that are at the same PC (program counter). This group is a subset of a workers group. Because all threads execute asynchronously, a lockstep group only exists for stopped threads. All threads in the lockstep group are also in a workers group.

In the list, the first two groups contain processes and the last two groups contain threads. And, notice that "same code" means that the processes have the same executable file name.

TotalView's commands let you manipulate processes and threads individually and by groups. In addition, you can create your own groups and manipulate a group's contents (to some extent).

Note:   Not all operating systems let you individually run a thread.

The following figure shows a processor running five processes (ignoring daemons and other programs not related to your program) and the threads within them. The figure indicates a control group and two share groups.

Five Processors and Processor Groups

The elements in this figure are as follows:

CPU

Everything represented by this drawing exists within one processor.

Processes

Processes being executed by the CPU.

Control Group

The five processes make up the control group. This diagram does not indicate which process is the main procedure.

Share Groups

The control group has two share groups. The three processes in the first share group have the same executable. The two processes in the second share group share a second executable.

The next figure looks at how the threads in this drawing are organized. As you can see, this figure adds the workers group and two lockstep groups.

Five Processors and Processor Groups

Note:   The control group is not shown as it encompasses everything in this figure.

Here is a description of the added elements in this figure:

Workers Group

All non-manager threads within the control group make up the workers group. Notice that this group includes service threads.

Lockstep Group

Each share group has its own lockstep groups. Graphically, two lockstep groups are indicated, one in each share group.

If other threads are stopped, this picture indicates that they are not participating in either of these two lockstep groups. Recall that a stopped thread is always in a lockstep group. (It's OK if a lockstep group has only one member.)

Service Threads

Each process has one service thread. A process can have any number of service threads. This figure, however, only shows one.

Manager Threads

The only threads that are not participating in the workers group are the ten manager threads.

The following figure extends the previous figure to show the same kinds of information executing on two processors.

Five Processes and Their Groups on Two Computers

This figure differs from the one it is based on in that it has ten processes executing within two processors rather than five processes within one processor. Although the number of processors has changed, the number of control and share groups is unchanged. This is not to say that the number of groups could not be different. It's just they are not in this example.