Contents TotalView Features TotalView Advantages TotalView Windows Understanding Multiprocess Programs Understanding Multithreaded Programs Controlling Processes and Threads Using Action Points Examining and Manipulating Data Distributed Debugging Visualization Context-Sensitive Help and Documentation TotalView Basics Compiling Programs Starting TotalView Initializing the Debugger Using the Mouse Buttons Using the Root Window The Process Window Diving into Objects Editing Text Searching for Text Searching for Functions and Variables Saving the Contents of Windows Exiting from TotalView Setting Up a Debugging Session Compiling Programs Starting TotalView Loading Executables More on Loading Remote Executables Reloading a Recompiled Executable Attaching to Processes Attaching Using the Unattached Page Attaching Using File > New Program Detaching from Processes Examining a Core File Processes and Thread State Attached Process States Unattached Process States Handling Signals Setting Search Paths Setting Command Arguments Setting Input and Output Files Setting Preferences Setting Preferences, Options, and X Resources Setting Environment Variables Monitoring TotalView Sessions Setting Up Remote Debugging Sessions Starting the TotalView Debugger Server Single Process Server Launch Options Bulk Launch Window Options Starting the Debugger Server Manually Single Process Server Launch Command Bulk Server Launch on an SGI MIPs Machine Bulk Server Launch on an IBM RS/6000 AIX Machine Disabling Autolaunch Changing the Remote Shell Command Changing the Arguments Autolaunch Sequence Debugging Over a Serial Line Start the TotalView Debugger Server Starting TotalView on a Serial Line New Program Window Setting Up Parallel Debugging Sessions Debugging MPI Applications Overview Debugging MPICH Applications Starting TotalView on an MPICH Job Attaching to an MPICH Job MPICH P4 procgroup Files Debugging Compaq MPI Applications Attaching to a Compaq MPI Job Debugging HP MPI Applications Attaching to an HP MPI Job Debugging IBM MPI (PE) Applications Preparing to Debug a PE Application Switch-Based Communication Remote Login Timeout Starting TotalView on a PE Job Setting Breakpoints Starting Parallel Tasks Attaching to a PE Job Attaching from a Node Running poe Attaching from a Node Not Running poe Debugging QSW RMS Applications Attaching to an RMS Job Debugging SGI MPI Applications Attaching to an SGI MPI Job Displaying the Message Queue Graph Displaying the Message Queue Message Queue Display Overview Message Operations MPI Process Diving MPI Buffer Diving Pending Receive Operations Unexpected Messages Pending Send Operations MPI Debugging Troubleshooting Debugging OpenMP Applications Debugging an OpenMP Program OpenMP Private and Shared Variables OpenMP THREADPRIVATE Common Blocks OpenMP Stack Parent Token Line Debugging PVM and DPVM Applications Supporting Multiple Sessions Setting Up ORNL PVM Debugging Starting an ORNL PVM Session Starting a DPVM Session Automatically Acquiring PVM/DPVM Processes Attaching to PVM/DPVM Tasks Reserved Message Tags Debugging Dynamic Libraries Cleanup of Processes Shared Memory Code Debugging Portland Group, Inc., HPF Applications Starting TotalView with HPF Dynamically Loaded Library Setting Up PGI HPF Compiler Defaults Setting Up MPICH Setting TotalView Defaults for HPF Compiling HPF for Debugging Starting HPF Programs PGI HPF smp and rpm Libraries Starting HPF Programs with MPICH Workstation Clusters Using MPICH IBM Parallel Environment Parallel Debugging Tips Attaching to Processes General Parallel Debugging Tips MPICH Debugging Tips IBM PE Debugging Tips Debugging Programs Displaying the Call Tree Finding the Source Code for Functions Resolving Ambiguous Names Finding the Source Code for Files Examining Source and Assembler Code Resetting the Current Stack Frame Editing Source Text Using the Toolbar to Select a Target Stopping Processes and Threads Updating Process Information Holding and Releasing Processes and Threads Examining Groups Displaying Groups Placing Processes into Groups Starting Processes and Threads Creating a Process Without Starting It Creating a Process by Single-Stepping Stepping Process-Width Stepping Group-Width Stepping Thread-Width Stepping Selecting Source Lines Using Single-Step Commands Stepping into Function Calls Stepping Over Function Calls Executing to a Selected Line Executing to the Completion of a Function Displaying Thread and Process Locations Continuing with a Specific Signal Setting the Program Counter Deleting Programs Restarting Programs Checkpointing Programs and Processes Interpreting Status and Control Registers Examining and Changing Data Displaying Variable Windows Displaying Local Variables and Registers Displaying a Global Variable Displaying All Global Variables Displaying Long Variable Names Displaying Areas of Memory Displaying Machine Instructions Closing Variable Windows Diving in Variable Windows Changing the Values of Variables Changing the Data Type of Variables How TotalView Displays C Data Types Pointers to Arrays Arrays Typedefs Structures Unions Built-In Types Character arrays (<string> Data Type) Areas of memory (<void> Data Type) Instructions (<code> Data Type) Type Casting Examples Displaying the argv Array Displaying Declared Arrays Displaying Allocated Arrays Working with Opaque Data Changing the Address of Variables Changing Types to Display Machine Instructions Displaying C++ Types Classes Changing Class Types in C++ Displaying Fortran Types Displaying Fortran Common Blocks Displaying Fortran Module Data Debugging Fortran 90 Modules Fortran 90 User-Defined Type Fortran 90 Deferred Shape Array Type Fortran 90 Pointer Type Displaying Fortran PARAMETERS Displaying Thread Objects Examining Arrays Examining and Analyzing Arrays Displaying Array Slices Slice Definitions Using Slices in the Variable Command Array Data Filtering Filtering by Comparison Filtering for IEEE Values Filtering by Range of Values Array Filter Expressions Filter Comparisons Filtering Array Data Sorting Array Data Array Statistics Displaying a Variable in All Processes or Threads Diving in a Laminated Pane Editing a Laminated Variable Visualizing Array Data Visualizing a Laminated Variable Window Setting Action Points Action Points Overview Setting Breakpoints and Barriers Setting Source-Level Breakpoints Selecting Ambiguous Source Lines Toggling Breakpoints at Locations Ambiguous Locations Displaying and Controlling Action Points Disabling Deleting Enabling Suppressing Setting Machine-Level Breakpoints Breakpoints for Multiple Processes Breakpoint When Using fork()/execve() Processes That Call fork() Processes That Call execve() Example: Multiprocess Breakpoint Barrier Breakpoints Barrier Breakpoint States Setting a Barrier Breakpoint Releasing Processes from Barrier Points Deleting a Barrier Point Changes When Setting and Clearing a Barrier Point Defining Evaluation Points Setting Evaluation Points Creating Conditional Breakpoint Examples Patching Programs Conditionally Patching Out Code Patching in a Function Call Correcting Code Interpreted vs. Compiled Expressions Interpreted Expressions Compiled Expressions Allocating Patch Space for Compiled Expressions Dynamic Patch Space Allocation Static Patch Space Allocation Using Watchpoints Architectures Creating Watchpoints Displaying Watchpoints Watching Memory Triggering Watchpoints Using Multiple Watchpoints Data Copies Conditional Watchpoints Saving Action Points to a File Evaluating Expressions Writing Code Fragments Intrinsic Variables Built-In Statements C Constructs Supported Data Types and Declarations Statements Fortran Constructs Supported Data Types and Declarations Statements Writing Assembler Code Visualizing Data How the Visualizer Works Configuring TotalView to Launch the Visualizer Data Types That TotalView Can Visualize Visualizing Data from the Variable Window Visualizing Data in Expressions Visualizer Animation Using the TotalView Visualizer Directory Window Data Windows Viewing of Data Graph Window Displaying Graphs Manipulating Graphs Surface Window Displaying Surface Data Manipulating Surface Data Launching the Visualizer from the Command Line Adapting a Third Party Visualizer Troubleshooting Overview The Problems X Resources TotalView X Resources Visualizer X Resources TotalView Command Syntax Syntax Options TotalView Debugger Server (tvdsvr) Command Syntax Replacement Characters A Compilers and Compiling with Debugging Symbols Compaq Tru64 UNIX HP-UX IBM AIX on RS/6000 Systems SGI IRIX-MIPS Systems SunOS 5 on SPARC Using Exception Data on Compaq Tru64 UNIX Linking with the dbfork Library Compaq Tru64 UNIX HP-UX IBM AIX on RS/6000 Systems Linking C++ Programs with dbfork SGI IRIX6-MIPS SunOS 5 SPARC B Operating Supported Operating Systems Mounting the /proc File System Compaq Tru64 UNIX and SunOS 5 SGI IRIX Swap Space Compaq Tru64 UNIX HP HP-UX Maximum Data Size IBM AIX Linux SGI IRIX SunOS 5 Shared Libraries Changing Linkage Table Entries and LD_BIND_NOW Using Shared Libraries on HP-UX Debugging Dynamically Loaded Libraries Known Limitations Remapping Keys Expression System Compaq Alpha Tru64 UNIX IBM AIX SGI IRIX C Compaq Alpha Alpha General Registers Alpha Floating-Point Registers Alpha FPCR Register HP PA-RISC PA-RISC General Registers PA-RISC Process Status Word PA-RISC Floating-Point Registers PA-RISC Floating-Point Format IBM Power Power General Registers Power MSR Register Power Floating-Point Registers Power FPSCR Register Using the Power FPSCR Register Intel-x86 Intel-x86 General Registers Intel-x86 Floating-Point Registers Intel-x86 FPCR Register Using the Intel-x86 FPCR Register Intel-x86 FPSR Register SGI MIPS MIPS General Registers MIPS SR Register MIPS Floating-Point Registers MIPS FCSR Register Using the MIPS FCSR Register MIPS Delay Slot Instructions Sun SPARC SPARC General Registers SPARC PSR Register SPARC Floating-Point Registers SPARC FPSR Register Using the SPARC FPSR Register Glossary Citations Index