HPCToolkit User Manual

HPCToolkit is an integrated suite of tools for measurement and analysis of program performance on computers ranging from multicore desktop systems to the world’s largest GPU-accelerated supercomputers. HPCToolkit can measure a program’s work, resource consumption, and inefficiency on both CPUs and GPUs (Adhianto et al. 2010; Zhou et al. 2020; Adhianto et al. 2024). HPCToolkit correlates such metrics with the program’s source code, works with multilingual, fully optimized binaries, has very low measurement overhead, and scales to large parallel systems. HPCToolkit’s measurements provide support for analyzing a program execution cost, inefficiency, and scaling characteristics both within and across nodes of a parallel system.

• Introduction
• HPCToolkit Overview
  • Asynchronous Sampling and Call Path Profiling
  • Recovering Static Program Structure
  • Aggregating and Attributing Performance Measurements
  • Presenting Performance Measurements
• Quick Start
  • Guided Tour
  • Additional Guidance
• Installing HPCToolkit with Spack
  • Config Files
  • Installing a Basic HPCToolkit
  • Installing Hpcviewer
  • Configuration Options
  • Running Spack for the First Time
• Building from Source with Meson
  • Quickstart
  • Configuration
  • Installing Dependencies without Root
  • Installing Dependencies as Root
  • Custom Dependencies
  • Meson Documentation References
• Effective Strategies for Analyzing Program Performance
  • Monitoring High-Latency Penalty Events
  • Computing Derived Metrics
  • Pinpointing and Quantifying Inefficiencies
  • Pinpointing and Quantifying Scalability Bottlenecks
• Monitoring Dynamically-linked Applications with hpcrun
  • Using hpcrun
  • Hardware Counter Event Names
  • Sample Sources
  • Experimental Python Support
  • Process Fraction
  • API to Start and Stop Sampling
  • Environment Variables for hpcrun
  • Cray System Specific Notes
• Monitoring MPI Applications
  • Running and Analyzing MPI Programs
  • Building and Installing HPCToolkit
• Measurement and Analysis of GPU-accelerated Applications
  • GPU Performance Measurement Substrate
  • NVIDIA GPUs
  • AMD GPUs
  • Intel GPUs
  • Performance Measurement of OpenCL Programs
• Measurement and Analysis of OpenMP Multithreading
  • Monitoring OpenMP on the Host
  • Monitoring OpenMP Offloading on GPUs
• Hpcviewer
• Overview
  • Downloading
  • Building from Source
  • Launching
  • Menus
  • Limitations
• Profile View
  • Panes
  • Understanding Metrics
  • Derived Metrics
  • Metrics in Execution-context level
  • Filtering Tree Nodes
  • Convenience Features
• Trace view
  • Action and Information Pane
  • Customizing the Color Map
  • Filtering Execution Contexts
• Accessing Remote Databases
  • Building and Installing hpcserver
  • Opening a Remote Database
• Known Issues
  • No support for CUDA 13
  • Using Level Zero, time may be observed as non-monotonic
  • When monitoring applications that use ROCm using LD_AUDIT in hpcrun may cause it to fail to elide OpenMP runtime frames
  • When using Intel GPUs, hpcrun may report that substantial time is spent in a partial call path consisting of only an unknown procedure
  • hpcrun reports partial call paths for code executed by a constructor prior to entering main
  • hpcrun may fail to measure a program execution on a CPU with hardware performance counters
  • hpcrun may associate several profiles and traces with rank 0, thread 0
  • hpcrun sometimes enables writing of read-only data
  • A confusing label for GPU theoretical occupancy
• FAQ and Troubleshooting
  • General Measurement Failures
  • Measurement Failures using NVIDIA GPUs
  • General Measurement Issues
  • Problems Recovering Loops in NVIDIA GPU binaries
  • Graphical User Interface Issues
  • Debugging
• Environment Variables
  • Environment Variables for Users
  • Environment Variables that May Avoid a Crash
  • Environment Variables for Developers
• Getting Help