Skip to main content
Ctrl+K
HPCToolkit - Home
  • HPCViewer
  • Download as EPUB
  • Download as PDF
  • 1. Introduction
  • 2. HPCToolkit Overview
  • 3. Quick Start
  • 4. Installing with Spack
  • 5. Meson Builds for Developers
  • 6. Monitoring Processes
  • 7. Monitoring MPI
  • 8. Monitoring OpenMP
  • 9. Monitoring GPU Operations
  • 10. Hpcviewer
  • 11. Strategies for Insight
  • 12. Known Issues
  • 13. FAQ and Troubleshooting
  • 14. Environment Variables
  • 15. Getting Help
  • Show source
  • Suggest edit
  • Open issue
  • .md

  • 1. Introduction
  • 2. HPCToolkit Overview
    • 2.1. Call Path Profiling
    • 2.2. Recovering Program Structure
    • 2.3. Attributing Performance to Code
    • 2.4. Interactive Performance Analysis
  • 3. Quick Start
    • 3.1. Guided Tour
      • 3.1.1. Compiling an Application
      • 3.1.2. Measuring Application Performance
        • 3.1.2.1. Specifying CPU Sample Sources
        • 3.1.2.2. Measuring GPU Computations
      • 3.1.3. Recovering Program Structure
        • 3.1.3.1. Caching Structure Results
      • 3.1.4. Attributing Performance to Code
      • 3.1.5. Interactive Performance Analysis
      • 3.1.6. Effective Analysis Strategies
    • 3.2. Additional Guidance
  • 4. Installing with Spack
    • 4.1. Config Files
      • 4.1.1. Config.yaml
      • 4.1.2. Modules.yaml
    • 4.2. Installing HPCToolkit
      • 4.2.1. Basic Installation
      • 4.2.2. Configuration Options
        • 4.2.2.1. CUDA (+cuda)
        • 4.2.2.2. Level Zero (+level_zero)
        • 4.2.2.3. ROCm (+rocm)
        • 4.2.2.4. OpenCL (+opencl)
        • 4.2.2.5. MPI (+mpi)
        • 4.2.2.6. PAPI (+papi)
        • 4.2.2.7. Python (+python)
    • 4.3. Installing Hpcviewer
    • 4.4. Spack for Beginners
  • 5. Meson Builds for Developers
    • 5.1. Quickstart
    • 5.2. Configuration
    • 5.3. Installing Dependencies without Root
      • 5.3.1. Meson Wraps
      • 5.3.2. Dev Containers (BETA)
    • 5.4. Installing Dependencies as Root
      • 5.4.1. Debian/Ubuntu and Derivatives
      • 5.4.2. Fedora/RHEL and Derivatives
      • 5.4.3. SUSE Leap/SLES 15 and Derivatives
    • 5.5. Custom Dependencies
    • 5.6. Links to Meson Documentation
  • 6. Monitoring Processes
    • 6.1. Using hpcrun
      • 6.1.1. If hpcrun causes your application to fail
        • 6.1.1.1. hpcrun causes failures related to loading or using shared libraries
        • 6.1.1.2. hpcrun causes your application to fail when gprof instrumentation is present
    • 6.2. Hardware Counter Event Names
    • 6.3. Sample Sources
      • 6.3.1. Linux perf_events
        • 6.3.1.1. Capabilities of perf_events
          • 6.3.1.1.1. Frequency-based sampling
          • 6.3.1.1.2. Multiplexing
          • 6.3.1.1.3. Thread blocking
        • 6.3.1.2. Profiling with perf_events
        • 6.3.1.3. Notes
      • 6.3.2. PAPI
        • 6.3.2.1. Proxy Sampling
      • 6.3.3. REALTIME and CPUTIME
      • 6.3.4. IO
      • 6.3.5. MEMLEAK
    • 6.4. Experimental Python Support
      • 6.4.1. Known Limitations
    • 6.5. Process Fraction
    • 6.6. API to Start and Stop Sampling
    • 6.7. Environment Variables for hpcrun
  • 7. Monitoring MPI
    • 7.1. Measuring MPI Ranks
  • 8. Monitoring OpenMP
    • 8.1. Monitoring OpenMP on the Host
    • 8.2. Monitoring OpenMP Offloading on GPUs
      • 8.2.1. AMD OpenMP
      • 8.2.2. Intel OneAPI OpenMP
      • 8.2.3. NVIDIA OpenMP
      • 8.2.4. HPE’s Cray OpenMP
      • 8.2.5. GCC OpenMP
  • 9. Monitoring GPU Operations
    • 9.1. GPU Measurement Quickstart
    • 9.2. Profiling GPU Operations
    • 9.3. Tracing GPU Operations
    • 9.4. PC Sampling within GPU Kernels
      • 9.4.1. Attributing PC Samples to Source Code
    • 9.5. AMD GPUs
      • 9.5.1. AMD GPU PC Sampling
        • 9.5.1.1. Attributing PC Samples to Source Code
      • 9.5.2. AMD GPU Hardware Counters
    • 9.6. NVIDIA GPUs
      • 9.6.1. NVIDIA GPU PC Sampling
      • 9.6.2. Attributing PC Samples to CUDA Source Code
      • 9.6.3. Binary analysis on NVIDIA GPUs
    • 9.7. Intel GPUs
      • 9.7.1. Intel GPU PC Sampling
      • 9.7.2. Attributing PC Samples to Source Code
  • 10. Hpcviewer
    • 10.1. Overview
      • 10.1.1. Downloading
      • 10.1.2. Building from Source
      • 10.1.3. Launching
      • 10.1.4. Menus
        • 10.1.4.1. File
        • 10.1.4.2. Filter
        • 10.1.4.3. View
        • 10.1.4.4. Help
      • 10.1.5. Limitations
    • 10.2. Profile View
      • 10.2.1. Panes
        • 10.2.1.1. Source Pane
        • 10.2.1.2. Navigation Pane
          • 10.2.1.2.1. Control Panel
          • 10.2.1.2.2. Context menus
        • 10.2.1.3. Metric Pane
      • 10.2.2. Understanding Metrics
        • 10.2.2.1. How Metrics are Computed
        • 10.2.2.2. Example
      • 10.2.3. Derived Metrics
        • 10.2.3.1. Formulae
        • 10.2.3.2. Examples
        • 10.2.3.3. Creating Derived Metrics
      • 10.2.4. Metrics in Execution-context level
        • 10.2.4.1. Plot Graphs
        • 10.2.4.2. Thread View
      • 10.2.5. Filtering Tree Nodes
      • 10.2.6. Convenience Features
        • 10.2.6.1. Source Code Pane
        • 10.2.6.2. Metric Pane
    • 10.3. Trace view
      • 10.3.1. Action and Information Pane
      • 10.3.2. Customizing the Color Map
      • 10.3.3. Filtering Execution Contexts
        • 10.3.3.1. Filtering Suggestions
    • 10.4. Accessing Remote Databases
      • 10.4.1. Building and Installing hpcserver
      • 10.4.2. Opening a Remote Database
  • 11. Strategies for Insight
    • 11.1. Monitoring High-Latency Events
    • 11.2. Computing Derived Metrics
    • 11.3. Pinpointing and Quantifying Inefficiencies
    • 11.4. Analyzing Scalability Bottlenecks
      • 11.4.1. Scalability Analysis Using Expectations
        • 11.4.1.1. Weak Scaling
        • 11.4.1.2. Exploring Scaling Losses
  • 12. Known Issues
    • 12.1. Limited cross-version compatibility with ROCm
    • 12.2. No support for CUDA 13
    • 12.3. When monitoring applications that use ROCm, using LD_AUDIT in hpcrun may cause it to fail to elide OpenMP runtime frames
    • 12.4. When using instrumentation on Intel GPUs, hpcrun may report that substantial time is spent in a partial call path consisting of only an unknown procedure
    • 12.5. hpcrun reports partial call paths for code executed by a constructor prior to entering main
    • 12.6. hpcrun may fail to measure a program execution on a CPU with hardware performance counters
    • 12.7. hpcrun may associate several profiles and traces with rank 0, thread 0
    • 12.8. hpcrun sometimes enables writing of read-only data
    • 12.9. A confusing label for GPU theoretical occupancy
  • 13. FAQ and Troubleshooting
    • 13.1. General Measurement Failures
      • 13.1.1. Profiling setuid programs
      • 13.1.2. Problems loading dynamic libraries
      • 13.1.3. Problems caused by gprof instrumentation
    • 13.2. Measurement Failures using NVIDIA GPUs
      • 13.2.1. Deadlock while monitoring a program that uses IBM Spectrum MPI and NVIDIA GPUs
      • 13.2.2. Ensuring permission to use GPU performance counters
      • 13.2.3. Avoiding the error cudaErrorUnknown
      • 13.2.4. Avoiding the error CUPTI_ERROR_NOT_INITIALIZED
      • 13.2.5. Avoiding the error CUPTI_ERROR_HARDWARE_BUSY
      • 13.2.6. Avoiding the error CUPTI_ERROR_UNKNOWN
    • 13.3. General Measurement Issues
      • 13.3.1. How do I choose sampling periods?
      • 13.3.2. Why do I see partial unwinds?
      • 13.3.3. Measurement with HPCToolkit has high overhead! Why?
      • 13.3.4. Some of my syscalls return EINTR
      • 13.3.5. My application spends a lot of time in C library functions with names that include mcount
    • 13.4. Problems Recovering Loops in NVIDIA GPU binaries
    • 13.5. Graphical User Interface Issues
      • 13.5.1. hpcviewer fails to launch
      • 13.5.2. Fail to run hpcviewer: executable launcher was unable to locate its companion shared library
      • 13.5.3. Launching hpcviewer is very slow on Windows
      • 13.5.4. Mac only: hpcviewer runs on Java X instead of “Java 17”
      • 13.5.5. When executing hpcviewer, it complains cannot create “Java Virtual Machine”
      • 13.5.6. hpcviewer fails to launch due to java.lang.NoSuchMethodError exception.
      • 13.5.7. hpcviewer fails due to java.lang.OutOfMemoryError exception.
      • 13.5.8. hpcviewer writes a long list of Java error messages to the terminal!
      • 13.5.9. hpcviewer attributes performance information only to functions and not to source code loops and lines! Why?
      • 13.5.10. hpcviewer hangs trying to open a large database! Why?
      • 13.5.11. hpcviewer runs glacially slowly! Why?
      • 13.5.12. hpcviewer does not show my source code! Why?
        • 13.5.12.1. An explanation how HPCToolkit finds source files
      • 13.5.13. hpcviewer’s reported line numbers do not exactly correspond to what I see in my source code! Why?
      • 13.5.14. hpcviewer claims that there are several calls to a function within a particular source code scope, but my source code only has one! Why?
      • 13.5.15. hpcviewer’s Trace view shows lots of white space on the left. Why?
    • 13.6. Debugging
      • 13.6.1. How do I debug HPCToolkit’s measurement?
      • 13.6.2. Tracing HPCToolkit’s Measurement Subsystem
      • 13.6.3. Using a debugger to inspect an execution being monitored by HPCToolkit
  • 14. Environment Variables
    • 14.1. Environment Variables for Users
    • 14.2. Environment Variables that May Avoid a Crash
    • 14.3. Environment Variables for Developers
  • 15. Getting Help

previous

HPCToolkit User Manual

next

1. Introduction

By The HPCToolkit Developers

© Copyright HPCToolkit Project a Series of LF Projects, LLC.