# OProfile manual

### John Levon

`<levon@movementarian.org>`

Copyright © 2000-2004 Victoria University of Manchester, John Levon and others

## Chapter 1. Introduction

This manual applies to OProfile version 0.8cvs. OProfile is a profiling system for Linux 2.2/2.4/2.6 systems on a number of architectures. It is capable of profiling all parts of a running system, from the kernel (including modules and interrupt handlers) to shared libraries to binaries. It runs transparently in the background collecting information at a low overhead. These features make it ideal for profiling entire systems to determine bottle necks in real-world systems.

Many CPUs provide "performance counters", hardware registers that can count "events"; for example, cache misses, or CPU cycles. OProfile provides profiles of code based on the number of these occurring events: repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded. This information is aggregated into profiles for each binary image.

Some hardware setups do not allow OProfile to use performance counters: in these cases, no events are available, and OProfile operates in timer/RTC mode, as described in later chapters.

## 1. Applications of OProfile

OProfile is useful in a number of situations. You might want to use OProfile when you :

- need low overhead

- cannot use highly intrusive profiling methods

- need to profile interrupt handlers

- need to profile an application and its shared libraries

- need to capture the performance behaviour of entire system

- want to examine hardware effects such as cache misses

- want detailed source annotation

- want instruction-level profiles

- want call-graph profiles

OProfile is not a panacea. OProfile might not be a complete solution when you :

- require call graph profiles on platforms other than 2.6/x86

- don't have root permissions

- require 100% instruction-accurate profiles

- need function call counts or an interstitial profiling API

- cannot tolerate any disturbance to the system whatsoever

- need to profile interpreted or dynamically compiled code such as Java or Python

## 2. System requirements

Linux kernel 2.2/2.4/2.6

  OProfile uses a kernel module that can be compiled for 2.2.11 or later and 2.4. Versions 2.4.10 or above are recommended, and required if you use the boot-time kernel option `nosmp`. AMD64 processors support requires a recent (>= 2.4.19) kernel with the line `EXPORT_SYMBOL(do_fork);` present in `kernel/ksyms.c`. Such a kernel is present in the x86-64.org CVS repository. 2.6 kernels are supported with the in-kernel OProfile driver.

modutils 2.4.6 or above

  You should have installed modutils 2.4.6 or higher (in fact earlier versions work well in almost all cases).

Supported architecture

  For Intel IA32, a CPU with either a P6 generation or Pentium 4 core is required. In marketing terms this translates to anything between an Intel Pentium Pro (not Pentium Classics) and a Pentium 4 / Xeon, including all Celerons. The AMD Athlon, Duron, and Hammer CPUs are also supported. Other IA32 CPU types only support the RTC mode of OProfile; please see later in this manual for details. Hyper-threaded Pentium IVs are not supported in 2.4. For 2.4 kernels, the Intel IA-64 CPUs are also supported. For 2.6 kernels, there is additionally support for Alpha processors, and sparc64, ppc64, and PA-RISC in timer mode.

Uniprocessor or SMP

  SMP machines are fully supported.

Required libraries

  These libraries are required : `popt`, `bfd`, `liberty` (debian users: libiberty is provided in binutils-dev package), `dl`, plus the standard C++ libraries.

Bash version 2

  The opcontrol script requires bash version 2 at least to be installed as `/bin/bash` or `/bin/bash2`

OProfile GUI

  The use of the GUI to start the profiler requires the `Qt 2` library. `Qt 3` should also work.

ELF

  Probably not too strenuous a requirement, but older A.OUT binaries/libraries are not supported.

K&R coding style

  OK, so it's not really a requirement, but I wish it was...

## 3. Internet resources

Web page

  There is a web page (which you may be reading now) at http://oprofile.sf.net/.

Download

  You can download a source tarball or get anonymous CVS at the sourceforge page, http://sf.net/projects/oprofile/.

Mailing list

  There is a low-traffic OProfile-specific mailing list, details at http://sf.net/mail/?group_id=16191.

Bug tracker

  There is a bug tracker for OProfile at SourceForge, http://sf.net/tracker/?group_id=16191&atid=116191.

IRC channel

  Several OProfile developers and users sometimes hang out on channel #oprofile on the freenode network.

## 4. Installation

First you need to build OProfile and install it. ./configure, make, make install is often all you need, but note these arguments to ./configure :

`--with-linux`

  Use this option to specify the location of the kernel source tree you wish to compile against. The kernel module is built against this source and will only work with a running kernel built from the same source with exact same options, so it is important you specify this option if you need to.

`--with-kernel-support`

  Use this option with 2.6 and above kernels to indicate the kernel provides the OProfile device driver.

`--with-qt-dir/includes/libraries`

  Specify the location of Qt headers and libraries. It defaults to searching in `$QTDIR` if these are not specified.

`--enable-abi`

  Activate code within the OProfile sample collection daemon oprofiled which records information about the binary format of sample files in `/var/lib/oprofile/abi`, to permit their transport between hosts using the op_import utility. See `op_import`. This option is primarily intended for embedded systems or remote analysis of production machines; if you will be performing all sample analysis on the same machine as you are profiling, it is safe to omit this option.

`--disable-werror`

  Development versions of OProfile build by default with `-Werror`. This option turns `-Werror` off.

`--disable-optimization`

  Disable the `-O2` compiler flag (useful if you discover an OProfile bug and want to give a useful back-trace etc.)

You'll need to have a configured kernel source for the current kernel to build the module for 2.4 kernels. Since all distributions provide different kernels it's unlikely the running kernel match the configured source you installed. The safest way is to recompile your own kernel, run it and compile oprofile. It is also recommended that if you have a uniprocessor machine, you enable the local APIC / IO_APIC support for your kernel (this is automatically enabled for SMP kernels). On machines with power management, such as laptops, the power management must be turned off when using OProfile with 2.4 kernels. The power management software in the BIOS cannot handle the non-maskable interrupts (NMIs) used by OProfile for data collection. If you use the NMI watchdog, be aware that the watchdog is disabled when profiling starts, and not re-enabled until the OProfile module is removed (or, in 2.6, when OProfile is not running). If you compile OProfile for a 2.2 kernel you must be root to compile the module. If you are using 2.6 kernels or higher, you do not need kernel source, as long as the OProfile driver is enabled; additionally, you should not need to disable power management.

Please note that you must save or have available the `vmlinux` file generated during a kernel compile, as OProfile needs it (you can use `--no-vmlinux`, but this will prevent kernel profiling).

## 5. Uninstalling OProfile

You must have the source tree available to uninstall OProfile; a make uninstall will remove all installed files except your configuration file in the directory `~/.oprofile`.

## Chapter 2. Overview

## 1. Getting started

Before you can use OProfile, you must set it up. The minimum setup required for this is to tell OProfile where the `vmlinux` file corresponding to the running kernel is, for example :

    opcontrol --vmlinux=/boot/vmlinux-`uname -r`

If you don't want to profile the kernel itself, you can tell OProfile you don't have a `vmlinux` file :

    opcontrol --no-vmlinux

Now we are ready to start the daemon (oprofiled) which collects the profile data :

    opcontrol --start

When I want to stop profiling, I can do so with :

    opcontrol --shutdown

Note that unlike gprof, no instrumentation (`-pg` and `-a` options to gcc) is necessary.

Periodically (or on opcontrol --shutdown or opcontrol --dump) the profile data is written out into the `/var/lib/oprofile/samples` directory. These profile files cover shared libraries, applications, the kernel (vmlinux), and kernel modules. You can clear the profile data (at any time) with opcontrol --reset.

You can get summaries of this data in a number of ways at any time. To get a summary of data across the entire system for all of these profiles, you can do :

    opreport

Or to get a more detailed summary, for a particular image, you can do something like :

    opreport -l /boot/vmlinux-`uname -r`

There are also a number of other ways of presenting the data, as described later in this manual. Note that OProfile will choose a default profiling setup for you. However, there are a number of options you can pass to opcontrol if you need to change something, also detailed later.

## 2. Tools summary

This section gives a brief description of the available OProfile utilities and their purpose.

`op_help`

  This utility lists the available events and short descriptions.

`opcontrol`

  Used for controlling the OProfile data collection, discussed in Chapter 3.

`opreport`

  This is the main tool for retrieving useful profile data, described in Section 1.

`opannotate`

  This utility can be used to produce annotated source, assembly or mixed source/assembly. Source level annotation is available only if the application was compiled with debugging symbols. See Section 2.

`opstack`

  This utility can output call-graph profiles. This require an x86 based box and a 2.6 box with the call-graph patch. See Section 3.

`opgprof`

  This utility can output gprof-style data files for a binary, for use with gprof -p. See Section 4.

`op_import`

  This utility converts sample database files from a foreign binary format (abi) to the native format. This is useful only when moving sample files between hosts, for analysis on platforms other than the one used for collection. The abi format of the file to be imported is described in a text file located in `/var/lib/oprofile/abi`, if the `--enable-abi` configure-time option was enabled. Furthermore, the op_import tool is not built unless `--enable-abi` is given. See `--enable-abi`.

## Chapter 3. Controlling the profiler

## 1. Using opcontrol

In this section we describe the configuration and control of the profiling system with opcontrol in more depth. The opcontrol script has a default setup, but you can alter this with the options given below. In particular, if your hardware supports performance counters, you can configure them. There are a number of counters (for example, counter 0 and counter 1 on the Pentium III). Each of these counters can be programmed with an event to count, such as cache misses or MMX operations. The event chosen for each counter is reflected in the profile data collected by OProfile: functions and binaries at the top of the profiles reflect that most of the chosen events happened within that code.

Additionally, each counter has a "count" value: this corresponds to how detailed the profile is. The lower the value, the more frequently profile samples are taken. A counter can choose to sample only kernel code, user-space code, or both (both is the default). Finally, some events have a "unit mask" - this is a value that further restricts the types of event that are counted. The event types and unit masks for your CPU are listed by opcontrol --list-events.

The opcontrol script provides the following actions :

`--init`

  Loads the OProfile module if required and makes the OProfile driver interface available.

`--setup`

  Followed by list arguments for profiling set up. List of arguments saved in `/root/.oprofile/daemonrc`. Giving this option is not necessary; you can just directly pass one of the setup options, e.g. opcontrol --no-vmlinux.

`--start-daemon`

  Start the oprofile daemon without starting actual profiling. The profiling can then be started using `--start`. This is useful for avoiding measuring the cost of daemon startup, as `--start` is a simple write to a file in oprofilefs. Not available in 2.2/2.4 kernels.

`--start`

  Start data collection with either arguments provided by `--setup` or information saved in `/root/.oprofile/daemonrc`. Specifying the addition `--verbose` makes the daemon generate lots of debug data whilst it is running.

`--dump`

  Force a flush of the collected profiling data to the daemon.

`--stop`

  Stop data collection (this separate step is not possible with 2.2 or 2.4 kernels).

`--shutdown`

  Stop data collection and kill the daemon.

`--reset`

  Clears out data from current session, but leaves saved sessions.

`--save=`session_name

  Save data from current session to session_name.

`--deinit`

  Shuts down daemon. Unload the OProfile module and oprofilefs.

`--list-events`

  List event types and unit masks.

`--help`

  Generate usage messages.

There are a number of possible settings, of which, only `--vmlinux` (or `--no-vmlinux`) is required. These settings are stored in `~/.oprofile/daemonrc`.

`--buffer-size=`num

  Number of samples in kernel buffer.

`--cpu-buffer-size=`num

  Number of samples in kernel per-cpu buffer (2.6 only). If you profile at high rate it can help to increase this if the log file show excessive count of sample lost cpu buffer overflow.

`--event=`[eventspec]

  Use the given performance counter event to profile. See Section 1.2 below.

`--separate=`[none,library,kernel,thread,cpu,all]

  By default, every profile is stored in a single file. Thus, for example, samples in the C library are all accredited to the `/lib/libc.o` profile. However, you choose to create separate sample files by specifying one of the below options.

  `none` | No profile separation (default)
  `library` | Create per-application profiles for libraries
  `kernel` | Create per-application profiles for the kernel and kernel modules
  `thread` | Create profiles for each thread and each task
  `cpu` | Create profiles for each CPU
  `all` | All of the above options

  Note that `--separate=kernel` also turns on `--separate=library`. When using `--separate=kernel`, samples in hardware interrupts, soft-irqs, or other asynchronous kernel contexts are credited to the task currently running. This means you will see seemingly nonsense profiles such as `/bin/bash` showing samples for the PPP modules, etc.

  On 2.2/2.4 only kernel threads already started when profiling begins are correctly profiled; newly started kernel thread samples are credited to the vmlinux (kernel) profile.

  Using `--separate=thread` creates a lot of sample files if you leave OProfile running for a while; it's most useful when used for short sessions, or when using image filtering.

`--callgraph=`#depth

  Enable call-graph sample collection with a maximum depth. Use 0 to disable callgraph profiling. Please make sure to read Section 3.

`--image=`image,[images]|"all"

  Image filtering. If you specify one or more absolute paths to binaries, OProfile will only produce profile results for those binary images. This is useful for restricting the sometimes voluminous output you may get otherwise, especially with `--separate=thread`. Note that if you are using `--separate=library` or `--separate=kernel`, then if you specification an application binary, the shared libraries and kernel code are included. Specify the value "all" to profile everything (the default).

`--vmlinux=`file

  vmlinux kernel image.

`--no-vmlinux`

  Use this when you don't have a kernel vmlinux file, and you don't want to profile the kernel. This still counts the total number of kernel samples, but can't give symbol-based results for the kernel or any modules.

### 1.1. Examples

#### 1.1.1. Intel performance counter setup

Here, we have a Pentium III running at 800MHz, and we want to look at where data memory references are happening most, and also get results for CPU time.

    # opcontrol --event=CPU_CLK_UNHALTED:400000 --event=DATA_MEM_REFS:10000
    # opcontrol --vmlinux=/boot/2.6.0/vmlinux
    # opcontrol --start

#### 1.1.2. RTC mode

Here, we have an Intel laptop without support for performance counters, running on 2.4 kernels.

    # op_help -r
    CPU with RTC device
    # opcontrol --vmlinux=/boot/2.6.0/vmlinux --event=RTC_INTERRUPTS:1024
    # opcontrol --start

#### 1.1.3. Starting the daemon separately

If we're running 2.6 kernels, we can use `--start-daemon` to avoid the profiler startup affecting results.

    # opcontrol --vmlinux=/boot/2.6.0/vmlinux
    # opcontrol --start-daemon
    # my_favourite_benchmark --init
    # opcontrol --start ; my_favourite_benchmark --run ; opcontrol --stop

#### 1.1.4. Separate profiles for libraries and the kernel

Here, we want to see a profile of the OProfile daemon itself, including when it was running inside the kernel driver, and its use of shared libraries.

    # opcontrol --separate=kernel --vmlinux=/boot/2.6.0/vmlinux
    # opcontrol --start
    # my_favourite_stress_test --run
    # opreport -l -p /lib/modules/2.6.0/kernel /usr/local/bin/oprofiled

#### 1.1.5. Profiling sessions

It can often be useful to split up profiling data into several different time periods. For example, you may want to collect data on an application's startup separately from the normal runtime data. You can use the simple command opcontrol --save to do this. For example :

    # opcontrol --save=blah

will create a sub-directory in `/var/lib/oprofile/samples` containing the samples up to that point (the current session's sample files are moved into this directory). You can then pass this session name as a parameter to the post-profiling analysis tools, to only get data up to the point you named the session. If you do not want to save a session, you can do rm -rf /var/lib/oprofile/samples/sessionname or, for the current session, opcontrol --reset.

### 1.2. Specifying performance counter events

The `--event` option to opcontrol takes a specification that indicates how the details of each hardware performance counter should be setup. If you want to revert to OProfile's default setting (`--event` is strictly optional), use `--event=default`.

You can pass multiple event specifications. OProfile will allocate hardware counters as necessary. Note that some combinations are not allowed by the CPU; running opcontrol --list-events gives the details of each event. The event specification is a colon-separated string of the form `name:count:unitmask:kernel:user` as described in this table:

`name` | The symbolic event name, e.g. `CPU_CLK_UNHALTED`
`count` | The counter reset value, e.g. 100000
`unitmask` | The unit mask, as given in the events list, e.g. 0x0f
`kernel` | Whether to profile kernel code
`user` | Whether to profile userspace code

The last three values are optional, if you omit them (e.g. `--event=DATA_MEM_REFS:30000`), they will be set to the default values (a unit mask of 0, and profiling both kernel and userspace code). Note that some events require a unit mask.

If OProfile is using RTC mode, and you want to alter the default counter value, you can use something like `--event=RTC_INTERRUPTS:2048`. Note the last three values here are ignored. If OProfile is using timer-interrupt mode, there is no configuration possible.

## 2. Using oprof_start

The oprof_start application provides a convenient way to start the profiler. Note that oprof_start is just a wrapper around the opcontrol script, so it does not provide more services than the script itself.

After oprof_start is started you can select the event type for each counter; the sampling rate and other related parameters are explained in Section 1. The "Configuration" section allows you to set general parameters such as the buffer size, kernel filename etc. The counter setup interface should be self-explanatory; Section 3.1 and related links contain information on using unit masks.

A status line shows the current status of the profiler: how long it has been running, and the average number of interrupts received per second and the total, over all processors. Note that quitting oprof_start does not stop the profiler.

Your configuration is saved in the same file as opcontrol uses; that is, `~/.oprofile/daemonrc`.

## 3. Configuration details

### 3.1. Hardware performance counters

### Note

Your CPU type may not include the requisite support for hardware performance counters, in which case you must use OProfile in RTC mode in 2.4 (see Section 3.2), or timer mode in 2.6 (see Section 3.3). You do not really need to read this section unless you are interested in using events other than the default event chosen by OProfile.

The hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available from http://developer.intel.com/. The AMD Athlon/Duron implementation is detailed in http://www.amd.com/products/cpg/athlon/techdocs/pdf/22007.pdf. These processors are capable of delivering an interrupt when a counter overflows. This is the basic mechanism on which OProfile is based. The delivery mode is NMI, so blocking interrupts in the kernel does not prevent profiling. When the interrupt handler is called, the current PC value and the current task are recorded into the profiling structure. This allows the overflow event to be attached to a specific assembly instruction in a binary image. The daemon receives this data from the kernel, and writes it to the sample files.

If we use an event such as `CPU_CLK_UNHALTED` or `INST_RETIRED` (`GLOBAL_POWER_EVENTS` or `INSTR_RETIRED`, respectively, on the Pentium 4), we can use the overflow counts as an estimate of actual time spent in each part of code. Alternatively we can profile interesting data such as the cache behaviour of routines with the other available counters.

However there are several caveats. First, there are those issues listed in the Intel manual. There is a delay between the counter overflow and the interrupt delivery that can skew results on a small scale - this means you cannot rely on the profiles at the instruction level as being perfectly accurate. If you are using an "event-mode" counter such as the cache counters, a count registered against it doesn't mean that it is responsible for that event. However, it implies that the counter overflowed in the dynamic vicinity of that instruction, to within a few instructions. Further details on this problem can be found in Chapter 5 and also in the Digital paper "ProfileMe: A Hardware Performance Counter".

Each counter has several configuration parameters. First, there is the unit mask: this simply further specifies what to count. Second, there is the counter value, discussed below. Third, there is a parameter whether to increment counts whilst in kernel or user space. You can configure these separately for each counter.

After each overflow event, the counter will be re-initialized such that another overflow will occur after this many events have been counted. Thus, higher values mean less-detailed profiling, and lower values mean more detail, but higher overhead. Picking a good value for this parameter is, unfortunately, somewhat of a black art. It is of course dependent on the event you have chosen. Specifying too large a value will mean not enough interrupts are generated to give a realistic profile (though this problem can be ameliorated by profiling for longer). Specifying too small a value can lead to higher performance overhead.

### 3.2. OProfile in RTC mode

### Note

This section applies to 2.2/2.4 kernels only.

Some CPU types do not provide the needed hardware support to use the hardware performance counters. This includes some laptops, classic Pentiums, and other CPU types not yet supported by OProfile (such as Cyrix). On these machines, OProfile falls back to using the real-time clock interrupt to collect samples. This interrupt is also used by the rtc module: you cannot have both the OProfile and rtc modules loaded nor the rtc support compiled in the kernel.

RTC mode is less capable than the hardware counters mode; in particular, it is unable to profile sections of the kernel where interrupts are disabled. There is just one available event, "RTC interrupts", and its value corresponds to the number of interrupts generated per second (that is, a higher number means a better profiling resolution, and higher overhead). The current implementation of the real-time clock supports only power-of-two sampling rates from 2 to 4096 per second. Other values within this range are rounded to the nearest power of two.

Setting the value from the GUI should be straightforward. On the command line, you need to specify the event to opcontrol, e.g. :

opcontrol --event=RTC_INTERRUPTS:256

### 3.3. OProfile in timer interrupt mode

### Note

This section applies to 2.6 kernels and above only.

In 2.6 kernels on CPUs without OProfile support for the hardware performance counters, the driver falls back to using the timer interrupt for profiling. Like the RTC mode in 2.4 kernels, this is not able to profile code that has interrupts disabled. Note that there are no configuration parameters for setting this, unlike the RTC and hardware performance counter setup.

You can force use of the timer interrupt by using the `timer=1` module parameter (or `oprofile.timer=1` on the boot command line if OProfile is built-in).

### 3.4. Pentium 4 support

The Pentium 4 / Xeon performance counters are organized around 3 types of model specific registers (MSRs): 45 event selection control registers (ESCRs), 18 counter configuration control registers (CCCRs) and 18 counters. ESCRs describe a particular set of events which are to be recorded, and CCCRs bind ESCRs to counters and configure their operation. Unfortunately the relationship between these registers is quite complex; they cannot all be used with one another at any time. There is, however, a subset of 8 counters, 8 ESCRs, and 8 CCCRs which can be used independently of one another, so OProfile only accesses those registers, treating them as a bank of 8 "normal" counters, similar to those in the P6 or Athlon families of CPU.

There is currently no support for Precision Event-Based Sampling (PEBS), nor any advanced uses of the Debug Store (DS). Current support is limited to the conservative extension of OProfile's existing interrupt-based model described above. Performance monitoring hardware on Pentium 4 / Xeon processors with Hyperthreading enabled (multiple logical processors on a single die) is not supported in 2.4 kernels (you can use OProfile if you disable hyper-threading, though).

### 3.5. Intel Itanium 2 support

The Itanium 2 performance monitoring unit (PMU) organizes the counters as four pairs of performance event monitoring registers. Each pair is composed of a Performance Monitoring Configuration (PMC) register and Performance Monitoring Data (PMD) register. The PMC selects the performance event being monitored and the PMD determines the sampling interval. The IA64 Performance Monitoring Unit (PMU) triggers sampling with maskable interrupts. Thus, samples will not occur in sections of the IA64 kernel where interrupts are disabled.

None of the advance features of the Itanium 2 performance monitoring unit such as opcode matching, address range matching, or precise event sampling are supported by this version of OProfile. The Itanium 2 support only maps OProfile's existing interrupt-based model to the PMU hardware.

### 3.6. Dangerous counter settings

OProfile is a low-level profiler which allow continuous profiling with a low-overhead cost. If too low a count reset value is set for a counter, the system can become overloaded with counter interrupts, and seem as if the system has frozen. Whilst some validation is done, it is not foolproof.

### Note

This can happen as follows: When the profiler count reaches zero an NMI handler is called which stores the sample values in an internal buffer, then resets the counter to its original value. If the count is very low, a pending NMI can be sent before the NMI handler has completed. Due to the priority of the NMI, the local APIC delivers the pending interrupt immediately after completion of the previous interrupt handler, and control never returns to other parts of the system. In this way the system seems to be frozen.

If this happens, it will be impossible to bring the system back to a workable state. There is no way to provide real security against this happening, other than making sure to use a reasonable value for the counter reset. For example, setting `CPU_CLK_UNHALTED` event type with a ridiculously low reset count (e.g. 500) is likely to freeze the system.

In short : Don't try a foolish sample count value. Unfortunately the definition of a foolish value is really dependent on the event type - if ever in doubt, e-mail

`<oprofile-list@lists.sf.net>`.

## 4. Other features

### 4.1. pid/pgrp filter

There are situations where you are only interested in the profiling results of a particular running process, or process tty group. You can set the pid/pgrp values via the `--pid-filter` and `--pgrp-filter` options to opcontrol, which will make the daemon ignore samples for processes that don't match the filter. These options are not available in 2.6 and above kernels.

### 4.2. Unloading the kernel module

### Note

This section applies to 2.2/2.4 kernels only, OProfile in 2.6 can be unloaded safely.

The kernel module can be unloaded, but is designed to take very little memory when profiling is not underway. There is no need to unload the module between profiler runs.

lsmod and similar utilities will still show the module's use count as `-1`. However, this is not to be relied on - the module will become unloadable some short time after stopping profiling.

Note that by default module unloading is disabled when used on SMP systems. This is because of a small chance of a module unload race crashing the kernel. As the race is very small, it is allowed to re-enable the module unload by specifying the "allow_unload" parameter to the module :

modprobe oprofile allow_unload=1

This option can be DANGEROUS and should only be used on non-production systems.

## Chapter 4. Obtaining results

OK, so the profiler has been running, but it's not much use unless we can get some data out. Fairly often, OProfile does a little too good a job of keeping overhead low, and no data reaches the profiler. This can happen on lightly-loaded machines. Remember you can force a dump at any time with :

opcontrol --dump

Remember to do this before complaining there is no profiling data ! Now that we've got some data, it has to be processed. That's the job of opreport, opannotate, opstack or opgprof.

## 1. Image summaries and symbol summaries

The opreport utility is the primary utility you will use for getting formatted data out of OProfile. It produces two types of data: image summaries and symbol summaries. An image summary lists the number of samples for individual binary images such as libraries or applications. Symbol summaries provide per-symbol profile data. In the following example, we're getting an image summary for the whole system:

    $ opreport --long-filenames
    CPU: PIII, speed 863.195 MHz (estimated)
    Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 23150
       905898 59.7415 /usr/lib/gcc-lib/i386-redhat-linux/3.2/cc1plus
       214320 14.1338 /boot/2.6.0/vmlinux
       103450  6.8222 /lib/i686/libc-2.3.2.so
        60160  3.9674 /usr/local/bin/madplay
        31769  2.0951 /usr/local/oprofile-pp/bin/oprofiled
        26550  1.7509 /usr/lib/libartsflow.so.1.0.0
        23906  1.5765 /usr/bin/as
        18770  1.2378 /oprofile
        15528  1.0240 /usr/lib/qt-3.0.5/lib/libqt-mt.so.3.0.5
        11979  0.7900 /usr/X11R6/bin/XFree86
        11328  0.7471 /bin/bash
        ...

If we had specified `--symbols` in the previous command, we would have gotten a symbol summary of all the images across the entire system. We can restrict this to only part of the system profile; for example, below is a symbol summary of the OProfile daemon. Note that as we used opcontrol --separate=kernel, symbols from images that oprofiled has used are also shown.

    $ opreport -l `which oprofiled` 2>/dev/null | more
    CPU: PIII, speed 863.195 MHz (estimated)
    Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 23150
    vma      samples  %           image name               symbol name
    0804be10 14971    28.1993     oprofiled                odb_insert
    0804afdc 7144     13.4564     oprofiled                pop_buffer_value
    c01daea0 6113     11.5144     vmlinux                  __copy_to_user_ll
    0804b060 2816      5.3042     oprofiled                opd_put_sample
    0804b4a0 2147      4.0441     oprofiled                opd_process_samples
    0804acf4 1855      3.4941     oprofiled                opd_put_image_sample
    0804ad84 1766      3.3264     oprofiled                opd_find_image
    0804a5ec 1084      2.0418     oprofiled                opd_find_module
    0804ba5c 741       1.3957     oprofiled                odb_hash_add_node
    ...

These are the two basic ways you are most likely to use regularly, but opreport can do a lot more than that. For more details, see Section 6.

## 2. Outputting annotated source

The opannotate utility generates annotated source files or assembly listings, optionally mixed with source. If you want to see the source file, the profiled application needs to have debug information, and the source must be available through this debug information. For GCC, you must use the `-g` option when you are compiling. If the binary doesn't contain sufficient debug information, you can still use opannotate `--assembly` to get annotated assembly.

Note that for the reason explained in Section 3.1 the results can be inaccurate. The debug information itself can add other problems; for example, the line number for a symbol can be incorrect. Assembly instructions can be re-ordered and moved by the compiler, and this can lead to crediting source lines with samples not really "owned" by this line. Also see Chapter 5.

You can output the annotation to one single file, containing all the source found using the `--source`. You can use this in conjunction with `--assembly` to get combined source/assembly output.

You can also output a directory of annotated source files that maintains the structure of the original sources. Each line in the annotated source is prepended with the samples for that line. Additionally, each symbol is annotated giving details for the symbol as a whole. An example:

    $ opannotate --source --output-dir=annotated /usr/local/oprofile-pp/bin/oprofiled
    $ ls annotated/home/moz/src/oprofile-pp/daemon/
    opd_cookie.h  opd_image.c  opd_kernel.c  opd_sample_files.c  oprofiled.c

Line numbers are maintained in the source files, but each file has a footer appended describing the profiling details. The actual annotation looks something like this :

    ...
                   :static uint64_t pop_buffer_value(struct transient * trans)
     11510  1.9661 :{ /* pop_buffer_value total:  89901 15.3566 */
                   :        uint64_t val;
                   :
     10227  1.7469 :        if (!trans->remaining) {
                   :                fprintf(stderr, "BUG: popping empty buffer !\n");
                   :                exit(EXIT_FAILURE);
                   :        }
                   :
                   :        val = get_buffer_value(trans->buffer, 0);
      2281  0.3896 :        trans->remaining--;
      2296  0.3922 :        trans->buffer += kernel_pointer_size;
                   :        return val;
     10454  1.7857 :}
    ...

The first number on each line is the number of samples, whilst the second is the relative percentage of total samples. There are a number of options for specifying the output; for more details, see Section 7.

### 2.1. Locating source files

Of course, opannotate needs to be able to locate the source files for the binary image(s) in order to produce output. Some binary images have debug information where the given source file paths are relative, not absolute. You can specify search paths to look for these files (similar to gdb's `dir` command) with the `--search-dirs` option.

Sometimes you may have a binary image which gives absolute paths for the source files, but you have the actual sources elsewhere (commonly, you've installed an SRPM for a binary on your system and you want annotation from an existing profile). You can use the `--base-dirs` option to redirect OProfile to look somewhere else for source files. For example, imagine we have a binary generated from a source file that is given in the debug information as `/tmp/build/libfoo/foo.c`, and you have the source tree matching that binary installed in `/home/user/libfoo/`. You can redirect OProfile to find `foo.c` correctly like this :

    $ opannotate --source --base-dirs=/tmp/build/libfoo/ --search-dirs=/home/user/libfoo/ --output-dir=annotated/ /lib/libfoo.so

You can specify multiple (comma-separated) paths to both options.

## 3. Outputting call-graph profile

The opstack utility generates call-graph profile at symbol level. It's able to traverse shared library boundaries, so you can trace calls into an application's loaded libraries. You can also get kernel-based call-graph profiles; currently OProfile cannot trace across a system call boundary. For example, consider the following C program:

    void a() { for (int i = 0; i < 1000000; ++i) ; }

    void b() { a(); for (int i = 0; i < 1000000; ++i) ; }

    int main() { a(); b(); }

Here we can see the logical structure is that `a()` is called twice, once from `main()`, and once from `b()`. Let's look at a portion of the output from opstack :

    $ opstack ./cgtest
      self     %        child    %        image name               symbol name
    ...
    -------------------------------------------------------------------------------
      0              0  2053     100.000  cgtest                   main
    408      19.4842  715      34.1452  cgtest                   b
      1645     100.000  0              0  cgtest                   a
    -------------------------------------------------------------------------------
    ...

The output is similar to the output of GNU gprof. Each section refers to one function; here we have shown only one section for clarity, which focuses on the function `b()`. We say that (for example) `main()` is a caller of `b()`, and conversely `b()` is a callee of `main()`. Functions listed above the non-indented line in each section are callers of the function; functions listed below the non-indented line are direct callees. Note that functions are only listed if samples were attributed against them in the call-graph.

Let's go through this section line by line.

      0              0  2053     100.000  cgtest                   main

The function `main()` is a caller of `b()`. No samples were taken inside main itself, but all samples (2053) were taken by functions called by `main()`. Note this number refers to all callees of `main()`, not just the one for this section.

    408      19.4842  715      34.1452  cgtest                   b

This is the function that's the focus of this section (it is not indented). We can see that there were 408 samples inside `b()`. The percentage figure refers to the relative percentage of sample count for the entire program: here, we spent 19% of our time in `b()` itself. Additionally, there were 715 samples inside functions that `b()` called. In this case, there is only one such function - `a()`. The percentage has the same meaning - of all the samples taken in the program, 34% of them were spent in `a()` when it was called by `b()`.

      1645     100.000  0              0  cgtest                   a

And here we have `a()`, which is indented and below `b()`, meaning that `b()` called `a()`, as you can see in the source code above. The report shows that `a()` received 1645 samples in total (whether called by `b()` or not). The percentage shows that of all the callees of `b()`, 100% of the samples were in `a()`. This is to be expected, since `b()` only calls one function.

See Section 8.

If you would like to use call-graph profiling, you need to be running on an x86 machine with 2.6 kernel. You must also apply a kernel patch to generate the data.

## 4. gprof-compatible output

If you're familiar with the output produced by GNU gprof, you may find opgprof useful. It takes a single binary as an argument, and produces a `gmon.out` file for use with gprof -p. Note that only a flat profile is included; OProfile cannot produce call graphs. For example:

    $ opgprof `which oprofiled` # generates gmon.out file
    $ gprof -p `which oprofiled` | head
    Flat profile:

    Each sample counts as 1 samples.
      %   cumulative   self              self     total
     time   samples   samples    calls  T1/call  T1/call  name
     33.13 206237.00 206237.00                             odb_insert
     22.67 347386.00 141149.00                             pop_buffer_value
      9.56 406881.00 59495.00                             opd_put_sample
      7.34 452599.00 45718.00                             opd_find_image
      7.19 497327.00 44728.00                             opd_process_samples

A few options to opgprof are available, see Section 9.

## 5. Profile specifications

All of the analysis tools take a profile specification. This is a set of definitions that describe which actual profiles should be examined. The simplest profile specification is empty: this will match all the available profile files for the current session (this is what happens when you do opreport).

Specification parameters are of the form `name:value[,value]`. For example, if I wanted to get a combined symbol summary for `/bin/myprog` and `/bin/myprog2`, I could do opreport -l image:/bin/myprog,/bin/myprog2. As a special case, you don't actually need to specify the `image:` part here: anything left on the command line is assumed to be an `image:` name. Similarly, if no `session:` is specified, then `session:current` is assumed ("current" is a special name of the current / last profiling session).

In addition to the comma-separated list shown above, some of the specification parameters can take glob-style values. For example, if I want to see image summaries for all binaries profiled in `/usr/bin/`, I could do opreport image:/usr/bin/\*. Note the necessity to escape the special character from the shell.

### 5.1. Examples

Image summaries for all profiles with `DATA_MEM_REFS` samples in the saved session called "stresstest" :

    # opreport session:stresstest event:DATA_MEM_REFS

Symbol summary for the application called "test_sym53c8xx,9xx". Note the escaping is necessary as `image:` takes a comma-separated list.

    # opreport -l ./test/test_sym53c8xx\,9xx

Get a symbol summary if you have the actual profile file and binary (perhaps somebody has sent you them via e-mail). These parameters cannot be used in combination with any others as the default searches are not performed.

    # opreport -l binary:./mybinary sample-file:./CPU_CLK_UNHALTED.231500.0.all.all.all

Image summaries for all binaries in the `test` directory, excepting `boring-test` :

    # opreport image:./test/\* image-exclude:./test/boring-test

### 5.2. Profile specification parameters

`session:` sessionlist

  A comma-separated list of session names to resolve in. Absence of this tag, unlike all others, means "the current session", equivalent to specifying "session:current".

`session-exclude:` sessionlist

  A comma-separated list of sessions to exclude.

`image:` imagelist

  A comma-separated list of image names to resolve. Each entry may be relative path, glob-style name, or full path, e.g.

      opreport 'image:/usr/bin/oprofiled,*op*,./opreport'

`image-exclude:` imagelist

  Same as `image:`, but the matching images are excluded.

`lib-image:` imagelist

  Same as `image:`, but only for images that are for a particular primary binary image (namely, an application). This only makes sense to use if you're using `--separate`. This includes kernel modules and the kernel when using `--separate=kernel`.

`lib-image-exclude:` imagelist

  Same as `lib-image:`, but the matching images are excluded.

`event:` eventlist

  The symbolic event name to match on, e.g. `event:DATA_MEM_REFS`. You can pass a list of events for side-by-side comparison with opreport. When using the timer interrupt, the event is always "TIMER".

`count:` eventcountlist

  The event count to match on, e.g. `event:DATA_MEM_REFS count:30000`. Note that this value refers the setting used for opcontrol only, and has nothing to do with the sample counts in the profile data itself. You can pass a list of events for side-by-side comparison with opreport. When using the timer interrupt, the count is always 0 (indicating it cannot be set).

`unit-mask:` masklist

  The unit mask value of the event to match on, e.g. `unit-mask:1`. You can pass a list of events for side-by-side comparison with opreport.

`binary:` file

  Give results only for the given file. This can only be used with `sample-file:`.

`sample-file:` file

  Give results only for the given sample file. This can only be used with `binary:`.

`cpu:` cpulist

  Only consider profiles for the given numbered CPU (starting from zero). This is only useful when using CPU profile separation.

`tgid:` pidlist

  Only consider profiles for the given task groups. Unless some program is using threads, the task group ID of a process is the same as its process ID. This option corresponds to the POSIX notion of a thread group. This is only useful when using per-process profile separation.

`tid:` tidlist

  Only consider profiles for the given threads. When using recent thread libraries, all threads in a process share the same task group ID, but have different thread IDs. You can use this option in combination with `tgid:` to restrict the results to particular threads within a process. This is only useful when using per-process profile separation.

### 5.3. Locating and managing binary images

Each session's sample files can be found in the `/var/lib/oprofile/samples/` directory. These are used, along with the binary image files, to produce human-readable data. In some circumstances (kernel modules in an initrd, or modules on 2.6 kernels), OProfile will not be able to find the binary images. All the tools have an `--image-path` to which you can pass a comma-separated list of alternate paths to search. For example, I can let OProfile find my 2.6 modules by using --image-path /lib/modules/2.6.0/kernel/. It is your responsibility to ensure that the correct images are found when using this option.

Note that if a binary image changes after the sample file was created, you won't be able to get useful symbol-based data out. This situation is detected for you. If you replace a binary, you should make sure to save the old binary if you need to do comparative profiles.

### 5.4. What to do when you don't get any results

When attempting to get output, you may see the error :

    error: no sample files found: profile specification too strict ?

What this is saying is that the profile specification you passed in, when matched against the available sample files, resulted in no matches. There are a number of reasons this might happen:

spelling

  You specified a binary name, but spelt it wrongly. Check your spelling !

profiler wasn't running

  Make very sure that OProfile was actually up and running when you ran the binary.

binary didn't run long enough

  Remember OProfile is a statistical profiler - you're not guaranteed to get samples for short-running programs. You can help this by using a lower count for the performance counter, so there are a lot more samples taken per second.

binary spent most of its time in libraries

  Similarly, if the binary spends little time in the main binary image itself, with most of it spent in shared libraries it uses, you might not see any samples for the binary image itself. You can check this by using opcontrol --separate=library before the profiling session, so opreport and friends show the library profiles on a per-application basis.

specification was really too strict

  For example, you specified something like `tgid:3433`, but no task with that group ID ever ran the code.

binary didn't generate any events

  If you're using a particular event counter, for example counting MMX operations, the code might simply have not generated any events in the first place. Verify the code you're profiling does what you expect it to.

you didn't specify kernel module name correctly

  If you're using 2.6 kernels, and trying to get reports for a kernel module, make sure to use the `-p` option, and specify the module name with the `.ko` extension. Check if the module is one loaded from initrd.

## 6. Usage of opreport

### 6.1. Merging separate profiles

If you have used one of the `--separate=` options whilst profiling, there can be several separate profiles for a single binary image within a session. Normally the output will keep these images separated (so, for example, the image summary output shows library image summaries on a per-application basis, when using `--separate=lib`). Sometimes it can be useful to merge these results back together before getting results. The `--merge` option allows you to do that.

### 6.2. Side-by-side multiple results

If you have used multiple events when profiling, by default you get side-by-side results of each event's sample values from opreport. You can restrict which events to list by appropriate use of the `event:` profile specifications, etc.

### 6.3. Options for opreport

`--accumulated / -c`

  Accumulate sample and percentage counts in the symbol list.

`--debug-info / -g`

  Show source file and line for each symbol.

`--demangle none|normal|smart`

  none: no demangling. normal: use default demangler (default) smart: use pattern-matching to make C++ symbol demangling more readable.

`--details / -d`

  Show per-instruction details for all selected symbols.

`--exclude-dependent / -x`

  Do not include application-specific images for libraries, kernel modules and the kernel. This option only makes sense if the profile session used --separate.

`--exclude-symbols / -e [symbols]`

  Exclude all the symbols in the given comma-separated list.

`--global-percent`

  Make all percentages relative to the whole profile.

`--help / -? / --usage`

  Show help message.

`--image-path / -p [paths]`

  Comma-separated list of additional paths to search for binaries. This is needed to find modules in kernels 2.6 and upwards.

`--include-symbols / -i [symbols]`

  Only include symbols in the given comma-separated list.

`--long-filenames / -l`

  Output full paths instead of basenames.

`--merge / -m [lib,cpu,tid,tgid,unitmask,all]`

  Merge any profiles separated in a --separate session.

`--no-header`

  Don't output a header detailing profiling parameters.

`--output-file / -o [file]`

  Output to the given file instead of stdout.

`--reverse-sort / -r`

  Reverse the sort from the default.

`--show-address / -w`

  Show the VMA address of each symbol (off by default).

`--sort / -s [vma,sample,symbol,debug,image]`

  Sort the list of symbols by, respectively, symbol address, number of samples, symbol name, debug filename and line number, binary image filename.

`--symbols / -l`

  List per-symbol information instead of a binary image summary.

`--threshold / -t [percentage]`

  Only output data for symbols that have more than the given percentage of total samples.

`--verbose / -V [options]`

  Give verbose debugging output.

`--version / -v`

  Show version.

## 7. Usage of opannotate

`--assembly / -a`

  Output annotated assembly. If this is combined with --source, then mixed source / assembly annotations are output.

`--demangle none|normal|smart`

  none: no demangling. normal: use default demangler (default) smart: use pattern-matching to make C++ symbol demangling more readable.

`--exclude-dependent / -x`

  Do not include application-specific images for libraries, kernel modules and the kernel. This option only makes sense if the profile session used --separate.

`--exclude-file [files]`

  Exclude all files in the given comma-separated list of glob patterns.

`--exclude-symbols / -e [symbols]`

  Exclude all the symbols in the given comma-separated list.

`--help / -? / --usage`

  Show help message.

`--image-path / -p [paths]`

  Comma-separated list of additional paths to search for binaries. This is needed to find modules in kernels 2.6 and upwards.

`--include-file [files]`

  Only include files in the given comma-separated list of glob patterns.

`--include-symbols / -i [symbols]`

  Only include symbols in the given comma-separated list.

`--objdump-params [params]`

  Pass the given parameters as extra values when calling objdump.

`--output-dir / -o [dir]`

  Output directory. This makes opannotate output one annotated file for each source file.

`--search-dirs / -d [paths]`

  Comma-separated list of paths to search for source files. This is useful to find source files when the debug information only contains relative paths.

`--base-dirs / -b [paths]/`

  Comma-separated list of path prefixes. This can be used to point OProfile to a different location for source files when the debug information specifies an absolute path on your system for the source that does not exist. The prefix is stripped from the debug source file paths, then searched in the search dirs specified by `--search-dirs`.

`--source / -s`

  Output annotated source. This requires debugging information to be available for the binaries.

`--threshold / -t [percentage]`

  Only output data for symbols that have more than the given percentage of total samples.

`--verbose / -V [options]`

  Give verbose debugging output.

`--version / -v`

  Show version.

## 8. Usage of opstack

`--demangle none|normal|smart`

  none: no demangling. normal: use default demangler (default) smart: use pattern-matching to make C++ symbol demangling more readable.

`--accumulated / -c`

  Accumulate sample and percentage counts in the symbol list.

`--exclude-dependent / -x`

  Do not include application-specific images for libraries, kernel modules and the kernel. This option only makes sense if the profile session used --separate.

`--image-path / -p [paths]`

  Comma-separated list of additional paths to search for binaries. This is needed to find modules in kernels 2.6 and upwards.

`--long-filenames / -l`

  Output full paths instead of basenames.

`--debug-info / -g`

  Show source file and line for each symbol.

`--threshold / -t [percentage]`

  Only output data for symbols that have more than the given percentage of total samples.

`--no-header`

  Don't output a header detailing profiling parameters.

`--show-address / -w`

  Show the VMA address of each symbol (off by default).

`--verbose / -V [options]`

  Give verbose debugging output.

`--version / -v`

  Show version.

`--help / -? / --usage`

  Show help message.

## 9. Usage of opgprof

`--help / -? / --usage`

  Show help message.

`--version / -v`

  Show version.

`--verbose / -V [options]`

  Give verbose debugging output.

`--image-path / -p [paths]`

  Comma-separated list of additional paths to search for binaries. This is needed to find modules in kernels 2.6 and upwards.

`--threshold / -t [percentage]`

  Only output data for symbols that have more than the given percentage of total samples.

`--output-filename / -o [file]`

  Output to the given file instead of the default, gmon.out

## Chapter 5. Interpreting profiling results

The standard caveats of profiling apply in interpreting the results from OProfile: profile realistic situations, profile different scenarios, profile for as long as a time as possible, avoid system-specific artifacts, don't trust the profile data too much. Also bear in mind the comments on the performance counters above - you cannot rely on totally accurate instruction-level profiling. However, for almost all circumstances the data can be useful. Ideally a utility such as Intel's VTUNE would be available to allow careful instruction-level analysis; go hassle Intel for this, not me ;)

## 1. Profiling interrupt latency

This is an example of how the latency of delivery of profiling interrupts can impact the reliability of the profiling data. This is pretty much a worst-case-scenario example: these problems are fairly rare.

    double fun(double a, double b, double c)
    {
     double result = 0;
     for (int i = 0 ; i < 10000; ++i) {
      result += a;
      result *= b;
      result /= c;
     }
     return result;
    }

Here the last instruction of the loop is very costly, and you would expect the result reflecting that - but (cutting the instructions inside the loop):

    $ opannotate -a -t 10 ./a.out

         88 15.38% : 8048337:       fadd   %st(3),%st
         48 8.391% : 8048339:       fmul   %st(2),%st
         68 11.88% : 804833b:       fdiv   %st(1),%st
        368 64.33% : 804833d:       inc    %eax
                   : 804833e:       cmp    $0x270f,%eax
                   : 8048343:       jle    8048337

The problem comes from the x86 hardware; when the counter overflows the IRQ is asserted but the hardware has features that can delay the NMI interrupt: x86 hardware is synchronous (i.e. cannot interrupt during an instruction); there is also a latency when the IRQ is asserted, and the multiple execution units and the out-of-order model of modern x86 CPUs also causes problems. This is the same function, with annotation :

    $ opannotate -s -t 10 ./a.out

                   :double fun(double a, double b, double c)
                   :{ /* _Z3funddd total:     572 100.0% */
                   : double result = 0;
        368 64.33% : for (int i = 0 ; i < 10000; ++i) {
         88 15.38% :  result += a;
         48 8.391% :  result *= b;
         68 11.88% :  result /= c;
                   : }
                   : return result;
                   :}

The conclusion: don't trust samples coming at the end of a loop, particularly if the last instruction generated by the compiler is costly. This case can also occur for branches. Always bear in mind that samples can be delayed by a few cycles from its real position. That's a hardware problem and OProfile can do nothing about it.

## 2. Kernel profiling

### 2.1. Interrupt masking

OProfile uses non-maskable interrupts (NMI) on the P6 generation, Pentium 4, Athlon and Duron processors. These interrupts can occur even in section of the Linux where interrupts are disabled, allowing collection of samples in virtually all executable code. The RTC, timer interrupt mode, and Itanium 2 collection mechanisms use maskable interrupts. Thus, the RTC and Itanium 2 data collection mechanism have "sample shadows", or blind spots: regions where no samples will be collected. Typically, the samples will be attributed to the code immediately after the interrupts are re-enabled.

### 2.2. Idle time

Your kernel is likely to support halting the processor when a CPU is idle. As the typical hardware events like `CPU_CLK_UNHALTED` do not count when the CPU is halted, the kernel profile will not reflect the actual amount of time spent idle. You can change this behaviour by booting with the `idle=poll` option, which uses a different idle routine. This will appear as `poll_idle()` in your kernel profile.

### 2.3. Exiting tasks

The internal implementation of the 2.6 OProfile code means that tasks that within the kernel `do_exit()` routine cannot be profiled.

### 2.4. Profiling kernel modules

OProfile profiles kernel modules by default. However, there are a couple of problems you may have when trying to get results. First, you may have booted via an initrd; this means that the actual path for the module binaries cannot be determined automatically. To get around this, you can use the `-p` option to the profiling tools to specify where to look for the kernel modules.

In 2.6, the information on where kernel module binaries are located has been removed. This means OProfile needs guiding with the `-p` option to find your modules. Normally, you can just use your standard module top-level directory for this. Note that due to this problem, OProfile cannot check that the modification times match; it is your responsibility to make sure you do not modify a binary after a profile has been created.

If you have run insmod or modprobe to insert a module in a particular directory, it is important that you specify this directory with the `-p` option first, so that it over-rides an older module binary that might exist in other directories you've specified with `-p`. It is up to you to make sure that these values are correct: 2.6 kernels simply do not provide enough information for OProfile to get this information.

## 3. Interpreting call-graph profiles

Sometimes the results from call-graph profiles may be different to what you expect to see. The first thing to check is whether the target binaries where compiled with frame pointers enabled (if the binary was compiled using gcc's `-fomit-frame-pointer` option, you will not get meaningful results). Note that as of this writing, the GCC developers plan to disable frame pointers by default. The Linux kernel is built without frame pointers by default; there is a configuration option you can use to turn it on under the "Kernel Hacking" menu.

Like the rest of OProfile, call-graph profiling uses a statistical approach; this means that sometimes a backtrace sample is truncated, or even partially wrong. Bear this in mind when examining results.

## 4. Inaccuracies in annotated source

### 4.1. Side effects of optimizations

The compiler can introduce some pitfalls in the annotated source output. The optimizer can move pieces of code in such manner that two line of codes are interlaced (instruction scheduling). Also debug info generated by the compiler can show strange behavior. This is especially true for complex expressions e.g. inside an if statement:

    	if (a && ..
    	    b && ..
    	    c &&)

here the problem come from the position of line number. The available debug info does not give enough details for the if condition, so all samples are accumulated at the position of the right brace of the expression. Using opannotate `-a` can help to show the real samples at an assembly level.

### 4.2. Prologues and epilogues

The compiler generally needs to generate "glue" code across function calls, dependent on the particular function call conventions used. Additionally other things need to happen, like stack pointer adjustment for the local variables; this code is known as the function prologue. Similar code is needed at function return, and is known as the function epilogue. This will show up in annotations as samples at the very start and end of a function, where there is no apparent executable code in the source.

### 4.3. Inlined functions

You may see that a function is credited with a certain number of samples, but the listing does not add up to the correct total. To pick a real example :

                   :internal_sk_buff_alloc_security(struct sk_buff *skb)
     353 2.342%    :{ /* internal_sk_buff_alloc_security total: 1882 12.48% */
                   :
                   :        sk_buff_security_t *sksec;
      15 0.0995%   :        int rc = 0;
                   :
      10 0.06633%  :        sksec = skb->lsm_security;
     468 3.104%    :        if (sksec && sksec->magic == DSI_MAGIC) {
                   :                goto out;
                   :        }
                   :
                   :        sksec = (sk_buff_security_t *) get_sk_buff_memory(skb);
       3 0.0199%   :        if (!sksec) {
      38 0.2521%   :                rc = -ENOMEM;
                   :                goto out;
      10 0.06633%  :        }
                   :        memset(sksec, 0, sizeof (sk_buff_security_t));
      44 0.2919%   :        sksec->magic = DSI_MAGIC;
      32 0.2123%   :        sksec->skb = skb;
      45 0.2985%   :        sksec->sid = DSI_SID_NORMAL;
      31 0.2056%   :        skb->lsm_security = sksec;
                   :
                   :      out:
                   :
     146 0.9685%   :        return rc;
                   :
      98 0.6501%   :}

Here, the function is credited with 1,882 samples, but the annotations below do not account for this. This is usually because of inline functions - the compiler marks such code with debug entries for the inline function definition, and this is where opannotate annotates such samples. In the case above, `memset` is the most likely candidate for this problem. Examining the mixed source/assembly output can help identify such results.

When running opannotate, you may get a warning "some functions compiled without debug information may have incorrect source line attributions". In some rare cases, OProfile is not able to verify that the derived source line is correct (when some parts of the binary image are compiled without debugging information). Be wary of results if this warning appears.

Furthermore, for some languages the compiler can implicitly generate functions, such as default copy constructors. Such functions are labelled by the compiler as having a line number of 0, which means the source annotation can be confusing.

### 4.4. Inaccuracy in line number information

Depending on your compiler you can fall into the following problem:

    struct big_object { int a[500]; };

    int main()
    {
    	big_object a, b;
    	for (int i = 0 ; i != 1000 * 1000; ++i)
    		b = a;
    	return 0;
    }

Compiled with gcc 3.0.4 the annotated source is clearly inaccurate:

                   :int main()
                   :{  /* main total: 7871 100% */
                   :        big_object a, b;
                   :        for (int i = 0 ; i != 1000 * 1000; ++i)
                   :                b = a;
     7871 100%     :        return 0;
                   :}

The problem here is distinct from the IRQ latency problem; the debug line number information is not precise enough; again, looking at output of opannoatate -as can help.

                   :int main()
                   :{
                   :        big_object a, b;
                   :        for (int i = 0 ; i != 1000 * 1000; ++i)
                   : 80484c0:       push   %ebp
                   : 80484c1:       mov    %esp,%ebp
                   : 80484c3:       sub    $0xfac,%esp
                   : 80484c9:       push   %edi
                   : 80484ca:       push   %esi
                   : 80484cb:       push   %ebx
                   :                b = a;
                   : 80484cc:       lea    0xfffff060(%ebp),%edx
                   : 80484d2:       lea    0xfffff830(%ebp),%eax
                   : 80484d8:       mov    $0xf423f,%ebx
                   : 80484dd:       lea    0x0(%esi),%esi
                   :        return 0;
        3 0.03811% : 80484e0:       mov    %edx,%edi
                   : 80484e2:       mov    %eax,%esi
        1 0.0127%  : 80484e4:       cld
        8 0.1016%  : 80484e5:       mov    $0x1f4,%ecx
     7850 99.73%   : 80484ea:       repz movsl %ds:(%esi),%es:(%edi)
        9 0.1143%  : 80484ec:       dec    %ebx
                   : 80484ed:       jns    80484e0
                   : 80484ef:       xor    %eax,%eax
                   : 80484f1:       pop    %ebx
                   : 80484f2:       pop    %esi
                   : 80484f3:       pop    %edi
                   : 80484f4:       leave
                   : 80484f5:       ret

So here it's clear that copying is correctly credited with of all the samples, but the line number information is misplaced. objdump -dS exposes the same problem. Note that maintaining accurate debug information for compilers when optimizing is difficult, so this problem is not suprising. The problem of debug information accuracy is also dependent on the binutils version used; some BFD library versions contain a work-around for known problems of gcc, some others do not. This is unfortunate but we must live with that, since profiling is pointless when you disable optimisation (which would give better debugging entries).

## 5. Assembly functions

Often the assembler cannot generate debug information automatically. This means that you cannot get a source report unless you manually define the neccessary debug information; read your assembler documentation for how you might do that. The only debugging info needed currently by OProfile is the line-number/filename-VMA association. When profiling assembly without debugging info you can always get report for symbols, and optionally for VMA, through opreport -l or opreport -d, but this works only for symbol with the right attributes. For gas you can get this by

    .globl foo
    	.type	foo,@function

whilst for nasm you must use

    	  GLOBAL foo:function		; [1]

Note that OProfile does not need the global attribute, only the function attribute.

## 6. Other discrepancies

Another cause of apparent problems is the hidden cost of instructions. A very common example is two memory reads: one from L1 cache and the other from memory: the second memory read is likely to have more samples. There are many other causes of hidden cost of instructions. A non-exhaustive list: mis-predicted branch, TLB cache miss, partial register stall, partial register dependencies, memory mismatch stall, re-executed µops. If you want to write programs at the assembly level, be sure to take a look at the Intel and AMD documentation at http://developer.intel.com/ and http://www.amd.com/products/cpg/athlon/techdocs/.

## Chapter 6. Acknowledgments

Thanks to (in no particular order) : Arjan van de Ven, Rik van Riel, Juan Quintela, Philippe Elie, Phillipp Rumpf, Tigran Aivazian, Alex Brown, Alisdair Rawsthorne, Bob Montgomery, Ray Bryant, H.J. Lu, Jeff Esper, Will Cohen, Graydon Hoare, Cliff Woolley, Alex Tsariounov, Al Stone, Jason Yeh, Randolph Chung, Anton Blanchard, Richard Henderson, Andries Brouwer, Bryan Rittmeyer, Richard Reich (rreich@rdrtech.com), Zwane Mwaikambo, Dave Jones, Charles Filtness; and finally Pulp, for "Intro".