'\" t
.\" Title: perf-sched
.\" Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author]
.\" Generator: DocBook XSL Stylesheets vsnapshot
.\" Date: 09/04/2026
.\" Manual: perf Manual
.\" Source: perf
.\" Language: English
.\"
.TH "PERF\-SCHED" "1" "09/04/2026" "perf" "perf Manual"
.\" -----------------------------------------------------------------
.\" * Define some portability stuff
.\" -----------------------------------------------------------------
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.\" http://bugs.debian.org/507673
.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.ie \n(.g .ds Aq \(aq
.el .ds Aq '
.\" -----------------------------------------------------------------
.\" * set default formatting
.\" -----------------------------------------------------------------
.\" disable hyphenation
.nh
.\" disable justification (adjust text to left margin only)
.ad l
.\" -----------------------------------------------------------------
.\" * MAIN CONTENT STARTS HERE *
.\" -----------------------------------------------------------------
.SH "NAME"
perf-sched \- Tool to trace/measure scheduler properties (latencies)
.SH "SYNOPSIS"
.sp
.nf
\fIperf sched\fR {record|latency|map|replay|script|timehist|stats}
.fi
.SH "DESCRIPTION"
.sp
There are several variants of \fIperf sched\fR:
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched record \*(Aq to record the scheduling events
of an arbitrary workload\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched latency\*(Aq to report the per task scheduling latencies
and other scheduling properties of the workload\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Example usage:
perf sched record \-\- sleep 1
perf sched latency
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Task | Runtime ms | Count | Avg delay ms | Max delay ms | Max delay start | Max delay end |
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
perf:(2) | 2\&.804 ms | 66 | avg: 0\&.524 ms | max: 1\&.069 ms | max start: 254752\&.314960 s | max end: 254752\&.316029 s
NetworkManager:1343 | 0\&.372 ms | 13 | avg: 0\&.008 ms | max: 0\&.013 ms | max start: 254751\&.551153 s | max end: 254751\&.551166 s
kworker/1:2\-xfs:4649 | 0\&.012 ms | 1 | avg: 0\&.008 ms | max: 0\&.008 ms | max start: 254751\&.519807 s | max end: 254751\&.519815 s
kworker/3:1\-xfs:388 | 0\&.011 ms | 1 | avg: 0\&.006 ms | max: 0\&.006 ms | max start: 254751\&.519809 s | max end: 254751\&.519815 s
sleep:147736 | 0\&.938 ms | 3 | avg: 0\&.006 ms | max: 0\&.007 ms | max start: 254751\&.313817 s | max end: 254751\&.313824 s
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
It shows Runtime(time that a task spent actually running on the CPU),
Count(number of times a delay was calculated) and delay(time that a
task was ready to run but was kept waiting)\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Tasks with the same command name are merged and the merge count is
given within (), However if \-p option is used, pid is mentioned\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched script\*(Aq to see a detailed trace of the workload that
was recorded (aliased to \*(Aqperf script\*(Aq for now)\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched replay\*(Aq to simulate the workload that was recorded
via perf sched record\&. (this is done by starting up mockup threads
that mimic the workload based on the events in the trace\&. These
threads can then replay the timings (CPU runtime and sleep patterns)
of the workload as it occurred when it was recorded \- and can repeat
it a number of times, measuring its performance\&.)
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched map\*(Aq to print a textual context\-switching outline of
workload captured via perf sched record\&. Columns stand for
individual CPUs, and the two\-letter shortcuts stand for tasks that
are running on a CPU\&. A \*(Aq*\*(Aq denotes the CPU that had the event, and
a dot signals an idle CPU\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched timehist\*(Aq provides an analysis of scheduling events\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Example usage:
perf sched record \-\- sleep 1
perf sched timehist
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
By default it shows the individual schedule events, including the wait
time (time between sched\-out and next sched\-in events for the task), the
task scheduling delay (time between runnable and actually running) and
run time for the task:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
time cpu task name wait time sch delay run time
[tid/pid] (msec) (msec) (msec)
\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-
79371\&.874569 [0011] gcc[31949] 0\&.014 0\&.000 1\&.148
79371\&.874591 [0010] gcc[31951] 0\&.000 0\&.000 0\&.024
79371\&.874603 [0010] migration/10[59] 3\&.350 0\&.004 0\&.011
79371\&.874604 [0011] 1\&.148 0\&.000 0\&.035
79371\&.874723 [0005] 0\&.016 0\&.000 1\&.383
79371\&.874746 [0005] gcc[31949] 0\&.153 0\&.078 0\&.022
\&.\&.\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Times are in msec\&.usec\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\*(Aqperf sched stats {record | report | diff} \*(Aq to capture, report the diff
in schedstat counters and show the difference between perf sched stats report
respectively\&. schedstat counters which are present in the linux kernel and are
exposed through the file ``/proc/schedstat``\&. These counters are enabled or disabled
via the sysctl governed by the file ``/proc/sys/kernel/sched_schedstats``\&. These
counters accounts for many scheduler events such as ``schedule()`` calls, load\-balancing
events, ``try_to_wakeup()`` call among others\&. This is useful in understanding the
scheduler behavior for the workload\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Note: The tool will not give correct results if there is topological reordering or
online/offline of cpus in between capturing snapshots of `/proc/schedstat`\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Example usage:
perf sched stats record \-\- sleep 1
perf sched stats report
perf sched stats diff
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
A detailed description of the schedstats can be found in the Kernel Documentation:
https://www\&.kernel\&.org/doc/html/latest/scheduler/sched\-stats\&.html
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
The result can be interpreted as follows:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
The `perf sched stats report` starts with description of the columns present in
the report\&. These column names are given before cpu and domain stats to improve
the readability of the report\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
DESC \-> Description of the field
COUNT \-> Value of the field
PCT_CHANGE \-> Percent change with corresponding base value
AVG_JIFFIES \-> Avg time in jiffies between two consecutive occurrence of event
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Next is the total profiling time in terms of jiffies:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Time elapsed (in jiffies) : 2323
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Next is CPU scheduling statistics\&. These are simple diffs of /proc/schedstat CPU lines
along with description\&. The report also prints % relative to base stat\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
In the example below, schedule() left the CPU0 idle 36\&.58% of the time\&. 0\&.45% of total
try_to_wake_up() was to wakeup local CPU\&. And, the total waittime by tasks on CPU0 is
48\&.70% of the total runtime by tasks on the same CPU\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
CPU 0
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
DESC COUNT PCT_CHANGE
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
yld_count : 0
array_exp : 0
sched_count : 402267
sched_goidle : 147161 ( 36\&.58% )
ttwu_count : 236309
ttwu_local : 1062 ( 0\&.45% )
rq_cpu_time : 7083791148
run_delay : 3449973971 ( 48\&.70% )
pcount : 255035
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Next is load balancing statistics\&. For each of the sched domains
(eg: `SMT`, `MC`, `DIE`\&.\&.\&.), the scheduler computes statistics under
the following three categories:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
1) Idle Load Balance: Load balancing performed on behalf of a long
idling CPU by some other CPU\&.
2) Busy Load Balance: Load balancing performed when the CPU was busy\&.
3) New Idle Balance : Load balancing performed when a CPU just became
idle\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Under each of these three categories, sched stats report provides
different load balancing statistics\&. Along with direct stats, the
report also contains derived metrics prefixed with *\&. Example:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
CPU 0, DOMAIN SMT CPUS 0,64
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
DESC COUNT AVG_JIFFIES
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
busy_lb_count : 136 $ 17\&.08 $
busy_lb_balanced : 131 $ 17\&.73 $
busy_lb_failed : 0 $ 0\&.00 $
busy_lb_imbalance_load : 58
busy_lb_imbalance_util : 0
busy_lb_imbalance_task : 0
busy_lb_imbalance_misfit : 0
busy_lb_gained : 7
busy_lb_hot_gained : 0
busy_lb_nobusyq : 2 $ 1161\&.50 $
busy_lb_nobusyg : 129 $ 18\&.01 $
*busy_lb_success_count : 5
*busy_lb_avg_pulled : 1\&.40
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
idle_lb_count : 449 $ 5\&.17 $
idle_lb_balanced : 382 $ 6\&.08 $
idle_lb_failed : 3 $ 774\&.33 $
idle_lb_imbalance_load : 0
idle_lb_imbalance_util : 0
idle_lb_imbalance_task : 71
idle_lb_imbalance_misfit : 0
idle_lb_gained : 67
idle_lb_hot_gained : 0
idle_lb_nobusyq : 0 $ 0\&.00 $
idle_lb_nobusyg : 382 $ 6\&.08 $
*idle_lb_success_count : 64
*idle_lb_avg_pulled : 1\&.05
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
newidle_lb_count : 30471 $ 0\&.08 $
newidle_lb_balanced : 28490 $ 0\&.08 $
newidle_lb_failed : 633 $ 3\&.67 $
newidle_lb_imbalance_load : 0
newidle_lb_imbalance_util : 0
newidle_lb_imbalance_task : 2040
newidle_lb_imbalance_misfit : 0
newidle_lb_gained : 1348
newidle_lb_hot_gained : 0
newidle_lb_nobusyq : 6 $ 387\&.17 $
newidle_lb_nobusyg : 26634 $ 0\&.09 $
*newidle_lb_success_count : 1348
*newidle_lb_avg_pulled : 1\&.00
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Consider following line:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
newidle_lb_balanced : 28490 $ 0\&.08 $
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
While profiling was active, the load\-balancer found 28490 times the load
needs to be balanced on a newly idle CPU 0\&. Following value encapsulated
inside $ is average jiffies between two events (2323 / 28490 = 0\&.08)\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Next are active_load_balance() stats\&. alb did not trigger while the
profiling was active, hence it\*(Aqs all 0s\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
alb_count : 0
alb_failed : 0
alb_pushed : 0
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Next are sched_balance_exec() and sched_balance_fork() stats\&. They are
not used but we kept it in RFC just for legacy purpose\&. Unless opposed,
we plan to remove them in next revision\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Next are wakeup statistics\&. For every domain, the report also shows
task\-wakeup statistics\&. Example:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
ttwu_wake_remote : 1590
ttwu_move_affine : 84
ttwu_move_balance : 0
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Same set of stats are reported for each CPU and each domain level\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
How to interpret the diff
~~~~~~~~~~~~~~~~~~~~~~~~~
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
The `perf sched stats diff` will also start with explaining the columns
present in the diff\&. Then it will show the diff in time in terms of
jiffies\&. The order of the values depends on the order of input data
files\&. It will take `perf\&.data\&.old` and `perf\&.data` respectively as the
defaults for comparison\&. Example:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Time elapsed (in jiffies) : 2009, 2001
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Below is the sample representing the difference in cpu and domain stats of
two runs\&. Here third column or the values enclosed in `|\&.\&.\&.|` shows the
percent change between the two\&. Second and fourth columns shows the
side\-by\-side representions of the corresponding fields from `perf sched
stats report`\&.
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
CPU
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
DESC COUNT1 COUNT2 PCT_CHANG>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
yld_count : 0, 0 | 0\&.00>
array_exp : 0, 0 | 0\&.00>
sched_count : 528533, 412573 | \-21\&.94>
sched_goidle : 193426, 146082 | \-24\&.48>
ttwu_count : 313134, 385975 | 23\&.26>
ttwu_local : 1126, 1282 | 13\&.85>
rq_cpu_time : 8257200244, 8301250047 | 0\&.53>
run_delay : 4728347053, 3997100703 | \-15\&.47>
pcount : 335031, 266396 | \-20\&.49>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
Below is the sample of domain stats diff:
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
CPU , DOMAIN SMT
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
DESC COUNT1 COUNT2 PCT_CHANG>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
busy_lb_count : 122, 80 | \-34\&.43>
busy_lb_balanced : 115, 76 | \-33\&.91>
busy_lb_failed : 1, 3 | 200\&.00>
busy_lb_imbalance_load : 35, 49 | 40\&.00>
busy_lb_imbalance_util : 0, 0 | 0\&.00>
busy_lb_imbalance_task : 0, 0 | 0\&.00>
busy_lb_imbalance_misfit : 0, 0 | 0\&.00>
busy_lb_gained : 7, 2 | \-71\&.43>
busy_lb_hot_gained : 0, 0 | 0\&.00>
busy_lb_nobusyq : 0, 0 | 0\&.00>
busy_lb_nobusyg : 115, 76 | \-33\&.91>
*busy_lb_success_count : 6, 1 | \-83\&.33>
*busy_lb_avg_pulled : 1\&.17, 2\&.00 | 71\&.43>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
idle_lb_count : 568, 620 | 9\&.15>
idle_lb_balanced : 462, 449 | \-2\&.81>
idle_lb_failed : 11, 21 | 90\&.91>
idle_lb_imbalance_load : 0, 0 | 0\&.00>
idle_lb_imbalance_util : 0, 0 | 0\&.00>
idle_lb_imbalance_task : 115, 189 | 64\&.35>
idle_lb_imbalance_misfit : 0, 0 | 0\&.00>
idle_lb_gained : 103, 169 | 64\&.08>
idle_lb_hot_gained : 0, 0 | 0\&.00>
idle_lb_nobusyq : 0, 0 | 0\&.00>
idle_lb_nobusyg : 462, 449 | \-2\&.81>
*idle_lb_success_count : 95, 150 | 57\&.89>
*idle_lb_avg_pulled : 1\&.08, 1\&.13 | 3\&.92>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
newidle_lb_count : 16961, 3155 | \-81\&.40>
newidle_lb_balanced : 15646, 2556 | \-83\&.66>
newidle_lb_failed : 397, 142 | \-64\&.23>
newidle_lb_imbalance_load : 0, 0 | 0\&.00>
newidle_lb_imbalance_util : 0, 0 | 0\&.00>
newidle_lb_imbalance_task : 1376, 655 | \-52\&.40>
newidle_lb_imbalance_misfit : 0, 0 | 0\&.00>
newidle_lb_gained : 917, 457 | \-50\&.16>
newidle_lb_hot_gained : 0, 0 | 0\&.00>
newidle_lb_nobusyq : 3, 1 | \-66\&.67>
newidle_lb_nobusyg : 14480, 2103 | \-85\&.48>
*newidle_lb_success_count : 918, 457 | \-50\&.22>
*newidle_lb_avg_pulled : 1\&.00, 1\&.00 | 0\&.11>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
alb_count : 0, 1 | 0\&.00>
alb_failed : 0, 0 | 0\&.00>
alb_pushed : 0, 1 | 0\&.00>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
sbe_count : 0, 0 | 0\&.00>
sbe_balanced : 0, 0 | 0\&.00>
sbe_pushed : 0, 0 | 0\&.00>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
sbf_count : 0, 0 | 0\&.00>
sbf_balanced : 0, 0 | 0\&.00>
sbf_pushed : 0, 0 | 0\&.00>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
ttwu_wake_remote : 2031, 2914 | 43\&.48>
ttwu_move_affine : 73, 124 | 69\&.86>
ttwu_move_balance : 0, 0 | 0\&.00>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.fi
.if n \{\
.RE
.\}
.SH "OPTIONS"
.sp
Applicable to {record|latency|map|replay|script}
.PP
\-i, \-\-input=
.RS 4
Input file name\&. (default: perf\&.data unless stdin is a fifo)
.RE
.PP
\-v, \-\-verbose
.RS 4
Be more verbose\&. (show symbol address, etc)
.RE
.PP
\-D, \-\-dump\-raw\-trace=
.RS 4
Display verbose dump of the sched data\&.
.RE
.PP
\-f, \-\-force
.RS 4
Don\(cqt complain, do it\&.
.RE
.SH "OPTIONS FOR \FIPERF SCHED LATENCY\FR"
.PP
\-C, \-\-CPU
.RS 4
CPU to profile on\&.
.RE
.PP
\-p, \-\-pids
.RS 4
latency stats per pid instead of per command name\&.
.RE
.PP
\-s, \-\-sort
.RS 4
sort by key(s): runtime, switch, avg, max by default it\(cqs sorted by "avg ,max ,switch ,runtime"\&.
.RE
.SH "OPTIONS FOR \FIPERF SCHED MAP\FR"
.PP
\-\-compact
.RS 4
Show only CPUs with activity\&. Helps visualizing on high core count systems\&.
.RE
.PP
\-\-cpus
.RS 4
Show just entries with activities for the given CPUs\&.
.RE
.PP
\-\-color\-cpus
.RS 4
Highlight the given cpus\&.
.RE
.PP
\-\-color\-pids
.RS 4
Highlight the given pids\&.
.RE
.PP
\-\-task\-name
.RS 4
Map output only for the given task name(s)\&. Separate the task names with a comma (without whitespace)\&. The sched\-out time is printed and is represented by
\fI*\-\fR
for the given task name(s)\&. (\fI\-\fR
indicates other tasks while
\fI\&.\fR
is idle)\&.
.RE
.PP
\-\-fuzzy\-name
.RS 4
Given task name(s) can be partially matched (fuzzy matching)\&.
.RE
.SH "OPTIONS FOR \FIPERF SCHED TIMEHIST\FR"
.PP
\-k, \-\-vmlinux=
.RS 4
vmlinux pathname
.RE
.PP
\-\-kallsyms=
.RS 4
kallsyms pathname
.RE
.PP
\-g, \-\-call\-graph
.RS 4
Display call chains if present (default on)\&.
.RE
.PP
\-\-max\-stack
.RS 4
Maximum number of functions to display in backtrace, default 5\&.
.RE
.PP
\-C=, \-\-cpu=
.RS 4
Only show events for the given CPU(s) (comma separated list)\&.
.RE
.PP
\-p=, \-\-pid=
.RS 4
Only show events for given process ID (comma separated list)\&.
.RE
.PP
\-t=, \-\-tid=
.RS 4
Only show events for given thread ID (comma separated list)\&.
.RE
.PP
\-s, \-\-summary
.RS 4
Show only a summary of scheduling by thread with min, max, and average run times (in sec) and relative stddev\&.
.RE
.PP
\-S, \-\-with\-summary
.RS 4
Show all scheduling events followed by a summary by thread with min, max, and average run times (in sec) and relative stddev\&.
.RE
.PP
\-\-symfs=
.RS 4
Look for files with symbols relative to this directory\&. The optional layout can be
\fIhierarchy\fR
(default, matches full path) or
\fIflat\fR
(only matches base name)\&. This is useful when debug files are stored in a flat directory structure\&.
.RE
.PP
\-V, \-\-cpu\-visual
.RS 4
Show visual aid for sched switches by CPU:
\fIi\fR
marks idle time,
\fIs\fR
are scheduler events\&.
.RE
.PP
\-w, \-\-wakeups
.RS 4
Show wakeup events\&.
.RE
.PP
\-M, \-\-migrations
.RS 4
Show migration events\&.
.RE
.PP
\-n, \-\-next
.RS 4
Show next task\&.
.RE
.PP
\-I, \-\-idle\-hist
.RS 4
Show idle\-related events only\&.
.RE
.PP
\-\-time
.RS 4
Only analyze samples within given time window: ,\&. Times have the format seconds\&.microseconds\&. If start is not given (i\&.e\&., time string is
\fI,x\&.y\fR) then analysis starts at the beginning of the file\&. If stop time is not given (i\&.e, time string is
\fIx\&.y,\fR) then analysis goes to end of file\&.
.RE
.PP
\-\-state
.RS 4
Show task state when it switched out\&.
.RE
.PP
\-\-show\-prio
.RS 4
Show task priority\&.
.RE
.PP
\-\-prio
.RS 4
Only show events for given task priority(ies)\&. Multiple priorities can be provided as a comma\-separated list with no spaces: 0,120\&. Ranges of priorities are specified with \-: 120\-129\&. A combination of both can also be provided: 0,120\-129\&.
.RE
.PP
\-P, \-\-pre\-migrations
.RS 4
Show pre\-migration wait time\&. pre\-migration wait time is the time spent by a task waiting on a runqueue but not getting the chance to run there and is migrated to a different runqueue where it is finally run\&. This time between sched_wakeup and migrate_task is the pre\-migration wait time\&.
.RE
.SH "OPTIONS FOR \FIPERF SCHED REPLAY\FR"
.PP
\-r, \-\-repeat
.RS 4
repeat the workload n times (0: infinite)\&. Default is 10\&.
.RE
.SH "SEE ALSO"
.sp
\fBperf-record\fR(1)