TRACE-CMD-RECORD(1) libtracefs Manual TRACE-CMD-RECORD(1) (NAME) trace-cmd-record - Ftrace (SYNOPSIS) trace-cmd record [OPTIONS] [command] (DESCRIPTION) trace-cmd(1) record Ftrace command . Ctrl-C . record trace-cmd Ftrace . ( CPU) . ( Ctrl-C) trace.dat ( trace-cmd-report(1) ). (OPTIONS) -p tracer (tracer) . . : function function_graph preemptirqsoff irqsoff preemptoff wakeup. . trace-cmd-list(1) . -e event . (static trace points) . . event "subsystem:event-name" . :event-name "subsystem:" . <<-e sched_switch>> "sched_switch" <<-e sched>> "sched" . 'event' (glob expressions) "*stat*" ( ) "stat" . 'all' . -a . trace-cmd trace-cmd report . -a . -T (stacktrace) . : -0 [003] 58549.289091: sched_switch: kworker/0:1:0 [120] R ==> trace-cmd:2603 [120] -0 [003] 58549.289092: kernel_stack: => schedule (ffffffff814b260e) => cpu_idle (ffffffff8100a38c) => start_secondary (ffffffff814ab828) --func-stack . "function" -l . (live lock) . -f filter . -e . . . C . ==, >=, <=, >, <, &, |, && || . --no-filter (threads) trace-cmd . . trace-cmd . -R trigger (trigger) . -e . . -v . Documentation/trace/events.txt . -v . . : <<-e sched -v -e "*stat\*">> sched "stat" . : *-v* grep(1) . -F . ( ). -F . -P pid -F (pid) . -c -F ( -P ) . --user command . -C clock "clock". trace-cmd(1) list -C . -o output-file trace-cmd report trace.dat . -o . -l function-name function function_graph . -l . regex(3) glob . - _ * ? . glob . regex ^ $ regex . -g function-name function_graph . (graph) . -g . -n function-name -l . -n . -n -l . -d (latency tracers) . . -D -d function-trace ( ) proc /proc/sys/kernel/ftrace_enabled function-trace . -D proc ftrace_enabled function-trace . ftrace ( stack_tracer perf ) . -O option Ftrace . . no : <<-O nograph-time>> <> Ftrace . -s interval trace-cmd . interval . Ftrace . . 'interval' . 1000 ( ) . . -r priority . . (FIFO) . . -b size size . Ftrace CPU CPU . <<-b 10000>> (CPU) Ftrace . --subbuf-size (sub-buffer) . <> ( x86). . . ( ) . <<>> (read) . . ! -B buffer-name . record . ( -k start )..sp . ( toplevel) . trace-cmd record -e sched -B block -e block -B time -e timer sleep 1 sched . 'block' block . 'time' timer . -m size CPU . (page size) . . . -M cpumask cpumask . . . (hex) . trace-cmd record -p function -M c -B events13 -e all -M 5 -M . CPU '-1' . -k trace-cmd . trace-cmd . trace-cmd . : trace-cmd "tracing_on" . . -K trace-cmd . trace-cmd . trace-cmd set . -i trace-cmd . . -N host:port "trace-cmd listen" UDP . . . : (latency tracer) : wakeup, wakeup_rt, irqsoff, preemptoff and preemptirqsoff -V cid:port (guest VM) trace-cmd listen -V trace-cmd listen -V ( -N ). -N . -t -N UDP TCP . TCP UDP . -q | --quiet . ( ) . --date --date "trace-cmd" (timestamps) . gettimeofday (wall time) trace.dat . --max-graph-depth depth function_graph . . . --cmdlines-size size "saved_cmdlines" . (cmdlines) (PIDs) . ("<...>") . . --module module . :mod:module . . '--module snd' '-l :mod:snd' '--module snd -l "*jack*"' '-l "*jack*:mod:snd"' '--module snd -n "*"' '-n :mod:snd' --proc-map trace.dat . -P command . --profile --profile "trace-cmd" --profile trace-cmd-report(1) . -p function graph function_graph ( ). (tracepoints) (tasks) . trace-cmd-profile(1) . -G ( ) ( CPU ) . --profile . -H event-hooks . --profile trace-cmd report --profile : trace-cmd record -H hrtimer_expire_entry,hrtimer/hrtimer_expire_exit,hrtimer,sp trace-cmd report --profile hrtimer_expire_entry hrtimer_expire_ext . trace-cmd-profile(1) . -S ( --profile) . function_graph ( sched_switch) ( -p function -e sched_switch -e sched_wakeup) --temp directory trace-cmd CPU . trace.dat . . --temp trace-cmd . --ts-offset offset ( ) trace.dat . offset trace.dat trace-cmd report . . --tsync-interval ( ) (guests) : . 0 - : . VSOCK . --tsc2nsec (multiplier) tsc perf . tsc . "-C tsc2nsec" . --stderr stdout stderr . trace-cmd . --poll (ring-buffers) (IPI) . (real-time) . poll trace-cmd O_NONBLOCK . (busy waiting) . --name (agent) . -A . vsocket . --verbose[=level] (log level) . "none" "critical" "error" "warning" "info" "debug" "all" "0" "1" "2" "3" "4" "5" "6". . "info" . : critical error warning trace-cmd record --verbose=warning --file-version . 6 7 . --compression : 'any' - 'none' - 'name' - . : trace-cmd list -c --proxy vsocket agent vsocket . -A ( agent) agent . ( cid -P agent ). --daemonize trace-cmd . pidfile /var/run/trace-cmd-record.pid pid trace-cmd . (EXAMPLES) : # trace-cmd record -e all ls > /dev/null # trace-cmd report trace-cmd-13541 [003] 106260.693809: filemap_fault: address=0x128122 offset=0xce trace-cmd-13543 [001] 106260.693809: kmalloc: call_site=81128dd4 ptr=0xffff88003dd83800 bytes_req=768 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_ZERO ls-13545 [002] 106260.693809: kfree: call_site=810a7abb ptr=0x0 ls-13545 [002] 106260.693818: sys_exit_write: 0x1 (function tracer) sched switch: # trace-cmd record -p function -e sched_switch ls > /dev/null # trace-cmd report ls-13587 [002] 106467.860310: function: hrtick_start_fair <-- pick_next_task_fair ls-13587 [002] 106467.860313: sched_switch: prev_comm=trace-cmd prev_pid=13587 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=13583 next_prio=120 trace-cmd-13585 [001] 106467.860314: function: native_set_pte_at <-- __do_fault trace-cmd-13586 [003] 106467.860314: function: up_read <-- do_page_fault ls-13587 [002] 106467.860317: function: __phys_addr <-- schedule trace-cmd-13585 [001] 106467.860318: function: _raw_spin_unlock <-- __do_fault ls-13587 [002] 106467.860320: function: native_load_sp0 <-- __switch_to trace-cmd-13586 [003] 106467.860322: function: down_read_trylock <-- do_page_fault : # trace-cmd record -p function_graph -e irq_handler_entry -l do_IRQ sleep 10 # trace-cmd report -0 [000] 157412.933969: funcgraph_entry: | do_IRQ() { -0 [000] 157412.933974: irq_handler_entry: irq=48 name=eth0 -0 [000] 157412.934004: funcgraph_exit: + 36.358 us | } -0 [000] 157413.895004: funcgraph_entry: | do_IRQ() { -0 [000] 157413.895011: irq_handler_entry: irq=48 name=eth0 -0 [000] 157413.895026: funcgraph_exit: + 24.014 us | } -0 [000] 157415.891762: funcgraph_entry: | do_IRQ() { -0 [000] 157415.891769: irq_handler_entry: irq=48 name=eth0 -0 [000] 157415.891784: funcgraph_exit: + 22.928 us | } -0 [000] 157415.934869: funcgraph_entry: | do_IRQ() { -0 [000] 157415.934874: irq_handler_entry: irq=48 name=eth0 -0 [000] 157415.934906: funcgraph_exit: + 37.512 us | } -0 [000] 157417.888373: funcgraph_entry: | do_IRQ() { -0 [000] 157417.888381: irq_handler_entry: irq=48 name=eth0 -0 [000] 157417.888398: funcgraph_exit: + 25.943 us | } : # trace-cmd record --profile sleep 1 # trace-cmd report --profile --comm sleep task: sleep-21611 Event: sched_switch:R (1) Total: 99442 Avg: 99442 Max: 99442 Min:99442 1 total:99442 min:99442 max:99442 avg=99442 => ftrace_raw_event_sched_switch (0xffffffff8105f812) => __schedule (0xffffffff8150810a) => preempt_schedule (0xffffffff8150842e) => ___preempt_schedule (0xffffffff81273354) => cpu_stop_queue_work (0xffffffff810b03c5) => stop_one_cpu (0xffffffff810b063b) => sched_exec (0xffffffff8106136d) => do_execve_common.isra.27 (0xffffffff81148c89) => do_execve (0xffffffff811490b0) => SyS_execve (0xffffffff811492c4) => return_to_handler (0xffffffff8150e3c8) => stub_execve (0xffffffff8150c699) Event: sched_switch:S (1) Total: 1000506680 Avg: 1000506680 Max: 1000506680 Min:1000506680 1 total:1000506680 min:1000506680 max:1000506680 avg=1000506680 => ftrace_raw_event_sched_switch (0xffffffff8105f812) => __schedule (0xffffffff8150810a) => schedule (0xffffffff815084b8) => do_nanosleep (0xffffffff8150b22c) => hrtimer_nanosleep (0xffffffff8108d647) => SyS_nanosleep (0xffffffff8108d72c) => return_to_handler (0xffffffff8150e3c8) => tracesys_phase2 (0xffffffff8150c304) Event: sched_wakeup:21611 (1) Total: 30326 Avg: 30326 Max: 30326 Min:30326 1 total:30326 min:30326 max:30326 avg=30326 => ftrace_raw_event_sched_wakeup_template (0xffffffff8105f653) => ttwu_do_wakeup (0xffffffff810606eb) => ttwu_do_activate.constprop.124 (0xffffffff810607c8) => try_to_wake_up (0xffffffff8106340a) --daemonize /: $ sudo trace-cmd record --daemonize -p nop -e 'sched:sched_process_exec' -A guest -p nop -e net && > ping -c 1 10.20.1.2 && > sudo start-stop-daemon --stop --signal INT --retry 20 --pidfile /var/run/trace-cmd-record.pid && > sudo trace-cmd report -i trace.dat -i trace-guest.dat | head Negotiated kvm time sync protocol with guest guest Send SIGINT to pid 3071371 to stop recording PING 10.20.1.2 (10.20.1.2) 56(84) bytes of data. 64 bytes from 10.20.1.2: icmp_seq=1 ttl=64 time=0.134 ms --- 10.20.1.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.134/0.134/0.134/0.000 ms CPU0 data recorded at offset=0x14f000 229 bytes in size (4096 uncompressed) .... trace.dat: cpus=28 trace-guest.dat: cpus=1 trace.dat: ping-3071450 [013] 1196830.834258: sched_process_exec: filename=/bin/ping pid=3071450 old_pid=3071450 trace-guest.dat: -0 [000] 1196830.835990: napi_gro_receive_entry: dev=eth1 napi_id=0x2002 queue_mapping=1 skbaddr=0xffff95d051a5c400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0 trace-guest.dat: -0 [000] 1196830.835997: napi_gro_receive_exit: ret=3 trace-guest.dat: -0 [000] 1196830.835998: netif_receive_skb: dev=eth1 skbaddr=0xffff95d051a5c400x len=84 trace-guest.dat: -0 [000] 1196830.836021: net_dev_queue: dev=eth1 skbaddr=0xffff95d051a5c700x len=98 trace-guest.dat: -0 [000] 1196830.836024: net_dev_start_xmit: dev=eth1 queue_mapping=0 skbaddr=0xffff95d051a5c700 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0 trace-guest.dat: -0 [000] 1196830.836069: net_dev_xmit: dev=eth1 skbaddr=0xffff95d051a5c700 len=98 rc=0 trace.dat: sudo-3071451 [015] 1196830.838262: sched_process_exec: filename=/usr/bin/sudo pid=3071451 old_pid=3071451 trace-cmd(1), trace-cmd-report(1), trace-cmd-start(1), trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1), trace-cmd-list(1), trace-cmd-listen(1), trace-cmd-profile(1) (AUTHORS) Steven Rostedt (RESOURCES) https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git (COPYRIGHT) (C) 2010 Red Hat, Inc. << >> (GPL) . (NOTES) 1. rostedt@goodmis.org mailto:rostedt@goodmis.org libtracefs 03/09/2026 TRACE-CMD-RECORD(1)