PCP-ATOP(1) General Commands Manual PCP-ATOP(1) (NAME) pcp-atop - PCP (SYNOPSIS) : pcp [pcp options] atop [-aABcCdDfFgGHmMnNopRsuvxyY1] [-L linelen] [-Plabel[,label]... [-Z]] [interval [samples]] PCP: pcp [pcp options] atop -w directory [-a] [-S] [interval [samples]] pcp [pcp options] atop -r archive [-AcCdDfFgGmMnNopRsuvxy1] [-b [yy-mm- dd] hh:mm] [-e yy-mm-dd] hh:mm] [-L linelen] [-Plabel[,label]... [-Z]] [interval [samples]] (DESCRIPTION) pcp-atop . interval (: ) ( ) CPU . ( ) CPU . pmdabpf(1) . (BAR GRAPH MODE) pcp-atop . CPU ( ) . 'B' . 'B' . pcp-atop -B 'B' .atoprc ( pcp-atoprc(5) ). : (Processors) CPU 'Avg' ( 'Av' 'A' ). CPU . CPU CPU . CPU 'S' ( ) 'U' ( ) 'I' ( ) 's' ( steal) 'G' ( ) . <<>> . . ( ) ( ) . <<>> . 'H' pcp-atop -H . . 'cpucritperc' ( pcp-atoprc(5) ). . (Memory and swap space) . ( ) / ( ) tmpfs (static huge pages) (page cache) . Swap ( ) /tmpfs . . . . . ( ) (swap out). 'oomkills' ( Out Of Memory) : ( ) . ( ) . pcp-atop OOM killing 'oomkills' OOM killing . (swapped in) . . / . (Disks) . . 'R' () 'W' () . <<>> . . ( ) ( ) . <<>> . 'H' pcp-atop -H . . 'dskcritperc' ( atoprc ). . (Interfaces) . ('TX') ('RX') . . 'L' . ( ) . . . . . . . . . (TEXT MODE IN GENERAL) pcp-atop ( ) ( ) . pcp-atop ( root ) . ( CPU ) . 'a' ( ). ( ) . samples ( ) 'q' . pcp(1) PCPIntro(1) -A/--align -a/--archive -h/--host -O/--origin -S/--start -s/--samples -T/--finish -t/--interval -v/--version -z/--hostzone -z/--timezone . --hotproc PCP proc pmdaproc(1) . pcp-atop /. ( ncurses) . pcp-atop / . . . . / . pcp-atop . . pcp-atop . . pcp-atop (pipe) ( ) . ( ). pcp-atop PCP ( PCP ). (COLORS) pcp-atop () . . : / CPU . pcp-atop : (Processor) <<>> ( ). (Disk) <<>> . (Network) <<>> . (Memory) <<>> . ( ) . ('cache' 'buff' MEM) ('slrec') ! ('swout' PAG) <<>> . << >> . ('vmcom' 'vmlim' SWP) SWP . (Swap) <<>> . . ( pcp-atoprc(5) ). . ( ) ( ) (cyan) . << >> ( ) ( pcp-atoprc(5) ). ( pcp-atoprc(5) ). 'x' ( -x) . . BPF NETATOP (NETATOP BPF MODULE) BPF netatop pmdabpf(1) . pcp-atop eBPF . pcp-atop / . 'n' . (GPU STATISTICS GATHERING) (GPU) pmdanvidia(1) . Nvidia GPU . pcp-atop GPU . . (INTERACTIVE COMMANDS) pcp-atop ( ) . / . . g (). : (process-id) CPU . / pcp-atop (root) . pmdabpf(1) netatop / . (: CPU) . . m . : (minor and major faults) . . ( ). d . pcp-atop : (WCANCL) . n . : TCP ( ) TCP ( ) UDP ( ) UDP ( ) . pmdabpf(1) netatop . . s (scheduling). : 'running' (R) 'interruptible sleeping' (S) 'uninterruptible sleeping' (D) 'idle' (I) ( realtime round-robin realtime fifo) nice . . v . : ( ) . . c . : . X cgroup v2. : cgroup ( ). e GPU. : GPU GPU ( ) GPU. pmdanvidia GPU . GPU GPU GPU ( ). o . ownprocline . pcp-atoprc(5) . y ( - toggle). . ( ) . 'a' ( ) . 'Y' ( ) . . Y 'y' ( ). u . : ( 'a') CPU ( 'a'). pcp-atop . pmdabpf(1) 'netproc' . (: CPU) . p ( ). : ( 'a') CPU ( 'a'). pcp-atop . pmdabpf(1) 'netproc' . (: CPU) . j /. ( Docker/Podman) ( Kubernetes) : ( 'a') CPU ( 'a'). pcp-atop . pmdabpf(1) 'netproc' . (: CPU) / (CID/POD) . C CPU (). 'CPU' . E GPU ( pmdanvidia ) GPU. 'GPU' . M . 'MEM' . ( ). D . 'DSK' . N ( ). 'NET' . A . 'ACPU' 'AMEM' 'ADSK' 'ANET' ( 'A' ). . . pcp-atop . pmdabpf(1) 'netproc' . : ? ( 'h' ). V ( ). R (PSIZE) ( ). PSIZE . W WCHAN ( ). WCHAN . x ( ). . z (pause) . pcp-atop . pcp-atop . . i (: ). . 't' . . t . ( ). pcp-atop . . . T . . b ( ) . . r (boot). . . U (regular expression). () . . Enter () . . I PID . PID . . Enter PID () . . P . . . Enter () . . / . . . Enter () . . J ( Docker Podman) ( Kubernetes) . . . . Enter () . . Q . . : R ( ) S ( ) D ( ) T () t ( ) X () Z () P (). . Enter () . . S . . . Enter () . . a '/' / () / . . G pcp-atop / . ( ) / . . f () ( ). (CPU ) . pcp-atop . . F ( ). (CPU ) . . 1 ( '..../s') ( ). . l CPU . CPU . CPU pcp-atop / . . -l CPU . . k ( kill ). q . . PgDn /. . ^F / ( ). . PgUp /. . ^B / ( ). . ^L . PCP (PCP DATA STORAGE) ( : :) pcp-atop PCP ( mkaf(1) ). . (: ) (: ) . -S pcp-atop . PCP -r . . $PCP_LOG_DIR/pmlogger/HOST/YYYYMMDD ( YYYYMMDD HOST ). YYYYMMDD ( ) $PCP_LOG_DIR/pmlogger/HOST/YYYYMMDD . y ( 'yyyy' ). 't' 'T' 'b' 'r' . . (pipe) pcp-atop . . -L ( ) . -b ( ) / -e ( ) [YYYYMMDD]hhmm[ss] . (OUTPUT DESCRIPTION) (boot) ( ). pcp-atop . . . . CPU . pcp-atop . CPU '233s' ( ) . (Mb Kb Gb Mb Tb Gb ). ( 123e6 ). - (OUTPUT DESCRIPTION - SYSTEM LEVEL) : PRC . CPU ('sys') ('user') ('#proc') 'running' ('#trun') 'sleeping interruptible' ('#tslpi') 'sleeping uninterruptible' ('#tslpu') 'idle' ('#tidle') ('#zombie') clone ('clones') ('#exit') (process accounting) . '#exit' ('no procacct'). . CPU (CPU). CPU . ( 'cpu' ) . CPU . CPU CPU . CPU ('sys') CPU ('user') ( nice ) CPU ('irq') (softirq) CPU / ('idle') CPU / ('wait') . CPU CPU ('w') CPU . CPU . ('steal') CPU . ('guest') CPU . ! PMC CPU ( pmdaperfevent(1) ) CPU 'ipc' . 'initial' pcp-atop . CPU IPC CPU ( CPU ). CPU IPC ( / ). CPU ('cycl') . CPU () CPU . CPU . CPU . : http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html (frequency scaling) CPU CPU . CPU CPU CPU . 'cpufreq_stats' ( 'modprobe cpufreq_stats') ('avgf') ('avgscal') . ('curf') ('curscal') . CPU . CPU . . CPL CPU. (load average) ( ) / . ('avg1') ('avg5') ('avg15') . ('csw') ('intr') CPU . . GPU GPU (). pmdanvidia . (bus-id ) GPU . GPU ('gpubusy') ('membusy') ('memocc') ('total') ('used') ('usavg') GPU ('#proc') GPU. . GPU . MEM ( ). ('tot') ('free') ('avail') ('cache') ('dirty') ('buff') ('slab') ('slrec') SYSV tmpfs ('shmem') SYSV ('shrss') SYSV ('shswp') ('pgtab') NUMA ('numnode') vmware ('vmbal') ARC () ZFSonlinux ('zfarc') ('anthp') ('hptot') ('hpuse') TCP ('tcps') UDP ('udps') . . SWP (overcommit). ('tot') ('free') ('swcac') zswap ('zswap') () zswap ('zstor') KSM ('ksuse' ) () KSM ('kssav' ) . ('vmcom') ('vmlim' ) . . (vm.overcommit_memory ) . LLC (Last-Level Cache) . LLC ('tot') NUMA ('loc') LLC ('LLCXX YY%') . 'resctrl' . (mount) : mount -t resctrl resctrl -o mba_MBps /sys/fs/resctrl ( Intel) mount -t resctrl resctrl -o cdp /sys/fs/resctrl ( AMD) NUM NUMA ( NUMA ). ('tot') ('free') ('file') ('dirty') ('activ') ('inact') ('slab') ('slrec') tmpfs ('shmem') ('hptot') ('hpuse') ('frag') . NUC CPU NUMA ( NUMA ). CPU NUMA ('sys') ('user') nice ('niced') ('idle') ('w' ) ('irq') ('sirq') ('steal') ('guest') . PAG (Paging frequency). ('scan') ('stall') ('compact') NUMA ('numamig') NUMA ('migrate') . ('pgin') ('pgout') zswap ('zswin') zswap ('zswout') ('swin') ('swout') ('oomkill') . PSI (Pressure Stall Information). CPU / (I/O) . 'some' / . 'full' . 'cpusome' 'memsome' 'memfull' 'iosome' 'iofull' . 'cs' ( CPU) 'ms' ( ) 'mf' ( ) 'is' ( I/O) 'if' ( I/O) : . LVM/MDD/DSK / /. . ( VolGroup00-lvtmp sda ) I/O ('busy') ( RAID SSD NVMe ) ('read') ('write') (discard) ('discrd') ('KiB/r') ('KiB/w') ('KiB/d') ('MBr/s') ('MBw/s') ('inflt') ('avq') ('avio') (seek) . . (LVM MDD DSK) 'l' ( pcp-atoprc(5) ). . NFM (NFS) . NFS ('srv') ('read') ('write') . : read() ('nread') read() ('nwrit') / Direct I/O ('dread') Direct I/O ('dwrit') I/O ('mread') I/O ('mwrit'). NFC (NFS). RPC ('rpc') RPC ('read') ('rpwrite') NFS RPC ('retxmit') ('autref') . NFS (NFS). RPC NFS ('rpc') RPC ('cread') RPC ('cwrit') ('MBcr/s') ('MBcw/s') TCP ('nettcp') UDP ('netudp') ('rchits') ('rcmiss') ('rcnoca') . ('badfmt') ('badaut') ('badcln') . NET (TCP/IP). (TCP UDP) IP . TCP ('tcpi') TCP ('tcpo') UDP ('udpi') UDP ('udpo') TCP ('tcpao') TCP ('tcppo') TCP ('tcprs') TCP ('tcpie') TCP ('tcpor') UDP ('udpnp') UDP ('udpie') TCP ('csumie') . . IPv4 IPv6 . IP IP ('ipi') IP ('ipo') IP ('ipfrw') IP ('deliv') ICMP ('icmpi') ICMP ('icmpo') . . IPv4 IPv6 . . . (half duplex) (full duplex) . ( loopback) '---' . ('pcki') ('pcko') ('sp') ('si') ('so') ('coll') (multicast) ('mlti') ('erri') ('erro') ('drpi') ('drpo') . . . IFB (Infiniband). Infiniband . . (lanes) . ('pcki') ('pcko') ('sp') ('si') ('so') ('lanes') . . Infiniband . - (OUTPUT DESCRIPTION - PROCESS LEVEL) . CPU . ( ) . ( ) : AVGRSZ . AVGWSZ . BANDWI TCP UDP ( ). 'si' ( ) . pmdabpf(1) 'netproc' . BANDWO TCP UDP ( ). 'so' ( ) . pmdabpf(1) 'netproc' . BDELAY / / . CGROUP cgroup ( ) . cgroup '/sys/fs/cgroup' . CID/POD ( Docker Podman) ( Kubernetes) / / . . '?' . atop (superuser) . CMD . <</>> ('') . . . 'CMD' / . COMMAND-LINE ( ). -> <- . '-z ' . POSIX ( regex(3) ). ( ) (truncate) . 'COMMAND-LINE' / . CPU . CPUNR CPU () . CTID (OpenVZ). '?' . DSK ( ). << >> (storage accounting) . EGID (effective group-id) . ENDATE . 'active' . ENTIME . 'active' . ENVID ( OpenVZ). EUID (effective user- id) . EXC ( 'ST' E ) ( 'ST' S C ). FSGID . FSUID . GPU pmdanvidia GPU GPU ( GPU ). pmdanvidia GPU GPU . GPUBUSY GPU ( GPU ). pmdanvidia . GPUNUMS GPU . . LOCKSZ ( swap) ( ). MAJFLT / (major page faults). MEM . MEMAVG GPU . MEMBUSY GPU ( GPU ) . pmdanvidia . MEMNOW GPU . MINFLT (minor page faults). NET ( ). pmdabpf(1) 'netproc' . NICE -20 ( ) +19 ( ) . NIVCSW / (time slice) . NPROCS . NVCSW / (blocking) /. PID (Process-id). POLI 'norm' ( SCHED_OTHER ) 'btch' ( batch) 'idle' . 'fifo' (SCHED_FIFO) 'rr' ( round robin SCHED_RR ) (realtime) . PPID (Parent process-id). PRI ( ) ( ) . ( ) ( CPU nice) . PSIZE ( ). . (text) . . PSIZE . . PSIZE . PSIZE 'R' ( '-R') . ( '?K' ). RDDSK ( ). () cron bash init . RDELAY (runqueue delay) . RGID (real group-id) . RGROW . / ( load-on-demand). . . RNET TCP UDP . pmdabpf(1) netatop . . RSIZE ( ). RSIZE ( PSIZE ). RTPR POSIX. ( 'norm' 'btch' 'idle') () () ( 'rr' 'fifo'). RUID (real user-id) . S (): 'R' ( ) 'S' ( ) 'D' 'Z' ( ) 'T' ( ) 'W' 'E' () . SGID (saved group-id) . SNET TCP UDP . pmdabpf(1) netatop . ST . ( N ' ' ). . E '' 'EXC' . S 'EXC' . C core dump 'EXC' . STDATE . STTIME . SUID (saved user-id) . SWAPSZ (swap) ( ). SYSCPU CPU ( ) . TCPRASZ TCP . pmdabpf(1) netproc . TCPRCV tcp_recvmsg()/tcp_cleanup_rbuf() . pmdabpf(1) netproc . TCPSASZ TCP . pmdabpf(1) netproc . TCPSND tcp_sendmsg() . pmdabpf(1) netproc . THR . pcp-atop 'y' ( -y) . TID (Thread-id). PID TID . ( 'y') . TIDLE 'idle' (I) (load average) . TRUN 'running' (R) . TSLPI 'interruptible sleeping' (S) . TSLPU 'uninterruptible sleeping' (D) . UDPRASZ UDP . pmdabpf(1) netproc . UDPRCV udp_recvmsg()/skb_consume_udp() . pmdabpf(1) netproc . UDPSASZ UDP . pmdabpf(1) netproc . UDPSND udp_sendmsg() . pmdabpf(1) netproc . USRCPU CPU . VDATA ( heap ). VGROW . malloc() . free() . . VPID ( OpenVZ). '?' . VSIZE ( ). VSLIBS () . VSTACK () . VSTEXT () . WCHAN (wait channel) . 'W' ( '-W') . WRDSK ( ). ( ). / swap . () cron bash init . WCANCL . . WRDSK / WCANCL . (PARSABLE OUTPUT) -P ( ) (parsable) . ( ) : "CPU" "cpu" "CPL" "GPU" "MEM" "SWP" "PAG" "PSI" "LVM" "MDD" "DSK" "NFM" "NFC" "NFS" "NET" "IFB" "LLC" "NUM" "NUC". : "PRG" () "PRC" () "PRE" ( ) "PRM" () "PRD" ( " " ) "PRN" ( pmdabpf(1) netproc ). "ALL" . (space) . -Z ( ) (underscore) . pcp-atop "SEP" . (boot) pcp-atop "RESET" . : label ( ) host ( ) epoch ( 1-1-1970) date ( YYYY/MM/DD) time ( HH:MM:SS) interval ( ). : CPU : CPU ( ) CPU ( ) CPU nice ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU CPU. cpu : CPU ( ) CPU ( ) CPU nice ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU CPU CPU CPU. CPL : . GPU : GPU (bus-id) GPU GPU ( -1) ( -1) (KiB) (KiB) GPU ( -1) ( -1) ( ). MEM : ( ) () () () () () () () vmware () () () () ( ) ( ) ( ) ARC () ZFSonlinux () KSM () KSM () TCP () UDP () () ( ) ( ) ( ) () ( ''). SWP : ( ) () () () () () () () zswap () zswap (). LLC : LLC LLC LLC ( ) NUMA LLC ( ). PAG : ( ) (allocstalls) ( ) (swapins) (swapouts) OOM ( -1) NUMA zswap zswap. PSI : PSI (y n) CPU some avg10 CPU some avg60 CPU some avg300 CPU memory some avg10 memory some avg60 memory some avg300 memory full avg10 memory full avg60 memory full avg300 I/O some avg10 I/O some avg60 I/O some avg300 I/O I/O full avg10 I/O full avg60 I/O full avg300 I/O . LVM/MDD/DSK / / . : I/O (sectors) ( -1) () . NFM : NFS direct I/O direct I/O I/O I/O . NFC : RPC RPC RPC RPC . NFS : RPC RPC RPC RPC RPC RPC TCP UDP TCP . NET TCP/IP . : "upper" TCP TCP UDP UDP IP IP IP IP (UDP) noport ( UDP) (TCP) (TCP) (TCP) (TCP) (TCP) (TCP) (TCP) (TCP). . : (0= 1=). IFB : InfiniBand (Mbps) . NUM : NUMA ( ) () () () () () () () () () () tmpfs () ( ) ( ). NUC : NUMA CPU ( ) CPU ( ) CPU nice ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) CPU ( ) . PRG . : PID ( ) ( ) uid gid TGID ( / ) ( : + ) (epoch) ( ) PPID 'running' (R) 'interruptible sleeping' (S) 'uninterruptible sleeping' (D) uid gid uid gid uid gid () is_process (y/n) OpenVZ (VPID) OpenVZ (CTID) / (CID/POD) ('N') cgroup v2 ( ) (epoch ) 'idle' (I). PRC . : PID ( ) CPU ( ) CPU ( ) nice CPU TGID ( / ) is_process (y/n) ( ) ( ) / ( ) . PRE . : PID ( ) GPU ( A E N GPU) GPU GPU GPU (KiB) (KiB) . PRM . : PID ( ) ( ) () () () () () () () () () TGID ( / ) is_process (y/n) () 'R' (). PRD . : PID ( ) ('n') io ('y' 'n') TGID ( / ) ('n') is_process (y/n). PRN . : PID ( ) pmdabpf(1) 'netproc' ('y' 'n') tcp_sendmsg() TCP tcp_recvmsg()/tcp_cleanup_rbuf() TCP udp_sendmsg() UDP udp_recvmsg()/skb_consume_udp() UDP ( ) ( ) TGID ( / ) is_process (y/n). (SIGNALS) SIGUSR1 pcp-atop . 't' . SIGUSR2 pcp-atop pcp-atop . (EXAMPLES) () : pcp atop : pcp atop -B 5 PCP : pcp atop -w /tmp/pcp-atop 600 6 : pcp atop -r /tmp/pcp-atop : pcp atop -PCPU,DSK -r /tmp/pcp-atop : pcp atop -r : pcp atop -r yy : : pcp atop -r 20230607 -b 14:00 ( ASCII) : pcp atop -M 60 30 > /log/pcp-atop.mem (NOTES) pcp-atop atop(1) https://atoptool.nl Gerlof Langeveld (gerlof.langeveld@atoptool.nl) . pcp-atop ( Apache HTTP NFS GPU TCP UDP ) PCP . pmdaapache(1) pmdabpf(1) pmdainfiniband(1) pmdanfsclient(1) pmdanvidia(1) pmdaperfevent(1) pmdaresctrl(1) . pcp-atop (-w) pmdapmimport(1) pcp(1) . atop(1) : TCP/UDP / ( TCP ) pcp-atop tcp_sendmsg()/udp_sendmsg()/etc. . (FILES) /etc/atoprc . pcp-atoprc(5) . ~/.atoprc . pcp-atoprc(5) . PCP (PCP ENVIRONMENT) PCP_ PCP . /etc/pcp.conf . $PCP_CONF pcp.conf(5) . PCP pmGetOptions(3) . (DEBUGGING OPTIONS) -D --debug pcp stderr . debug pmdbg(1) -l . pcp-atop : +----------------+---------------------------------------------------------------------------------------+ | | | +----------------+---------------------------------------------------------------------------------------+ |appl0 | | +----------------+---------------------------------------------------------------------------------------+ |appl1 | | +----------------+---------------------------------------------------------------------------------------+ (SEE ALSO) PCPIntro(1) pcp(1) pcp-atopsar(1) pmdaapache(1) pmdabpf(1) pmdainfiniband(1) pmdanfsclient(1) pmdanvidia(1) pmdaperfevent(1) pmdapmimport(1) pmdaproc(1) pmdaresctrl(1) mkaf(1) pmlogger(1) pmlogger_daily(1) pcp-atoprc(5). Performance Co-Pilot PCP PCP-ATOP(1)