nvidia-smi(1) NVSMI nvidia-smi(1) (NAME) nvidia-smi - NVIDIA (SYNOPSIS) nvidia-smi [OPTION1 [ARG1]] [OPTION2 [ARG2]] ... (DESCRIPTION) nvidia-smi ( NVSMI) Tesla Quadro GRID GeForce Fermi . GeForce Titan GeForce . NVSMI NVIDIA Windows Server 2008 R2 . stdout CSV XML . NVSMI C NVML . NVML NVIDIA . (Python bindings) NVML . ( ) NVSMI . NVML NVIDIA . NVML SDK: https://docs.nvidia.com/deploy/nvml-api/index.html (Python bindings): http://pypi.python.org/pypi/nvidia-ml-py (OPTIONS) (GENERAL OPTIONS) -h, --help . --version . (LIST OPTIONS) -L, --list-gpus GPU NVIDIA UUID . -B, --list-excluded-gpus GPU NVIDIA UUID . (SUMMARY OPTIONS) GPU . -col, --columns GPU . [ ] ([any one of]) -i, --id=ID GPU . -f FILE, --filename=FILE (stdout). -l SEC, --loop=SEC Ctrl+C. (QUERY OPTIONS) -q, --query GPU (Unit). (GPU ATTRIBUTES) (UNIT ATTRIBUTES) . / . "N/A" . GPU . -i GPU . [ ] ([plus optionally]) -u, --unit (Unit) GPU. Tesla S . -i, --id=ID GPU . (id) GPU/ GPU UUID PCI ( domain:bus:device.function ) . UUID PCI GPU . -f FILE, --filename=FILE (stdout). . -x, --xml-format XML . GPU DTD . --dtd . --dtd -x. DTD XML. --debug=FILE () NVIDIA. -d TYPE, --display=TYPE : MEMORY UTILIZATION ECC TEMPERATURE POWER CLOCK COMPUTE PIDS PERFORMANCE SUPPORTED_CLOCKS PAGE_RETIREMENT ACCOUNTING ENCODER_STATS SUPPORTED_GPU_TARGET_TEMP VOLTAGE FBC_STATS ROW_REMAPPER GSP_FIRMWARE_VERSION POWER_SMOOTHING POWER_PROFILES. "MEMORY,ECC". max min avg POWER UTILIZATION CLOCK . -u/--unit -x/--xml-format . -l SEC, --loop=SEC . . -x ECC Xid . Ctrl+C . -l . -lms ms, --loop-ms=ms -l,--loop . (SELECTIVE QUERY OPTIONS) . [ ] ([one of]) --query-gpu= GPU. --query-gpu=pci.bus_id,persistence_mode. --help-query-gpu . --query-supported-clocks= . --help-query-supported-clocks . . --help-query-compute-apps . --query-accounted-apps= (accounted). --help-query-accounted-apps . vGPU . --query-retired-pages= GPU (retired) . --help-query-retired-pages . --query-remapped-rows= (remapped rows). --help-query-remapped-rows . [] --format= : o csv - () o noheader - o nounits - [ ] -i, --id=ID GPU . (id) GPU GPU UUID PCI bus ID ( domain:bus:device.function ) . UUID PCI bus ID GPU . -f FILE, --filename=FILE stdout . . -l SEC, --loop=SEC . (sleep) . -x ECC Xid . Ctrl+C . -l . -lms ms, --loop-ms=ms -l,--loop . (DEVICE MODIFICATION OPTIONS) [ ] -pm, --persistence-mode=MODE (persistence mode) GPU . (GPU ATTRIBUTES) . (root) . GPU GPU -i . . "Disabled" . . -e, --ecc-config=CONFIG ECC GPU . ECC (GPU ATTRIBUTES) . (root) . GPU GPU -i . . -p, --reset-ecc-errors=TYPE ECC GPU . ECC (GPU ATTRIBUTES) . 0\|VOLATILE 1\|AGGREGATE . (root) . GPU GPU -i . . (aggregate) Ampere+ . -c, --compute-mode=MODE (compute mode) GPU . (GPU ATTRIBUTES) . (root) . GPU GPU -i . . "DEFAULT" . -dm TYPE, --driver-model=TYPE -fdm TYPE, --force-driver-model=TYPE (driver model). Windows. (administrator) . -dm -fdm . GPU GPU -i . GPU() GPU . '--no- driver-restart' . . Driver Model . . --gom=MODE (GPU Operation Mode): 0/ALL_ON 1/COMPUTE 2/LOW_DP GK110 M X Kepler . Quadro Tesla C . LOW_DP ALL_ON GeForce Titan . . GOM GPU Operation Mode . GOM . . GOM (Compute only) WDDM ( ) . -r, --gpu-reset (reset) GPU . GPU . ECC . -i . GPU . (root) . ( CUDA X nvidia-smi). GPU GPU . NVIDIA Ampere GPU NVLink . NVSwitch Ampere Fabric Manager . NVSwitch Hopper Fabric Manager . Fabric Manager GPU NVIDIA Ampere GPU NVLink GPU . -i -i GPU . -i GPU NVLink GPU . : NVSwitch: GPU Family | Fabric Manager running | Fabric Manager not running ------------|------------------------------|------------------------------ Pre-Ampere | All PEER connected GPUs must | All PEER connected GPUs must | be reset in same command. | be reset in same command Ampere+ | Each GPU can be reset | All PEER connected GPUs must | individually | be reset in same command NVLink : (FM NVSwitch ) GPU Family | Capabilities ------------|------------------------------------------------------- Pre-Ampere | All PEER connected GPUs must be reset in same command Ampere+ | Each GPU can be reset individually GPU . . . Fermi Kepler GPU (hung) . GPU . GPU (power cycle) . (Function Level Reset FLR) . (Bus Reset) -r bus . . GPU . -i GPU() . -i GPU . . (administrator) . GPU vGPU MIG . GDK http://developer.nvidia.com/gpu-deployment-kit . GPU. GPU 3/VGPU 4/VSGA . GPU (hypervisor) . -lgc, --lock-gpu-clocks=MIN_GPU_CLOCK,MAX_GPU_CLOCK ( 1500,1500) GPU (MHz) . ( ). --mode . Volta+ . (root) . --mode=0 () . --mode=1 (perf per watt) . 0 . -lmi, --lock-memory-clocks-info GPU (not supported [] deferred [] runtime modifiable [ ]) . : (settling time) . -lmc, --lock-memory-clocks=MIN_MEMORY_CLOCK,MAX_MEMORY_CLOCK ( 5100,5100) (MHz) . ( ). (root) . : GPU NVIDIA Hopper --lock-memory- clocks-deferred . -rgc, --reset-gpu-clocks GPU . Volta+ . (root) . -rmc, --reset-memory-clocks . Volta+ . (root) . -ac, --applications-clocks=MEM_CLOCK,GRAPHICS_CLOCK CUDA . -lmc -lgc . ( 2000,800) GPU GPU . GeForce Maxwell Kepler+ Tesla/Quadro/Titan . (root) . -rac, --reset-applications-clocks CUDA . . GeForce Maxwell Kepler+ Tesla/Quadro/Titan . (root) . -lmcd, --lock-memory-clocks-deferred (MHz) . GPU . (unload) (reload) . (root) . -rmcd, --reset-memory-clocks-deferred . . . (root) . -svfd, --set-vf-derate=OFFSET (MHz) - (VF) (deterministic) . GPU . Rubin . . -gvfd, --get-vf-derate-info VF VF GPU . Rubin . -pl, --power-limit=POWER_LIMIT . . --scope . Kepler. (Power Limit) nvidia-smi . (root) . -sc, --scope=0/GPU, 1/TOTAL_MODULE, 2/GPU_BASE (scope) . : 0/GPU: GPU . 1/Module: ( GPU CPU) . 2/GPU Base: GPU . -cc, --cuda-clocks=MODE CUDA . : 0\|RESTORE_DEFAULT 1\|OVERRIDE. (root) . -am, --accounting-mode=MODE GPU ( GPU Accounting) . GPU . Kepler. . 0\|DISABLED 1\|ENABLED. -caa, --clear-accounted-apps . Kepler. . --auto-boost-default=MODE CUDA . (auto boost) 0/DISABLED 1/ENABLED . Tesla Kepler+ GeForce Maxwell. (root) . --auto-boost-permission=MODE CUDA . (auto boost) / . 0\|UNRESTRICTED 1\|RESTRICTED. Tesla Kepler+ GeForce Maxwell. (root) . -mig, --multi-instance-gpu=MODE Multi Instance GPU . NVIDIA Ampere . (root) . 0\|DISABLED 1\|ENABLED. -gtt, --gpu-target-temp=MODE GPU GPU . GPU . SUPPORTED_GPU_TARGET_TEMP . (Root) . --set-hostname=hostname (hostname) . ( NULL). (root) . --get-hostname (hostname) . [ ] -i, --id=ID GPU . (id) GPU/ GPU UUID PCI bus ID ( domain:bus:device.function ) . UUID PCI bus ID GPU . -eom, --error-on-warning . (UNIT MODIFICATION OPTIONS) -t, --toggle-led=STATE LED . LED (UNIT ATTRIBUTES) . 0\|GREEN 1\|AMBER . (root) . -i, --id=ID Unit . (id) Unit . DTD (SHOW DTD OPTIONS) --dtd DTD Unit. [ ] -f FILE, --filename=FILE stdout . . -u, --unit DTD Unit DTD . topo . "nvidia-smi topo -h" . . GPU NVML (affinity) (CPU) NUMA GPU Kepler . : GPU NVML . drain (drain) . drain GPU . "nvidia-smi drain -h" . . nvlink nvlink. "nvidia-smi nvlink -h" . clocks ( ). "nvidia-smi clocks --help" . vgpu GPU GRID. "nvidia-smi vgpu -h" . mig MIG . "nvidia-smi mig -h" . boost-slider (boost sliders) . "nvidia-smi boost-slider -h" . power-hint power hint . "nvidia-smi power-hint -h" . event-log . "nvidia-smi event-log -h" . cper CPER base64 . "nvidia-smi cper -h" . conf-compute (confidential compute) . "nvidia-smi conf-compute -h" . power-smoothing (power smoothing) . "nvidia-smi power-smoothing -h" . power-profiles (workload power profiles) . "nvidia-smi power-profiles -h" . encodersessions (Encoder Sessions). "nvidia-smi encodersessions -h" . (RETURN VALUE) . o 0 - o 2 - o 3 - o 4 - o 6 - o 8 - o 9 - NVIDIA o 10 - NVIDIA (interrupt) o 12 - NVML o 13 - NVML o 14 - infoROM (corrupted) o 15 - (bus) o 255 - -q . . (Timestamp) nvidia-smi. "Day-of-week Month Day HH:MM:SS Year" . (Driver Version) KMD Version . KMD (KMD Version) << >> (Kernel Mode Driver NVIDIA) . . CUDA (CUDA Version) CUDA UMD Version . CUDA UMD (CUDA UMD Version) << >> (User Mode Driver) CUDA. CUDA . CUDA (CUDA toolkit) . . (Attached GPUs) NVIDIA . (Product Name) . . . (Product Brand) () . . . (Product Architecture) . . . (Display Mode) . (Display Attached) ( ) . "Yes" . "No" . (Display Active) ( ). . "Enabled" . "Disabled" . (Persistence Mode) (persistence mode) . "Enabled" "Disabled" . NVIDIA X11 nvidia-smi . CUDA . CUDA. . (Addressing Mode) . "ATS" "HMM" "None" . "ATS" malloc (Address Translation Services) . (page tables) CPU GPU . "HMM" malloc CPU GPU . "None" ATS HMM . . MIG (MIG Mode) MIG Current MIG - NA/Enabled/Disabled Pending MIG - Enabled/Disabled MIG (MIG Device) MIG MIG : Index MIG . GPU Instance ID (GPU instance) MIG . Compute Instance ID (compute instance) . Device Attributes MIG. . Multiprocessor count SM ( - Streaming Multiprocessors) Copy Engine count (copy engines) Encoder count (video encoders) Decoder count (video decoders) OFA count OFA ( - Optical Flow Accelerators) JPG count / JPEG ECC ECC MIG. SRAM SRAM. FB FB MIG. GPU . FB. FB. FB. FB. BAR1 BAR1 MIG. GPU . BAR1. BAR1. BAR1. (accounting mode) GPU . "Enabled" "Disabled" . GPU . . 0 . --help-query-accounted-apps . (Accounting Mode Buffer Size) . . (Driver Model) Windows TCC WDDM MCDM . (-dm) (-fdm) . TCC TCC . WDDM . "N/A" . . '--no-driver-restart' . Current . "N/A" . Pending . "N/A" . (Serial Number) . . (GPU UUID) GPU . . (GPU PDI) (Per Device Identifier) GPU . GPU . (Minor Number) (minor number) NVIDIA GPU /dev/nvidia[minor number] . . VBIOS (VBIOS Version) (BIOS) GPU. (MultiGPU Board) GPU (multiGPU) . (Board ID) . GPU "MultiGPU" true GPU . (Board Part Number) (part number) GPU. (GPU Part Number) GPU. FRU (FRU Part Number) FRU GPU. (Platform Info) (Platform Information) (compute tray) . GPU . Chassis Serial Number GPU. Slot Number GPU ( ). Tray Index (tray index) GPU ( ). Host ID (node) GPU. Peer Type NVLink ( ). Module Id GPU . GPU Fabric GUID Fabric GPU. Inforom (Inforom Version) inforom GPU. inforom GPU . inforom . GPU inforom . Unknown Error Inforom . Image Version infoROM. VBIOS infoROM infoROM . OEM Object OEM. ECC Object ECC. Power Management Object . Inforom checksum validation Inforom ( "valid" "invalid" "N/A"). --query-gpu inforom.checksum_validation . (flush) (blackbox) inforom. Latest Timestamp BBX . Latest Duration BBX . Time Run BBX GPU. GPU (GPU Operation Mode) GOM GPU GPU . GOM . <> . <> . <> (double precision) . GOM (--gom) . Tesla M X GK110 Kepler . Quadro Tesla C . Low Double Precision All On GeForce Titan . Current GOM . Pending GOM (reboot) . GPU C2C C2C (GPU). GPU (Reset) . . Reset Required Drain and Reset Recommended GPU (GPU Recovery Action) . . : None Reset Reboot Drain P2P Drain and Reset None Reset - UCE HBM/SRAM (reset) . GPU 'nvidia-smi -r' GPU . Reboot - UVM . (reboot) . (node) . (warm reboot) ( AC/DC ) Drain P2P - (N/A) (peer-to-peer) . GPU peer-to-peer (persistence mode) UVM . ( ) peer-to-peer NVML_FI_DEV_GET_GPU_RECOVERY_ACTION . DRAIN_P2P (reset) . Drain and Reset - UCE HBM . GPU frame buffer MIG . GPU (checkpoint) . ( ) ( row remap). UCE . GPU . GSP. - . (PCI) PCI . . . Bus PCI (hex) Device PCI (hex) Domain PCI (hex) Base Classcode (Base classcode) (hex) Sub Classcode (Sub classcode) (hex) Device Id PCI (hex) Sub System Id PCI (hex) Bus Id PCI "domain:bus:device.function" (hex) (GPU Link information) PCIe Current . (GPU) . Max . GPU PCIe PCIe . (Bridge Chip) (Bridge Chip) . GPU "N/A" . Type . N/A . Firmware Version . N/A . (Replays Since Reset) (replays) PCIe . (Replay Number Rollovers) PCIe . (link retraining) . (Tx Throughput) GPU PCIe MB/s . Maxwell . (Rx Throughput) GPU PCIe MB/s . Maxwell . (Atomic Caps) PCIe / (outbound/inbound) . (Fan Speed) . . : () . . / . . (Performance State) GPU. P0 ( ) P12 ( ) . (Clocks Event Reasons) (clocks) . "Not Active" . Idle CUDA . GPU Idle () . Application Clocks Setting CUDA . GPU . nvidia-smi --applications-clocks= . SW Power Cap (SW Power Scaling) GPU . (SW power cap) nvidia-smi --power-limit= . HW Slowdown CUDA . (HW Slowdown) . HW Thermal Slowdown HW Power Brake . HW Thermal Slowdown (HW Thermal Slowdown) . HW Power Brake (External Power Brake Assertion) ( ). Sync Boost GPU nvidia-smi DCGM Sync boost . GPU GPU . (throttle) GPU . SW Thermal Slowdown (SW Thermal capping) GPU (Max Operating Temp) . Display Clock Setting CUDA . GPU (Display clocks) . Volta . . SW Power Capping (SW Power Scaling) GPU . Sync Boost Group GPU (sync boost group) . SW Thermal Slowdown (SW Thermal capping) GPU (Max Operating Temp) . HW Thermal Slowdown (HW Thermal Slowdown) . HW Power Braking External Power Brake Assertion ( ). (Sparse Operation Mode) sparse operation mode GPU . "Enabled" "Disabled". "N/A" . FB (FB Memory Usage) (FB) . ECC . ECC (parity) . GPU . GPU NUMA FB nvidia-smi . FB GPU NVIDIA . FB . FB . . . Total FB. Reserved FB. Used FB. Free FB. BAR1 (BAR1 Memory Usage) BAR1 FB ( ) CPU ( -- PCIe) . Total BAR1. Used BAR1. Free BAR1. (Compute Mode) GPU . "Default" . "Exclusive Process" (thread) . "Prohibited" ( ). "EXCLUSIVE_PROCESS" CUDA 4.0 . CUDA "EXCLUSIVE_THREAD" CUDA 4.0 . CUDA. (Utilization) GPU GPU . : GPU MIG encoder decoder jpeg ofa gpu memory . : ECC GPU Memory . ECC (ECC Memory Scrubbing) . GPU GPU . / . Memory () . / . Encoder GPU . nvmlDeviceGetEncoderUtilization() . Decoder GPU . nvmlDeviceGetDecoderUtilization() . JPEG JPEG GPU . nvmlDeviceGetJpgUtilization() . OFA OFA ( ) GPU . nvmlDeviceGetOfaUtilization() . (Encoder Stats) (FPS) ( ) . Active Sessions . Average FPS (FPS) . Average Latency . DRAM (DRAM Encryption Mode) DRAM . "Enabled" "Disabled" . DRAM . Inforom ECC . Current DRAM GPU . Pending DRAM GPU . ECC (ECC Mode) ECC . "Enabled" "Disabled" . ECC . Inforom ECC 1.0 . Current ECC GPU . Pending ECC GPU . ECC (ECC Errors) NVIDIA ECC . ECC . (Texture memory) . ( volatile aggregate) . ECC . . ECC . . . : Windows . Linux . Linux . persistence ( X11) Linux . . Volta Tesla Quadro ECC . . Inforom ECC 2.0 . ECC ECC 1.0 . Device Memory . Register File (Register File). L1 Cache L1. L2 Cache L2. Texture Memory (Parity) . Total . Device Memory Register File L1 Cache L2 Cache Texture Memory. Turing : SRAM Correctable SRAM. SRAM Uncorrectable SRAM. DRAM Correctable DRAM. DRAM Uncorrectable DRAM. +Ampere SRAM . SRAM (Parity) SEC-DED ( / ) . () SRAM . SRAM . SRAM Uncorrectable Parity SRAM (Parity) . SRAM Uncorrectable SEC-DED SRAM SEC-DED . Aggregate Uncorrectable SRAM Sources SRAM. SRAM L2 L2 . SRAM SM SM . SRAM Microcontroller (PMU/GSP ) . SRAM PCIE PCIE . SRAM Other . (Pending) GPU Recovery . Channel Repair Pending Channel . TPC Repair Pending TPC . Unrepairable Memory . NVIDIA GPU . ECC (single bit) ECC (double bit). NVIDIA . Double Bit ECC GPU ECC . Single Bit ECC GPU ECC . Pending GPU (blacklist) . . (Row Remapper) NVIDIA GPU . ECC ECC . NVIDIA . . Ampere+ . Correctable Error ECC . Inactive Correctable Error . Uncorrectable Error ECC . Inactive Uncorrectable Error . Pending . GPU (reset) . Remapping Failure Occurred . Bank Remap Availability Histogram . Maximum High Partial Low None . Maximum None . . . (Temperature) . (C) . . (module form factors) (passive) . . T.Limit: T.Limit . . T.Limit . nvidia-smi T.Limit . T.Limit 0 GPU (clock) . T.Limit T.Limit ( ) . GPU Current Temp GPU. (discrete) S-class. GPU Current T.Limit Temp GPU. GPU Shutdown Temp GPU . GPU Shutdown T.Limit Temp Specification T.Limit GPU . GPU T.Limit . GPU Slowdown Temp GPU . GPU Slowdown T.Limit Temp Specification T.Limit GPU . GPU T.Limit . GPU Max Operating Temp GPU . GPU Max Operating T.Limit Temp Specification T.Limit GPU . GPU Target Temperature Specification GPU . GPU . nvidia-smi -gtt --gpu-target-temp . . Memory Current Temp GPU. . Memory Max Operating T.Limit Temp Specification GPU . . (GPU Power Readings) GPU . GPU (power envelope) . . Average Power Draw . Ampere ( GA100) . Instantaneous Power Draw . GPU Ceiling Power Limit GPU GPU . o Current Power Limit - GPU nvidia- smi . o Requested Power Limit - GPU . o Default Power Limit - GPU . GPU Base Power GPU GPU ( CPU NVIDIA) . o Current Base Power - GPU nvidia- smi . o Requested Base Power - GPU . o Default Base Power - GPU . Min Power Limit . Max Power Limit . . GPU NVIDIA . (power envelope) . Hopper . Average Power Draw ( ) . Instantaneous Power Draw ( ) . Requested Power Limit ( ) . nvidia-smi . -pl --power- limit= -s/--scope=1 . Enforced Power Limit ( ) . . . Default Power Limit ( ) . (driver unload) . Min Power Limit ( ) ( ) . Max Power Limit ( ) ( ) . (GPU Memory Power Readings) GPU. Average Power Draw ( ) GPU . Instantaneous Power Draw ( ) GPU . (Power Smoothing) (Power Smoothing) . . Enabled () "Yes" "No" . Delayed Power Smoothing Supported ( ) Delayed Power Smoothing "Yes" "No" . Privilege Level ( ) . 0 1 2 . . Immediate Ramp Down ( ) "Enabled" "Disabled" . (ramp down hysteresis) ( ) ( ). Current TMP ( ) (Total Module Power) . Current TMP FLoor ( ) (Total Module Power floor) . Max % TMP Floor ( TMP) Percent TMP Floor . Min % TMP Floor ( TMP) Percent TMP Floor . HW Lifetime % Remaining ( ) . . Current Primary Power Floor ( ) . TMP Ceiling * (% TMP FLoor value) . Current Secondary Power Floor ( ) . GPU primary floor activation window multiplier . Min Primary Floor Activation Offset ( ) . (static) . Min Primary Floor Activation Point ( ) ( ) . 'secondary power floor + primary floor activation offset' 'min primary floor activation point' . . Window Multiplier ( ) (ms) (primary floor activation window multiplier primary floor target window multiplier). (static) . Number of Preset Profiles ( ) (Preset Profiles) . (power smoothing) . **% TMP Floor** TMP (TMP Ceiling) TMP (TMP floor) . TMP 1000 W % TMP floor 50% TMP 500 W . [Min % TMP Floor, Max % TMP Floor] . Ramp Up Rate (ramp up rate) mW/s . Ramp Down Rate (ramp down rate) mW/s . Ramp Down Hysteresis (ramp down hysteresis) (ms) . Secondary Power Floor . GPU . Primary Floor Activation Window Multiplier . 'X' . ( + ) . 0 . Primary Floor Target Window Multiplier . 'X' . . 0 (Floor percentage) . Primary Floor Activation Offset . . Active Preset Profile Number . (Admin Overrides) (Admin overrides) . . **% TMP Floor** % TMP Floor. [Min % TMP Floor, Max % TMP Floor] . Ramp Up Rate (ramp up rate) mW/s. Ramp Down Rate (ramp down rate) mW/s. Ramp Down Hysteresis (ms). Secondary Power Floor . GPU . Primary Floor Activation Window Multiplier . 'X' . ( + ) . 0 . Primary Floor Target Window Multiplier . 'X' . . 0 (Floor percentage) . Primary Floor Activation Offset . . State-Of-Charge Power Smoothing State-Of-Charge Power Smoothing (Enabled/Disabled). (Workload Power Profiles) GPU (Datacenter) . . Requested Profiles . Enforced Profiles . . EDPp . NVML nvidia-smi . (Clocks) GPU . (MHz) . (HW) (SW) . Graphics (shader). SM SM ( / Streaming Multiprocessor). Memory . Video ( + ). (Applications Clocks) Applications Clocks CUDA . -lmc/-lgc / -rmc/-rgc / . . [-ac --applications-clocks] . Graphics (shader). Memory . (Default Applications Clocks) . [-ac --applications-clocks] . [-rac --reset-applications-clocks] . Graphics (shader) . Memory . (Deferred Clocks) . Memory (Memory Clock) MHz GPU . . (Max Clocks) GPU . MHz . P0 ( ) . Graphics (shader). SM SM ( / Streaming Multiprocessor). Memory . Video ( + ). (Max Customer Boost Clocks) GPU . MHz . Graphics (shader). (Clock Policy) auto boost. Auto Boost auto boost GPU (On) (Off). (N/A) . auto boost GPU . auto boost GPU Current Application Clocks ( CUDA ). auto boost . CUDA . NVML ( NVML SDK ) CUDA CUDA_AUTO_BOOST . CUDA . Auto Boost Default auto boost (On) (Off). (N/A) . . : Auto Boost "Persistence Mode" . CUDA . GPU State (handshake) nvidia-fabricmanager ( GPU GPU fabric probe ) . : Completed In Progress Not Started Not supported Status GPU nvidia-fabricmanager. : NVML_SUCCESS . Clique ID (clique) GPU NVLink . (Clique ID) . (multi-node) NVLink . Cluster UUID (UUID) NVLink GPU . Cluster UUID NVLink . Health (Summary) - (Bandwidth) - NVLink (Route Recovery in progress) - NVLink (Route Unhealthy) - NVLink (Access Timeout Recovery) - NVLink (Incorrect Configuration) - (Partition Assigned) - NVLink (Processes) (Compute) (Graphics) (Other) . . Kepler . (context-based) GPU ( VMM APIs) . " " > GPU Index NVML . GPU Instance Index GPU MIG ( ) . Compute Instance Index MIG ( ) . PID (Process ID) Compute Graphics Other GPU . Type "C" (Compute) "G" (Graphics) "M" (MPS "Multi-Process Service") "O" (Other - RM GPU ) MPS "C+G" "M+C" . Process Name Compute MPS Compute Graphics Other . GPU Memory Usage GPU FB GPU (discrete) GPU (integrated) . WDDM KMD NVIDIA. (Device Monitoring) "nvidia-smi dmon" GPU ( ) . . . . Tesla GRID Quadro GeForce Kepler bare metal . (Power Usage) (Temperature) SM (Utilization) SM JPEG OFA . frame buffer bar1 / / ecc . "-" . . . . : MIG encoder decoder jpeg ofa gpu . : 1) > nvidia-smi dmon > ( GPU 0) . ^C . > 2) > nvidia-smi dmon -i > . ( GPU 0) . > 3) > nvidia-smi dmon -s > : > p - ( ) GPU/ ( ) > u - ( SM JPEG OFA ) > c - ( ) > v - ( ) ( ) > m - (Frame Buffer) Bar1 Confidential Compute ( ) > e - ECC ( ecc) PCIe Replay > t - Rx Tx PCIe (Maxwell ) > n - ( MMA Stall) ( Rubin ) >.PP ) > nvidia-smi dmon -c > . > ) > nvidia-smi dmon -d