sacct(1) Slurm Commands sacct(1) (NAME) sacct - Slurm (SYNOPSIS) sacct [OPTIONS...] (DESCRIPTION) Slurm AccountingStorageType Slurm . sacct Slurm . sacct (exitcodes) . --format= . . Job Launch . root sacct . root sacct (UID) . --allusers --user --uid . [days-]hours:minutes:seconds[.microseconds] . 'CPU' . : PrivateData slurmdbd.conf SlurmUser root AdminLevel=Admin . slurmdbd.conf . : Slurm . sacct Slurm . : sacct wait3() getrusage() . sacct 0 . getrusage (3) . (OPTIONS) -A, --accounts= . --array . . -L, --allclusters . sacct . -X, --allocations . : () . -a, --allusers root PrivateData jobs . . -x, --associations= assoc_list . assoc_list . . -B, --batch-script . 'NONE' . : AccountingStoreFlags=job_script . : () '-j' . -b, --brief JobID State ExitCode. -M, --clusters= cluster_list . cluster_list . 'all' . sacct . --local . -c, --completion . JobCompType slurm.conf none . ( ). -C, --constraints= / . 'and' 'or' . --delimiter= ASCII -p -P. '|' . -p -P . -D, --duplicates (job ID) SLUID . SLUID . (requeues) MaxJobId . Submit OriginalSLUID SLUID . -E, --endtime= . -s . --starttime --endtime (DEFAULT TIME WINDOW) ( ) . : HH:MM[:SS][AM|PM] MMDD[YY][-HH:MM[:SS]] MM.DD[.YY][-HH:MM[:SS]] MM/DD[/YY][-HH:MM[:SS]] YYYY-MM-DD[THH:MM[:SS]] today, midnight, noon, elevenses (11 AM), fika (3 PM), teatime (4 PM) now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] --env-vars (batch job) 'NONE' . : AccountingStoreFlags=job_env . : () '-j' . --expand-patterns StdOut StdErr StdIn . . "%t" "0" . --federation . -f, --file= sacct Slurm file . jobcomp/filetxt . -c . -F, --flags= . SchedSubmit SchedMain SchedBackfill StartReceived. SchedSubmit SchedMain SchedBackfill . -o, --format . ( "--helpformat" ). : format %NUMBER . format=name%30 name . %-30 . SACCT_FORMAT . : SACCT_FORMAT="jobid,user,account,cluster" -g, --gid=, --group= GID GROUP gid_list group_list ( ) . . . -h, --help . -e, --helpformat --format . : Account AdminComment AllocCPUS AllocNodes AllocTRES AssocID AveCPU AveCPUFreq AveDiskRead AveDiskWrite AvePages AveRSS AveVMSize BlockID Cluster Comment Constraints ConsumedEnergy ConsumedEnergyRaw Container CPUTime CPUTimeRAW DBIndex DerivedExitCode Elapsed ElapsedRaw Eligible End Exclusive ExitCode Extra FailedNode Flags GID Group JobID JobIDRaw JobName Layout Licenses MaxDiskRead MaxDiskReadNode MaxDiskReadTask MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask MaxPages MaxPagesNode MaxPagesTask MaxRSS MaxRSSNode MaxRSSTask MaxVMSize MaxVMSizeNode MaxVMSizeTask McsLabel MinCPU MinCPUNode MinCPUTask NCPUS NNodes NodeList NTasks OriginalSLUID OverSubscribe Partition Planned PlannedCPU PlannedCPURAW Priority QOS QOSRAW QOSREQ Reason ReqCPUFreq ReqCPUFreqGov ReqCPUFreqMax ReqCPUFreqMin ReqCPUS ReqMem ReqNodes ReqReservation ReqTRES Reservation ReservationId Restarts SegmentSize SLUID Start State StdErr StdIn StdOut Submit SubmitLine Suspended SystemComment SystemCPU Timelimit TimelimitRaw TotalCPU TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot UID User UserCPU WCKey WCKeyID WorkDir : Ave[RSS|VM]Size TRESUsageIn[Ave|Tot]. / (high watermark) . sstat / . : TRESUsage*Min* . : jobacct_gather . jobacct_gather/cgroup cgroup/v2 cgroups () 0 . "Job Accounting Fields" . -j, --jobs= job[.step] job[.step] . job[.step] . . : 'batch' (batch step) . sacct (Eligible time) (non-eligible) . : --state PD . --starttime --endtime DEFAULT TIME WINDOW ( ) . --json, --json=list, --json= JSON data_parser data_parser . . --local . ( ) . --federation . -l, --long : --format=jobid,jobidraw,jobname,partition,maxvmsize,maxvmsizenode, maxvmsizetask,avevmsize,maxrss,maxrssnode,maxrsstask,averss,maxpages, maxpagesnode,maxpagestask,avepages,mincpu,mincpunode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exitcode,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreqgov,reqmem,consumedenergy,maxdiskread,maxdiskreadnode,maxdiskreadtask, avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwritetask,avediskwrite, reqtres,alloctres,tresusageinave,tresusageinmax, tresusageinmaxn,tresusageinmaxt,tresusageinmin,tresusageinminn,tresusageinmint, tresusageintot,tresusageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusageoutave,tresusageouttot --name= () . -i, --nnodes= . -I, --ncpus= cpu . --noconvert ( 2048M 2G ). -N, --nodelist= () . node_list . : Slurm . (-S -E) . -n, --noheader . . -p, --parsable '|' '|' . --delimiter . -P, --parsable2 '|' '|' . --delimiter . -r, --partition . . -q, --qos qos . . -R, --reason= / (resources/priority) . -S, --starttime . 00:00:00 '-s' '-j' . '-s' 'now' . '-s' . '-j' (Unix Epoch 0) . --starttime --endtime DEFAULT TIME WINDOW ( ) . : HH:MM[:SS][AM|PM] MMDD[YY][-HH:MM[:SS]] MM.DD[.YY][-HH:MM[:SS]] MM/DD[/YY][-HH:MM[:SS]] YYYY-MM-DD[THH:MM[:SS]] today, midnight, noon, elevenses (11 AM), fika (3 PM), teatime (4 PM) now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] -s, --state= . --state . . JOB STATE CODES . . ( CA CANCELLED) ( ca CA ). : PENDING "EligibleTime" "Unknown" . "EligibleTime" "scontrol show job" . "--hold" "EligibleTime=Unknown" . : 'now' . -j . -j 'Epoch' . 'now' . --starttime --endtime DEFAULT TIME WINDOW ( ) . -K, --timelimit-max timelimit_min (timelimit) . . -k, --timelimit-min . timelimit_max . . -T, --truncate () . --starttime --starttime . --endtime . -u, --uid=, --user= UID . UID . --units=[KMGTP] . --noconvert . --usage . --use-local-uid UID sacct UID Slurm . Slurm UID . UID . -v, --verbose . -V, --version . -W, --wckeys= wckey wckey_list wckey . wckey_list . wckey . --whole-hetjob[=yes|no] --jobs het_job_id ( ) Slurm . . . 'yes' ( ) . 'no' () . --yaml, --yaml=list, --yaml= YAML data_parser data_parser . . (Job Accounting Fields) . Ave* Max* Min* . MaxRSS MaxRSSTask . ALL . Account . AdminComment SlurmUser root . AllocCPUs . NCPUS. AllocNodes / . 0 . AllocTres . / . . AccountingStorageTRES slurm.conf . : no_consume . AssocID . AveCPU ( + ) . AveCPUFreq (kHz). AveDiskRead . AveDiskWrite . AvePages . AveRSS . AveVMSize . BlockID ( Blue Gene). Cluster . Comment AccountingStoreFlags slurm.conf 'job_comment' . Comment sacctmgr modify job sjobexitmod . Constraints . ConsumedEnergy . (K,M,G,T,P) . : . ConsumedEnergyRaw . : . Container OCI Container Bundle . CPUTime ( * ) HH:MM:SS. CPUTimeRAW ( * ) -. DBIndex . DerivedExitCode ( srun). . DerivedExitCode sacctmgr modify job sjobexitmod . Elapsed . : [DD-[HH:]]MM:SS : DD hh mm ss ElapsedRaw . Eligible . End. End . YYYY-MM-DDTHH:MM:SS SLURM_TIME_FORMAT . Exclusive . NO . NODE CPU GRES . --exclusive . USER . --exclusive=user . MCS MCS . --exclusive=mcs . TOPO . --exclusive=topo . : Partition Global . ExitCode salloc exit() . . Extra AccountingStoreFlags slurm.conf 'job_extra' . Extra sacctmgr modify job . FailedNode (kill) . Flags . SchedSubmit SchedMain SchedBackfill. GID . Group . JobID . : JobID[.JobStep] : ArrayJobID_ArrayTaskID : HetJobID+HetJobOffset . . SLURM_BITSTR_LEN . JobIDRaw . JobID JobID[.JobStep] . JobName . slurm_accounting.log . jobname (_) . sacct . Layout . (rank) . MaxDiskRead . MaxDiskReadNode maxdiskread . MaxDiskReadTask maxdiskread . MaxDiskWrite . MaxDiskWriteNode maxdiskwrite . MaxDiskWriteTask maxdiskwrite . MaxPages (page faults) . MaxPagesNode maxpages . MaxPagesTask maxpages . MaxRSS (resident set size) . MaxRSSNode maxrss . MaxRSSTask maxrss . MaxVMSize . MaxVMSizeNode (maxvmsize) . MaxVMSizeTask (maxvmsize) . MCSLabel (MCS) . MCSPlugin slurm.conf . MinCPU ( + ) . MinCPUNode (mincpu) . MinCPUTask (mincpu) . NCPUS . AllocCPUS. NNodes . . NodeList / . NTasks . OriginalSLUID SLUID . . . . SLUID . OverSubscribe (oversubscribe) . NO . YES . --oversubscribe . OK . : . Partition . Planned (wall clock) . . Elapsed . PlannedCPU . Elapsed . PlannedCPURAW . . Priority Slurm. QOS (Quality of Service). QOSRAW (Quality of Service). QOSREQ (Quality of Services) . Reason Priority Resources . . ReqCPUFreq kHz. : . . ReqCPUFreqGov kHz. : . . ReqCPUFreqMax kHz. : . . ReqCPUFreqMin kHz. : . . ReqCPUS . ReqMem . (M G ). : . ReqNodes / . ReqTres (TRES). / . AccountingStorageTRES slurm.conf . ReqReservation . Reservation . ReservationId . Restarts / . SegmentSize . SLUID . SLUID . -D/--duplicates . SLUID . Start . End. State . JOB STATE CODES . ( UID CANCELLED ) "+" . "%NUMBER" . NOTE: RUNNING . SUSPENDED RUNNING . NOTE: RUNNING ( ) RUNNING . () RUNNING . StdErr "filename pattern" stderr . batch . StdIn "filename pattern" stdin . batch . StdOut "filename pattern" stdout . batch . Submit . End. NOTE: . -D/--duplicates . SubmitLine . Suspended . Elapsed . SystemComment . Slurm . SystemCPU . Elapsed . NOTE: TotalCPU . Timelimit . Elapsed : Partition_limit MaxTime ( pending ). DefaultTime . UNLIMITED . TimelimitRaw . . NOTE: TimeLimit . TotalCPU SystemCPU UserCPU . . Elapsed . NOTE: ( scancel ) TotalCPU . wait3 (getrusage) . (forks execs) . proctrack . TresUsageInAve Tres . NOTE: TresUsageInMaxTask -1 . TresUsageInMax Tres . NOTE: TresUsageInMaxTask -1 . TresUsageInMaxNode TRES . TresUsageInMaxTask TRES . TresUsageInMin Tres . NOTE: TresUsageInMinTask -1 . TresUsageInMinNode TRES . TresUsageInMinTask TRES . TresUsageInTot Tres . TresUsageOutAve Tres . NOTE: TresUsageOutMaxTask -1 . TresUsageOutMax Tres . NOTE: TresUsageOutMaxTask -1 . TresUsageOutMaxNode TRES . TresUsageOutMaxTask TRES . TresUsageOutMin Tres . TresUsageOutMinNode TRES . TresUsageOutMinTask TRES . TresUsageOutTot Tres . UID (UID) . User . UserCPU (CPU) . Elapsed . NOTE: TotalCPU . WCKey (Workload Characterization Key). . WCKeyID wckey. WorkDir . (JOB STATE CODES) sacct . . BF BOOT_FAIL ( ). CA CANCELLED . . CD COMPLETED . DL DEADLINE (deadline) . F FAILED . NF NODE_FAIL . OOM OUT_OF_MEMORY (Out Of Memory) . PD PENDING . PR PREEMPTED (Preemption) . R RUNNING . RQ REQUEUED . RS RESIZING . RV REVOKED (Sibling) . S SUSPENDED . TO TIMEOUT . (DEFAULT TIME WINDOW) --starttime --endtime sacct . ( ) : --jobs --state. --jobs / --state --starttime --endtime : --jobs --state: --starttime (Midnight) . --endtime (Now) . --jobs --state: --starttime Epoch 0 . --endtime (Now) . --jobs --state: --starttime (Now) . --endtime --starttime --starttime (Now) . --jobs --state: --starttime Epoch 0 . --endtime --starttime --starttime (Now) . NOTE: -v/--verbose . (PERFORMANCE) sacct slurmdbd . sacct Slurm slurmdbd slurmdbd . sacct Slurm slurmdbd . sacct . (ENVIRONMENT VARIABLES) sacct . . (: .) SACCT_FEDERATION --federation SACCT_FORMAT . --format SACCT_LOCAL --local SLURM_BITSTR_LEN . . . . SLURM_CONF Slurm. SLURM_DEBUG_FLAGS sacct . DebugFlags slurm.conf(5) . slurm.conf . SLURM_JSON JSON: compact JSON . pretty JSON (pretty) . SLURM_TIME_FORMAT . standard "year-month-dateThour:minute:second" . relative "hour:minute:second" . "hour:minute" "Tomorr" () "Ystday" () ( "Mon" "Tue" ) ( "25 Apr") . ( "6 Jun 2012"). . strftime() . "%a %T" ( "Mon 12:34:56"). SLURM_YAML YAML: compact YAML . pretty YAML (pretty) . (EXAMPLES) sacct : # sacct Jobid Jobname Partition Account AllocCPUS State ExitCode ---------- ---------- ---------- ---------- ---------- ---------- -------- 2 script01 srun acct1 1 RUNNING 0 3 script02 srun acct1 1 RUNNING 0 4 endscript srun acct1 1 RUNNING 0 4.0 srun acct1 1 COMPLETED 0 brief . # sacct --brief Jobid State ExitCode ---------- ---------- -------- 2 RUNNING 0 3 RUNNING 0 4 RUNNING 0 4.0 COMPLETED 0 # sacct --allocations Jobid Jobname Partition Account AllocCPUS State ExitCode ---------- ---------- ---------- ---------- ------- ---------- -------- 3 sja_init andy acct1 1 COMPLETED 0 4 sjaload andy acct1 2 COMPLETED 0 5 sja_scr1 andy acct1 1 COMPLETED 0 6 sja_scr2 andy acct1 18 COMPLETED 2 7 sja_scr3 andy acct1 18 COMPLETED 0 8 sja_scr5 andy acct1 2 COMPLETED 0 9 sja_scr7 andy acct1 90 COMPLETED 1 10 endscript andy acct1 186 COMPLETED 0 sacct . . # sacct --format=jobid,elapsed,ncpus,ntasks,state Jobid Elapsed Ncpus Ntasks State ---------- ---------- ---------- -------- ---------- 3 00:01:30 2 1 COMPLETED 3.0 00:01:30 2 1 COMPLETED 4 00:00:00 2 2 COMPLETED 4.0 00:00:01 2 2 COMPLETED 5 00:01:23 2 1 COMPLETED 5.0 00:01:31 2 1 COMPLETED -T (--truncate) -S (--starttime) -E (--endtime) . -T -S . -E . -T ( ) sacct : # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state JobID Start End State --------- --------------------- -------------------- ------------ 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED 3 2014-07-03T11:35:21 Unknown RUNNING 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED 5 2014-07-03T11:41:01 Unknown RUNNING -T (truncate) . . -T . # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state JobID Start End State --------- --------------------- -------------------- ------------ 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING : -s (--state) sacct . sacct --state . --state . # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state JobID Start End State ------------ ------------------- ------------------- ---------- 2955 11:15:12 11:20:12 COMPLETED 2956 11:20:13 11:25:13 COMPLETED --state=pending 11:20:00 11:21:13 PENDING . PENDING (State) COMPLETED . # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state JobID Start End State ------------ ------------------- ------------------- ---------- 2956 11:20:13 11:25:13 COMPLETED (COPYING) Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company L.P. Copyright (C) 2008-2010 Lawrence Livermore National Security. ( ). Copyright (C) 2010-2022 SchedMD LLC. Slurm . . Slurm << >> (GNU General Public License) ( ) . Slurm << >> << >>. . (FILES) /etc/slurm.conf . /var/log/slurm_accounting.log . (root) . (SEE ALSO) sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2) Slurm 26.05 Slurm Commands sacct(1)