sacctmgr(1) Slurm Commands sacctmgr(1) (NAME) sacctmgr - Slurm . (SYNOPSIS) sacctmgr [OPTIONS...] [COMMAND...] (DESCRIPTION) sacctmgr Slurm . slurmdbd ( Slurm) . . Slurm association (/) . user cluster partition account. user (login name) . cluster Slurm ClusterName slurm.conf . partition Slurm . account (job) . sacctmgr association / (commit) . NOTE: Slurm (lower case) . sacctmgr Slurm . (OPTIONS) -s, --associations show list (associations) . associations . -h, --help sacctmgr . help . -i, --immediate . --json, --json=list, --json= JSON data_parser data_parser . . . -n, --noheader . -p, --parsable '|' '|' . -P, --parsable2 '|' '|' . -Q, --quiet . quiet . -r, --readonly sacctmgr . readonly (interactive) . --yaml, --yaml=list, --yaml= YAML data_parser data_parser . . . -v, --verbose (verbose logging). verbose . -V , --version . version . (COMMANDS) add (entity). create . archive {dump|load} (flat file) . clear stats . create . add . delete where . remove . dump . clustername.cfg . help sacctmgr. list [] . SPECS . show . load . sacctmgr dump . sacctmgr archive load . modify where set . ping slurmdbd. reconfigure SlurmDBD ( ). remove where . delete . show [] . SPECS . list . shutdown . version sacctmgr. : . exit sacctmgr. quit . quiet . quit sacctmgr. exit . verbose (detailed logging). (time-stamps) . . !! . (ENTITIES) account (account) (job submit time) --account= . 'chemistry' 'physics' 'science' . . association : account cluster partition () user. list show . add modify delete user account cluster association . (limits) association association . partition association ( ). association partition . association partition . association association . cluster ClusterName slurm.conf . configuration list show . coordinator (coordinator) . association (cancel) (requeue) (reassign) . event (downed) (drained) . DRAINING . federation . job (job) : Derived Exit Code Comment AdminComment Extra SystemComment TRES WCKey. problem show list . qos (Quality of Service). reservation . resource . (software licenses) . RunawayJobs list show (orphaned) (runaway) . <<>> (fix) . start eligible submit completed . SlurmDBD . AdminLevel Operator . : sreport . . stats list show (server) . ave_time total_time . RPC . transaction . tres list show (Trackable RESources) . user . ( ) PreserveCaseUser SlurmDBD . wckeys (Workload Characterization Key). (orthogonal accounts). : (GrpJobs GrpTRES ) . (preempt) . : TRES "+=" "-=" 0 18446744073709551600 . "-=" TRES 0 TRES . DefaultQOS= (QOS) QosLevel . . -1 . Fairshare={|parent} Share={|parent} fairshare . parent . fairshare . fairshare Fairshare=parent (re-parent) . fairshare . -1 . GrpJobs= . -1 . GrpJobsAccrue= . -1 . GrpSubmit= GrpSubmitJobs= . -1 . GrpTRES=TRES[+|-]=[,TRES[+|-]=,...] TRES . TRES (TRES information) . GrpTRESMins=TRES[+|-]=[,TRES[+|-]=,...] -TRES . TRES (TRES information) . : (root association) . sacctmgr . : Priority Multifactor . PriorityDecayHalfLife PriorityUsageResetPeriod slurm.conf . (kill) . GrpTRESRunMins=TRES[+|-]=[,TRES[+|-]=,...] -TRES . . . TRES (TRES information) . GrpWall= (wall clock time) . GrpWall :
:: -
:: -
. . -1 . : sacctmgr . : Priority Multifactor . PriorityDecayHalfLife PriorityUsageResetPeriod slurm.conf . (kill) . MaxJobs= . . . -1 . MaxJobsAccrue= . . . -1 . MaxSubmit= MaxSubmitJobs= . . -1 . MaxTRES=TRES[+|-]=[,TRES[+|-]=,...] MaxTRESPJ=TRES[+|-]=[,TRES[+|-]=,...] MaxTRESPerJob=TRES[+|-]=[,TRES[+|-]=,...] TRES . . . TRES (TRES information) . MaxTRESMins=TRES[+|-]=[,TRES[+|-]=,...] MaxTRESMinsPJ=TRES[+|-]=[,TRES[+|-]=,...] MaxTRESMinsPerJob=TRES[+|-]=[,TRES[+|-]=,...] -TRES . . . TRES (TRES information) . MaxTRESPN=TRES[+|-]=[,TRES[+|-]=,...] MaxTRESPerNode=TRES[+|-]=[,TRES[+|-]=,...] TRES . . . TRES (TRES information) . MaxWall= MaxWallDurationPerJob= . . . MaxWall :
:: -
:: -
. . -1 . : . Priority priority/multifactor. . . -1 . QosLevel QOS . QOS 'sacctmgr list qos' . . QosLevel '' ( ) . += -= QOS QOS . :.IP = QosLevel . : (QOS) . QOS = QOS . += QosLevel . QOS . -= QosLevel . (EXAMPLES) . (SPECIFICATIONS FOR ACCOUNTS) sacctmgr . . Cluster= . . Description= . Flags=[,,...] : NoUsersAreCoords UsersAreCoords . UsersAreCoords (coordinator) . Name= . . Organization= . Parent= . root ( ) . RawUsage= . 0 () . - () . WithAssoc (associations) . WithCoord (coordinators) . WithDeleted . . WithDeleted . : WithAssoc . SPECIFICATIONS FOR ASSOCIATIONS . GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES . / (LIST/SHOW ACCOUNT FORMAT OPTIONS) format= . : Account,Description,Organization Account . Description . Flags . Organization . Coordinators . ( WithCoordinator .) : WithAssoc . . . LIST/SHOW ASSOCIATION FORMAT OPTIONS . (Associations) sacctmgr . () . Clusters=[,,...] (). Accounts=[,,...] (). Users=[,,...] (). Partitions=[,,...] (). : Partitions="" Partitions='' . . . : Partitions=\"\". : GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES . : OnlyDefaults . Tree (). WithDeleted . . WithDeleted . WithSubAccounts . account= . . WOLimits . "Cluster,Account,User,Partition" . WOPInfo ( ). WOPLimits . WOPLimits ( ). / (LIST/SHOW ASSOCIATION FORMAT OPTIONS) format= . Account . Cluster . DefaultQOS (QOS) QosLevel . Fairshare Share (fairshare) . parent . fairshare . fairshare Fairshare=parent (re-parent) . fairshare . Flags . GrpJobs . GrpJobsAccrue . GrpSubmit GrpSubmitJobs . GrpTRES TRES . GrpTRESMins TRES- . GrpTRESRunMins TRES- . . . GrpWall (wall clock) . ID . Lineage . MaxJobs . MaxJobsAccrue . MaxSubmit MaxSubmitJobs . MaxTRES MaxTRESPJ TRES . MaxTRESMins MaxTRESMinsPJ MaxTRESMinsPerJob TRES- . MaxTRESPN MaxTRESPerNode TRES . MaxWall MaxWallDurationPerJob (wall clock time) . ParentID (association ID) . ParentName . Partition . Priority priority/multifactor . Qos QOS . QosRaw QOS . User . WithRawQOSLevel QosLevel QOS '' () '+' '-' . QOS +/- ( sacctmgr modify ... set QosLevel=qos_name) . QOS +/- QOS / ( sacctmgr modify ... set QosLevel=[+-]qos_name). WOPLimits QOS . (SPECIFICATIONS FOR CLUSTERS) sacctmgr . . Classification= capability capacity capapacity. Features[+|-]= . (federated jobs) . += -= . '' ( ) . Federation= . . FedState= . : ACTIVE . INACTIVE . DRAIN . DRAIN+REMOVE . . Name= . ClusterName slurm.conf Slurm . RPC= RPC. WithDeleted . . WithDeleted . WithFed ( Federation,ID,Features,FedState). WOLimits . Cluster,ControlHost,ControlPort,RPC . : GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES . format= . Classification capability capacity capapacity. Cluster . ControlHost slurmctld IP . ControlPort slurmctld . Features ( ). Federation ( ). FedState ( ). FedStateRaw FedState. Flags . Cray External MultipleSlurmd . (External) (registration only) . slurmctld slurmdbd AccountingStorageExternalHost slurm.conf . slurmctld slurmdbd slurmdbd Slurm . ID . . NodeCount . NodeNames . RPC slurmctld RPC . TRES Trackable RESources ( Billing BB (Burst buffer) CPU Energy GRES License Memory Node) . : (root association) . LIST/SHOW ASSOCIATION FORMAT OPTIONS . (SPECIFICATIONS FOR COORDINATOR) sacctmgr . . Account=[,,...] . Names=[,,...] . : WithCoordinator list account list user . (SPECIFICATIONS FOR EVENTS) slurmdbd . . All_Clusters . All_Time . Clusters=[,,...] (). . CondFlags=[,,...] . : Open ( / down) . End= . now . : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] Event= . Cluster Node . . MaxCPUs= CPU . MinCPUs= CPU . Nodes=[,,...] . Reason=[,,...] . (space) . Start= . 00:00:00 States= . . : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] States=[,,...] (node event). Node . User=[,,...] . Node slurm (cluster event) . / (LIST/SHOW EVENT FORMAT OPTIONS) Event format= . : Cluster,NodeName,TimeStart,TimeEnd,State,Reason,User Cluster . ClusterNodes (hostlist) . Duration . End . Event . EventRaw . NodeName . . Reason . Start . State . StateRaw . TRES TRES . User . (SPECIFICATIONS FOR FEDERATION) sacctmgr . . Clusters[+|-]=[,,...] / / . ( clusters=) . : . Name= . Tree . WithDeleted . . WithDeleted . format= . : Federation,Cluster,Features,FedState Cluster . Features (features) . Federation . FedState . FedStateRaw (FedState). Index . (SPECIFICATIONS FOR INSTANCES) (instances) slurmdbd . . Clusters=[,,...] . . End= . now () . : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] Extra= . InstanceId= (Cloud instance ID). InstanceType= (Cloud instance type). Nodes=[,,...] . Start= . 00:00:00 . : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] / (LIST/SHOW INSTANCE FORMAT OPTIONS) format= . : Cluster,NodeName,Start,End,InstanceID,InstanceType,Extra Cluster . End . Extra . InstanceId . InstanceType . NodeName . Start . slurmdbd . . . AdminComment= . Slurm . '' ( ) . Comment= AccountingStoreFlags slurm.conf 'job_comment' . . '' ( ) . Cluster= . DerivedExitCode= (derived exit code) . . EndTime . YYYY-MM-DDTHH:MM:SS SLURM_TIME_FORMAT . Extra= (extra) AccountingStoreFlags slurm.conf 'job_extra' . . '' ( ) . JobID= . wckey . NewWCKey= wckey () . StartTime EndTime. SystemComment= BurstBufferPlugin . Slurm . '' ( ) . TRES= TRES () . : . User= . WCKey= wckey . AdminComment Comment DerivedExitCode Extra SystemComment WCKey . / (LIST/SHOW JOB FORMAT OPTIONS) sacct Slurm . (SPECIFICATIONS FOR QOS) QOS sacctmgr . QOS . : (GrpJobs GrpTRES ) . (preempt) . : TRES ( GrpTRES MaxTRES ) "[+|-]=" (association) . Description QOS. Slurm . Flags slurmctld . += -= . -1 . : DenyOnLimit QOS (stand-alone) 'Max' 'Min' QOS . (pend) . ( GrpTRES) 'Max' ( MaxTRESPerNode) . QOS Association . EnforceUsageThreshold QOS UsageThreshold QOS UsageThreshold Fairshare Usage Threshold (hold) . NoDecay GrpTRESMins GrpWall UsageRaw QOS PriorityDecayHalfLife PriorityUsageResetPeriod slurm.conf (decay) . QOS . QOS (association) . / QOS . GrpTRESMins GrpWall QOS RawUsage QOS 0 () QOS ( DenyOnLimit ) ( QOSGrp{TRES}MinutesLimit QOSGrpWallLimit {TRES} ). NoReserve backfill QOS backfill . QOS QOS (preempt) ( QOS "standby"). QOS QOS (starvation) . OverPartQOS QOS QOS (override ). PartitionMaxNodes QOS MaxNodes (override ). PartitionMinNodes QOS MinNodes (override ). PartitionTimeLimit QOS TimeLimit (override )..IP Relative QOS ( ). Relative QOS (controller) . QOS : 1. . 2. QOS QOS . 3. QOS . QOS : . RequiresReservation QOS (reservation) . QOS (preemptive) . UsageFactorSafe AccountingStorageEnforce Safe UsageFactor . GraceTime (preemption grace time) . QOS . QOS PreemptMode=CANCEL PreemptMode=REQUEUE . : GraceTime PreemptMode ( PreemptMode=REQUEUE ). Slurm GraceTime . GrpJobs QOS. -1 . GrpJobsAccrue (age priority) QOS . QOS QOS . -1 . GrpSubmit GrpSubmitJobs QOS. -1 . GrpTRES TRES QOS. TRES information . GrpTRESMins -TRES QOS . TRES information . : Priority Multifactor . PriorityDecayHalfLife PriorityUsageResetPeriod slurm.conf (decay) . (kill) QOS . GrpTRESRunMins -TRES QOS. . . TRES information . GrpWall (wall clock time) QOS. (kill) . GrpWall :
:: -
:: -
. . -1 . : Priority Multifactor . PriorityDecayHalfLife PriorityUsageResetPeriod slurm.conf (decay) . (kill) QOS . LimitFactor [Grp|Max]TRES (association) . LimitFactor GrpTRES QOS . -1 . : QOS QOS . MaxJobsAccruePA MaxJobsAccruePerAccount ( ) . QOS QOS . -1 . MaxJobsAccruePU MaxJobsAccruePerUser . QOS QOS . -1 . MaxJobsPA MaxJobsPerAccount . -1 . MaxJobsPU MaxJobsPerUser . -1 . MaxSubmitJobsPA MaxSubmitJobsPerAccount . -1 . MaxSubmitJobsPU MaxSubmitJobsPerUser . -1 . MaxTRES MaxTRESPJ MaxTRESPerJob TRES . TRES information . MaxTRESMinsPJ MaxTRESMinsPerJob TRES . TRES information . MaxTRESPA MaxTRESPerAccount TRES . TRES information . MaxTRESPN MaxTRESPerNode TRES . TRES information . MaxTRESPU MaxTRESPerUser TRES . TRES information . MaxTRESRunMinsPA MaxTRESRunMinsPerAccount TRES . . . TRES information . MaxTRESRunMinsPU MaxTRESRunMinsPerUser TRES . . . TRES information . MaxWall MaxWallDurationPerJob (wall clock) . MaxWall :
:: -
:: -
. . -1 . MinPrioThreshold . -1 . MinTRES MinTRESPerJob TRES QOS . (pend) . TRES information . Name QOS. . Preempt QOS QOS (preempt ). '' ( ) . NOTE: Priority QOS QOS Preempt QOS . PreemptExemptTime QOS . QOS PreemptExemptTime . -1 QOS . 0 QOS ( OverPartQOS ) / slurm.conf . PreemptMode QOS PreemptType preempt/qos . PreemptMode QOS PreemptMode QOS . . NOTE: backfill . backfill . backfill . backfill . NOTE: (heterogeneous job) . PreemptMode ( SUSPEND () REQUEUE CANCEL ()) PreemptMode . GraceTime . GANG . CLUSTER PreemptMode . "OFF" "" ( ) . QOS QOS Preempt QOS . CANCEL . REQUEUE ( ) . --requeue sbatch JobRequeue slurm.conf 1 . SUSPEND Gang . SUSPEND GANG . Slurm . PreemptType=preempt/qos () Gang ( time-slicing). ( ) . NOTE: GRES . GRES . PRIORITY . WITHIN qos . PreemptType=preempt/qos qos . Priority QOS priority/multifactor . . NOTE: Priority QOS QOS Preempt ..IP RawUsage= QOS . 0 () . - . UsageFactor TRES ( RawUsage TRESMins TRESRunMins). usagefactor TRESBillingUnit . usagefactor . . QOS QOS . UsageFactorSafe AccountingStorageEnforce Safe UsageFactor (kill) . UsageFactorSafe AccountingStorageEnforce Safe UsageFactor . UsageFactorSafe AccountingStorageEnforce Safe . AccountingStorageEnforce slurm.conf . . -1 . UsageThreshold (fairshare) (association) . . sshare . -1 . / QOS (LIST/SHOW QOS FORMAT OPTIONS) QOS format= . Description QOS. Flags slurmctld . GraceTime (preemption grace time) hh:mm:ss. GrpJobs QOS. GrpJobsAccrue (age priority) QOS . QOS QOS . GrpSubmit GrpSubmitJobs QOS. GrpTRES TRES QOS. GrpTRESMins TRES QOS . GrpTRESRunMins TRES QOS. GrpWall (wall clock time) QOS. ID (ID) QOS. LimitFactor [Grp|Max]TRES . MaxJobsAccruePA MaxJobsAccruePerAccount ( ) . QOS QOS . MaxJobsAccruePU MaxJobsAccruePerUser . QOS QOS . MaxJobsPA MaxJobsPerAccount . MaxJobsPU MaxJobsPerUser . MaxSubmitJobsPA MaxSubmitJobsPerAccount . MaxSubmitJobsPU MaxSubmitJobsPerUser . MaxTRES MaxTRESPJ MaxTRESPerJob TRES . MaxTRESMins MaxTRESMinsPJ MaxTRESMinsPerJob TRES . MaxTRESPA MaxTRESPerAccount TRES . MaxTRESPN MaxTRESPerNode TRES ..IP MaxTRESPU MaxTRESPerUser TRES . MaxTRESRunMinsPA MaxTRESRunMinsPerAccount TRES (account) . MaxTRESRunMinsPU MaxTRESRunMinsPerUser TRES . MaxWall MaxWallDurationPerJob (wall clock) . MinPrioThreshold . MinTRES TRES QOS . (pend) . Name QOS. Preempt QOS QOS (preempt ). PreemptExemptTime QOS (preemption) . PreemptMode (preempt) (gang scheduling) QOS PreemptType preempt/qos . PreemptMode . Priority QOS priority/multifactor. UsageFactor TRES ( RawUsage TRESMins TRESRunMins). UsageThreshold (fairshare) (association) . WithDeleted . QOS . QOS WithDeleted . (SPECIFICATIONS FOR RESERVATIONS) scontrol slurmdbd . . Clusters=[,,...] (). . End= . (now) . : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] ID= (ID) . Names= . Nodes=[,,...] . Start= . 00:00:00 . : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] (Reservation) format= . : Cluster,Name,TRES,Start,End,UnusedWall Associations (ID) . Cluster . End . Flags . ID (Reservation ID). Name . NodeNames . Start . TRES TRES . UnusedWall (Wall clock) . (CPU) . unused_wall . (SPECIFICATIONS FOR RESOURCE) sacctmgr . . LastConsumed= Count . Clusters= . . Count= . Descriptions= . Flags[-|+]= . : Absolute Allowed Allocated . : flags-=absolute . . SharedPool . Allowed 0 Count/100% (/ Absolute). Names= . slurmctld name@server slurm.conf . ( ) Parameters=PreserveCaseResource slurmdbd.conf . Allowed= / . Server= . 'slurmdb' . . ServerType= . FlexNext Publisher Flexlm Reprise License Manager RLM. . Type= . License . WithClusters / . . WithDeleted . . WithDeleted . : (Resource) Slurm . Resource format= . : Name,Server,Type,Count,LastConsumed,Allocated,ServerType,Flags Allocated (licenses) . LastConsumed Count . Cluster . Count . Description . Name . Server . ServerType . Type . RUNAWAYJOB (SPECIFICATIONS FOR RUNAWAYJOB) (jobs) slurmdbd . <> ( ) slurmdbd . () slurmdbd . <> <> : EndState= . : Completed Failed. LIST/SHOW RUNAWAYJOB (LIST/SHOW RUNAWAYJOB FORMAT OPTIONS) runaway job format= . : ID,Name,Partition,Cluster,State,TimeSubmit,TimeStart,TimeEnd Cluster . ID (job). Name . Partition . State . TimeEnd . TimeStart . TimeSubmit () . (associations) slurmdbd . . Accounts=[,,...] . Action= (action) . Actor= . Clusters=[,,...] . End= . now . Start= . epoch . End Start : HH:MM[:SS] [AM|PM] MMDD[YY] MM/DD[/YY] MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]] Users=[,,...] . WithAssoc (associations) . / (LIST/SHOW TRANSACTIONS FORMAT OPTIONS) format= . : Time,Action,Actor,Where,Info Action (Action) . Actor (Actor) . Info . TimeStamp . Where . : WithAssoc . LIST/SHOW ASSOCIATION FORMAT OPTIONS . (SPECIFICATIONS FOR USERS) sacctmgr . . (User) (Association) . . (User Association) . Cluster Account User Partition User Association . User Association . Account= . AdminLevel= . None Operator Admin. Cluster= . . DefaultAccount= (job) . DefaultWCKey= (Workload Characterization Key) . Name= . NewName= . Partition= (Partition). : Partitions SPECIFICATIONS FOR ASSOCIATIONS . RawUsage= (raw usage) . 0 () . - . WCKeys= (Workload Characterization Key). WithAssoc (associations) . WithCoord (coordinator) . WithDeleted . . WithDeleted . : WithAssoc . SPECIFICATIONS FOR ASSOCIATIONS . GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES . format= . : User,DefaultAccount,DefaultWCKey,AdminLevel AdminLevel . Coordinators (coordinator) . ( WithCoordinator .) DefaultAccount . DefaultWCKey wckey . User . : WithAssoc (association) . () . . LIST/SHOW ASSOCIATION FORMAT OPTIONS . / WCKey (LIST/SHOW WCKey) WCKey format= . : WCKey,Cluster,User Cluster WCKey. ID WCKey. User WCKey. WCKey (Workload Characterization Key). WithDeleted . WCKey WCKey . WCKey WithDeleted . / TRES (LIST/SHOW TRES) TRES format= . : Type,Name,ID ID . Name . TRES BB ( Burst buffer) GRES License . CPU Energy Memory Node . GRES Name GRES GPU. Type . BB ( Burst buffer) CPU Energy GRES License Memory Node. TRES (TRES information) (Trackable RESources TRES) QOS (GrpTRES* / MaxTRES*) . "TRES=value" . TRES Slurm ( cpu mem node) (generic resource) . sacctmgr show tres . TRES "GrpTRES=cpu=16,mem=32G" : (CPU) (GiB) . TRES TRES . "GrpTRES=gres/gpu=2" "GrpTRES=cpu=16,mem=32G,gres/gpu=2" . -1 TRES "GrpTRES=cpu=-1" TRES "GrpTRES=mem=32G,gres/gpu=2" . Slurm 26.05 TRES += -= TRES "GrpTRES=mem-=8G,gres/gpu+=2" "GrpTRES=mem=24G,gres/gpu=4" . : (Memory) TRES MiB . G T P . : Billing TRES TRESBillingWeights . billing TRES . Billing TRES scontrol show jobs sacct ..SH " (GLOBAL FORMAT OPTION)" format %NUMBER . format=name%30 30 name . -30 30 . (FLAT FILE DUMP AND LOAD) sacctmgr (association) Slurm (dump) (load) . . Slurm . 25.05 QOS . # () . : QOS Cluster Parent Account User. (entity value) . . Organization . sacctmgr dump/load Slurm (root) . sacctmgr load (root) ( ). dump . . . : sacctmgr dump tux file=tux.cfg Cluster= . File= . clustername.cfg . load . . : sacctmgr load file=tux.cfg clean . . QOS . . 'account' 'qos' 'user' . qos clean=account,qos,user . Cluster= . File= . (SPECIFICATIONS FOR FLAT FILE) . (parent) . 'root' . . / cluster : Cluster - cluster_name:MaxTRESPerJob=node=15 . : FairShare= fairshare . GrpJobs= . GrpJobsAccrue= (age priority) . GrpNodes= TRES . GrpTRES=node=# . GrpSubmitJobs= . GrpTRES= TRES . GrpTRESMins= TRES . GrpTRESRunMins= TRES . . . GrpWall= (wall clock time) . MaxJobs= . MaxTRESPerJob= TRES . MaxWallDurationPerJob= . QOS= ( sacctmgr). root . : Parent - root Account - cs:MaxTRESPerJob=node=5:MaxJobs=4:FairShare=399:MaxWallDurationPerJob=40:Description='Computer Science':Organization='LC' Parent - cs Account - test:MaxTRESPerJob=node=1:MaxJobs=1:FairShare=1:MaxWallDurationPerJob=1:Description='Test Account':Organization='Test' ":" . (Parent THAT HAS ALREADY BEEN CREATED) . Account : Description= . FairShare= ..IP GrpTRES= TRES (association) . GrpTRESMins= TRES- . GrpTRESRunMins= TRES- . . . GrpJobs= . GrpJobsAccrue= (age priority) . GrpNodes= TRES . GrpTRES=node=# . GrpSubmitJobs= . GrpWall= (wall clock time) . MaxJobs= . MaxNodesPerJob= . MaxWallDurationPerJob= (wall clock) . Organization= . QOS(=,+=,-=) (QOS) ( sacctmgr). Parent : Parent - test User - adam:MaxTRESPerJob=node:2:MaxJobs=3:FairShare=1:MaxWallDurationPerJob=1:AdminLevel=Operator:Coordinator='test' : AdminLevel= (Administrator Operator) . Coordinator= (coordinator) . DefaultAccount= . FairShare= . MaxJobs= . MaxTRESPerJob= TRES . MaxWallDurationPerJob= (wall clock) . QOS(=,+=,-=) (QOS) ( sacctmgr). sacctmgr (flatfile) . slurmdbd sacctmgr . slurmdbd "man slurmdbd" . (rolled up) . . archive dump . (purge) slurmdbd.conf . . sacctmgr slurmdbd.conf . Directory= . Events . . Jobs . . JobScript . Jobs . JobEnv . Jobs . PurgeEventAfter= . () . "12hours" . PurgeJobAfter= ( ). "PurgeUsageAfter" . () . "12hours" . NOTE: (PurgeJobScriptAfter PurgeJobEnvAfter) . PurgeJobScriptAfter= ( ). () . "12hours" . PurgeJobEnvAfter= ( ). () . "12hours" . PurgeResvAfter= ( ). "PurgeUsageAfter" . () . "12hours" . PurgeStepAfter= (step) ( ). () . "12hours" . PurgeSuspendAfter= . () . "12hours" . PurgeTXNAfter= . () . "12hours" . PurgeUsageAfter= (Cluster Association QOS WCKey) ( ). sreport(1) . () . "12hours" . Reservations . . Script= . Steps . . Suspend . . TXN . . Usage . . . - . (purge) ArchiveDir ( ArchiveDir slurmdbd.conf ) . Slurm archive load . File= . slurmdbd . Insert= SQL . sql . (PERFORMANCE) sacctmgr (RPC) slurmdbd . sacctmgr Slurm slurmdbd slurmdbd (DoS) . sacctmgr Slurm slurmdbd . sacctmgr . (ENVIRONMENT VARIABLES) sacctmgr . . (: .) SLURM_CONF Slurm. SLURM_DEBUG_FLAGS sacctmgr . DebugFlags slurm.conf(5) . slurm.conf . SLURM_JSON JSON: compact JSON . pretty JSON . SLURM_YAML YAML: compact YAML . pretty YAML . (EXAMPLES) : (accounting associations) . . $ sacctmgr create cluster tux $ sacctmgr create account name=science fairshare=50 $ sacctmgr create account name=chemistry parent=science fairshare=30 $ sacctmgr create account name=physics parent=science fairshare=20 $ sacctmgr create user name=adam cluster=tux account=physics fairshare=10 $ sacctmgr delete user name=adam cluster=tux account=physics $ sacctmgr delete user name=adam cluster=tux account=science partition=\"\" $ sacctmgr delete account name=physics cluster=tux $ sacctmgr modify user where name=adam cluster=tux account=physics set maxjobs=2 maxwall=30:00 $ sacctmgr add user brian account=chemistry $ sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd $ sacctmgr list transactions Action="Add Users" Start=11/03-10:30:00 format=Where,Time $ sacctmgr dump cluster=tux file=tux_data_file $ sacctmgr load tux_data_file . . . 'set' 'where' . set . 'set' 'where' . : $ sacctmgr modify user name=adam set fairshare=10 cluster=tux modify user adam set fairshare=10 and cluster=tux . : $ sacctmgr modify user name=adam cluster=tux set fairshare=10 $ sacctmgr modify user name=adam set fairshare=10 where cluster=tux qos '=' qos . '+=' '-=' qos . qos normal,standby qos+=expedite . qos expedite sacctmgr . $ sacctmgr modify user name=adam set qos+=expedite $ sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite QOS . QOS : $ sacctmgr show qos format=name Name --------- normal expedite : $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS -------- ---------- -------------------- zebra root normal zebra root normal zebra g normal zebra g1 normal QOS expedite g1 . += QOS QOS : $ sacctmgr modify account name=g1 set qos+=expedite $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS -------- ---------- -------------------- zebra root normal zebra root normal zebra g normal zebra g1 expedite,normal QOS expedite QOS G . = expedite QOS G : $ sacctmgr modify account name=G set qos=expedite $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS -------- ---------- -------------------- zebra root normal zebra root normal zebra g expedite zebra g1 expedite,normal G QOS expedite QOS normal . $ sacctmgr add account banana parent=G $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS -------- ---------- -------------------- zebra root normal zebra root normal zebra g expedite zebra banana expedite zebra g1 expedite,normal (TRES): $ sacctmgr show tres Type Name ID ---------- ----------------- -------- cpu 1 mem 2 energy 3 node 4 billing 5 gres gpu:tesla 1001 license vcs 1002 bb cray 1003 (C) 2008-2010 Lawrence Livermore National Security. Lawrence Livermore National Laboratory (.. DISCLAIMER). (C) 2010-2022 SchedMD LLC. Slurm . . Slurm (GNU General Public License) ( ) / . Slurm . . (SEE ALSO) slurm.conf(5), slurmdbd(8) Slurm 26.05 Slurm Commands sacctmgr(1)