1sbatch(1) Slurm Commands sbatch(1)
2
6 sbatch - Submit a batch script to Slurm.
7
10 sbatch [OPTIONS(0)...] [ : [OPTIONS(N)...]] script(0) [args(0)...]
11 Option(s) define multiple jobs in a co-scheduled heterogeneous job.
13 For more details about heterogeneous jobs see the document
14 https://slurm.schedmd.com/heterogeneous_jobs.html
15
18 sbatch submits a batch script to Slurm. The batch script may be given
19 to sbatch through a file name on the command line, or if no file name
20 is specified, sbatch will read in a script from standard input. The
21 batch script may contain options preceded with "#SBATCH" before any
22 executable commands in the script. sbatch will stop processing further
23 #SBATCH directives once the first non-comment non-whitespace line has
24 been reached in the script.
25 sbatch exits immediately after the script is successfully transferred
27 to the Slurm controller and assigned a Slurm job ID. The batch script
28 is not necessarily granted resources immediately, it may sit in the
29 queue of pending jobs for some time before its required resources
30 become available.
31 By default both standard output and standard error are directed to a
33 file of the name "slurm-%j.out", where the "%j" is replaced with the
34 job allocation number. The file will be generated on the first node of
35 the job allocation. Other than the batch script itself, Slurm does no
36 movement of user files.
37 When the job allocation is finally granted for the batch script, Slurm
39 runs a single copy of the batch script on the first node in the set of
40 allocated nodes.
41 The following document describes the influence of various options on
43 the allocation of cpus to jobs and tasks.
44 https://slurm.schedmd.com/cpu_management.html
45
48 sbatch will return 0 on success or error code on failure.
49
52 The batch script is resolved in the following order:
53 1. If script starts with ".", then path is constructed as: current
55 working directory / script
56 2. If script starts with a "/", then path is considered absolute.
58 3. If script is in current working directory.
60 4. If script can be resolved through PATH. See path_resolution(7).
62 Current working directory is the calling process working directory
64 unless the --chdir argument is passed, which will override the current
65 working directory.
66
69 -a, --array=<indexes>
70 Submit a job array, multiple jobs to be executed with identical
71 parameters. The indexes specification identifies what array
72 index values should be used. Multiple values may be specified
73 using a comma separated list and/or a range of values with a "-"
74 separator. For example, "--array=0-15" or "--array=0,6,16-32".
75 A step function can also be specified with a suffix containing a
76 colon and number. For example, "--array=0-15:4" is equivalent to
77 "--array=0,4,8,12". A maximum number of simultaneously running
78 tasks from the job array may be specified using a "%" separator.
79 For example "--array=0-15%4" will limit the number of simultane‐
80 ously running tasks from this job array to 4. The minimum index
81 value is 0. the maximum value is one less than the configura‐
82 tion parameter MaxArraySize. NOTE: currently, federated job
83 arrays only run on the local cluster.
84 -A, --account=<account>
87 Charge resources used by this job to specified account. The
88 account is an arbitrary string. The account name may be changed
89 after job submission using the scontrol command.
90 --acctg-freq
93 Define the job accounting and profiling sampling intervals.
94 This can be used to override the JobAcctGatherFrequency parame‐
95 ter in Slurm's configuration file, slurm.conf. The supported
96 format is as follows:
97 --acctg-freq=<datatype>=<interval>
99 where <datatype>=<interval> specifies the task sam‐
100 pling interval for the jobacct_gather plugin or a
101 sampling interval for a profiling type by the
102 acct_gather_profile plugin. Multiple, comma-sepa‐
103 rated <datatype>=<interval> intervals may be speci‐
104 fied. Supported datatypes are as follows:
105 task=<interval>
107 where <interval> is the task sampling inter‐
108 val in seconds for the jobacct_gather plugins
109 and for task profiling by the
110 acct_gather_profile plugin. NOTE: This fre‐
111 quency is used to monitor memory usage. If
112 memory limits are enforced the highest fre‐
113 quency a user can request is what is config‐
114 ured in the slurm.conf file. They can not
115 turn it off (=0) either.
116 energy=<interval>
118 where <interval> is the sampling interval in
119 seconds for energy profiling using the
120 acct_gather_energy plugin
121 network=<interval>
123 where <interval> is the sampling interval in
124 seconds for infiniband profiling using the
125 acct_gather_infiniband plugin.
126 filesystem=<interval>
128 where <interval> is the sampling interval in
129 seconds for filesystem profiling using the
130 acct_gather_filesystem plugin.
131 The default value for the task sampling
133 interval is 30 seconds.
134 The default value for all other intervals is 0. An interval of
135 0 disables sampling of the specified type. If the task sampling
136 interval is 0, accounting information is collected only at job
137 termination (reducing Slurm interference with the job).
138 Smaller (non-zero) values have a greater impact upon job perfor‐
139 mance, but a value of 30 seconds is not likely to be noticeable
140 for applications having less than 10,000 tasks.
141 -B --extra-node-info=<sockets[:cores[:threads]]>
144 Restrict node selection to nodes with at least the specified
145 number of sockets, cores per socket and/or threads per core.
146 NOTE: These options do not specify the resource allocation size.
147 Each value specified is considered a minimum. An asterisk (*)
148 can be used as a placeholder indicating that all available
149 resources of that type are to be utilized. Values can also be
150 specified as min-max. The individual levels can also be speci‐
151 fied in separate options if desired:
152 --sockets-per-node=<sockets>
153 --cores-per-socket=<cores>
154 --threads-per-core=<threads>
155 If task/affinity plugin is enabled, then specifying an alloca‐
156 tion in this manner also results in subsequently launched tasks
157 being bound to threads if the -B option specifies a thread
158 count, otherwise an option of cores if a core count is speci‐
159 fied, otherwise an option of sockets. If SelectType is config‐
160 ured to select/cons_res, it must have a parameter of CR_Core,
161 CR_Core_Memory, CR_Socket, or CR_Socket_Memory for this option
162 to be honored. If not specified, the scontrol show job will
163 display 'ReqS:C:T=*:*:*'. This option applies to job alloca‐
164 tions.
165 --batch=<list>
168 Nodes can have features assigned to them by the Slurm adminis‐
169 trator. Users can specify which of these features are required
170 by their batch script using this options. For example a job's
171 allocation may include both Intel Haswell and KNL nodes with
172 features "haswell" and "knl" respectively. On such a configura‐
173 tion the batch script would normally benefit by executing on a
174 faster Haswell node. This would be specified using the option
175 "--batch=haswell". The specification can include AND and OR
176 operators using the ampersand and vertical bar separators. For
177 example: "--batch=haswell|broadwell" or
178 "--batch=haswell|big_memory". The --batch argument must be a
179 subset of the job's --constraint=<list> argument (i.e. the job
180 can not request only KNL nodes, but require the script to exe‐
181 cute on a Haswell node). If the request can not be satisfied
182 from the resources allocated to the job, the batch script will
183 execute on the first node of the job allocation.
184 --bb=<spec>
187 Burst buffer specification. The form of the specification is
188 system dependent. Note the burst buffer may not be accessible
189 from a login node, but require that salloc spawn a shell on one
190 of it's allocated compute nodes. See the description of Sal‐
191 locDefaultCommand in the slurm.conf man page for more informa‐
192 tion about how to spawn a remote shell.
193 --bbf=<file_name>
196 Path of file containing burst buffer specification. The form of
197 the specification is system dependent. These burst buffer
198 directives will be inserted into the submitted batch script.
199 -b, --begin=<time>
202 Submit the batch script to the Slurm controller immediately,
203 like normal, but tell the controller to defer the allocation of
204 the job until the specified time.
205 Time may be of the form HH:MM:SS to run a job at a specific time
207 of day (seconds are optional). (If that time is already past,
208 the next day is assumed.) You may also specify midnight, noon,
209 fika (3 PM) or teatime (4 PM) and you can have a time-of-day
210 suffixed with AM or PM for running in the morning or the
211 evening. You can also say what day the job will be run, by
212 specifying a date of the form MMDDYY or MM/DD/YY YYYY-MM-DD.
213 Combine date and time using the following format
214 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
215 count time-units, where the time-units can be seconds (default),
216 minutes, hours, days, or weeks and you can tell Slurm to run the
217 job today with the keyword today and to run the job tomorrow
218 with the keyword tomorrow. The value may be changed after job
219 submission using the scontrol command. For example:
220 --begin=16:00
221 --begin=now+1hour
222 --begin=now+60 (seconds by default)
223 --begin=2010-01-20T12:34:00
224 Notes on date/time specifications:
227 - Although the 'seconds' field of the HH:MM:SS time specifica‐
228 tion is allowed by the code, note that the poll time of the
229 Slurm scheduler is not precise enough to guarantee dispatch of
230 the job on the exact second. The job will be eligible to start
231 on the next poll following the specified time. The exact poll
232 interval depends on the Slurm scheduler (e.g., 60 seconds with
233 the default sched/builtin).
234 - If no time (HH:MM:SS) is specified, the default is
235 (00:00:00).
236 - If a date is specified without a year (e.g., MM/DD) then the
237 current year is assumed, unless the combination of MM/DD and
238 HH:MM:SS has already passed for that year, in which case the
239 next year is used.
240 --checkpoint=<time>
243 Specifies the interval between creating checkpoints of the job
244 step. By default, the job step will have no checkpoints cre‐
245 ated. Acceptable time formats include "minutes", "minutes:sec‐
246 onds", "hours:minutes:seconds", "days-hours", "days-hours:min‐
247 utes" and "days-hours:minutes:seconds".
248 --cluster-constraint=[!]<list>
251 Specifies features that a federated cluster must have to have a
252 sibling job submitted to it. Slurm will attempt to submit a sib‐
253 ling job to a cluster if it has at least one of the specified
254 features. If the "!" option is included, Slurm will attempt to
255 submit a sibling job to a cluster that has none of the specified
256 features.
257 --comment=<string>
260 An arbitrary comment enclosed in double quotes if using spaces
261 or some special characters.
262 -C, --constraint=<list>
265 Nodes can have features assigned to them by the Slurm adminis‐
266 trator. Users can specify which of these features are required
267 by their job using the constraint option. Only nodes having
268 features matching the job constraints will be used to satisfy
269 the request. Multiple constraints may be specified with AND,
270 OR, matching OR, resource counts, etc. (some operators are not
271 supported on all system types). Supported constraint options
272 include:
273 Single Name
275 Only nodes which have the specified feature will be used.
276 For example, --constraint="intel"
277 Node Count
279 A request can specify the number of nodes needed with
280 some feature by appending an asterisk and count after the
281 feature name. For example "--nodes=16 --con‐
282 straint=graphics*4 ..." indicates that the job requires
283 16 nodes and that at least four of those nodes must have
284 the feature "graphics."
285 AND If only nodes with all of specified features will be
287 used. The ampersand is used for an AND operator. For
288 example, --constraint="intel&gpu"
289 OR If only nodes with at least one of specified features
291 will be used. The vertical bar is used for an OR opera‐
292 tor. For example, --constraint="intel|amd"
293 Matching OR
295 If only one of a set of possible options should be used
296 for all allocated nodes, then use the OR operator and
297 enclose the options within square brackets. For example:
298 "--constraint=[rack1|rack2|rack3|rack4]" might be used to
299 specify that all nodes must be allocated on a single rack
300 of the cluster, but any of those four racks can be used.
301 Multiple Counts
303 Specific counts of multiple resources may be specified by
304 using the AND operator and enclosing the options within
305 square brackets. For example: "--con‐
306 straint=[rack1*2&rack2*4]" might be used to specify that
307 two nodes must be allocated from nodes with the feature
308 of "rack1" and four nodes must be allocated from nodes
309 with the feature "rack2".
310 NOTE: This construct does not support multiple Intel KNL
312 NUMA or MCDRAM modes. For example, while "--con‐
313 straint=[(knl&quad)*2&(knl&hemi)*4]" is not supported,
314 "--constraint=[haswell*2&(knl&hemi)*4]" is supported.
315 Specification of multiple KNL modes requires the use of a
316 heterogeneous job.
317 Parenthesis
320 Parenthesis can be used to group like node features
321 together. For example "--con‐
322 straint=[(knl&snc4&flat)*4&haswell*1]" might be used to
323 specify that four nodes with the features "knl", "snc4"
324 and "flat" plus one node with the feature "haswell" are
325 required. All options within parenthesis should be
326 grouped with AND (e.g. "&") operands.
327 --contiguous
330 If set, then the allocated nodes must form a contiguous set.
331 Not honored with the topology/tree or topology/3d_torus plugins,
332 both of which can modify the node ordering.
333 --cores-per-socket=<cores>
336 Restrict node selection to nodes with at least the specified
337 number of cores per socket. See additional information under -B
338 option above when task/affinity plugin is enabled.
339 --cpu-freq =<p1[-p2[:p3]]>
342 Request that job steps initiated by srun commands inside this
344 sbatch script be run at some requested frequency if possible, on
345 the CPUs selected for the step on the compute node(s).
346 p1 can be [#### | low | medium | high | highm1] which will set
348 the frequency scaling_speed to the corresponding value, and set
349 the frequency scaling_governor to UserSpace. See below for defi‐
350 nition of the values.
351 p1 can be [Conservative | OnDemand | Performance | PowerSave]
353 which will set the scaling_governor to the corresponding value.
354 The governor has to be in the list set by the slurm.conf option
355 CpuFreqGovernors.
356 When p2 is present, p1 will be the minimum scaling frequency and
358 p2 will be the maximum scaling frequency.
359 p2 can be [#### | medium | high | highm1] p2 must be greater
361 than p1.
362 p3 can be [Conservative | OnDemand | Performance | PowerSave |
364 UserSpace] which will set the governor to the corresponding
365 value.
366 If p3 is UserSpace, the frequency scaling_speed will be set by a
368 power or energy aware scheduling strategy to a value between p1
369 and p2 that lets the job run within the site's power goal. The
370 job may be delayed if p1 is higher than a frequency that allows
371 the job to run within the goal.
372 If the current frequency is < min, it will be set to min. Like‐
374 wise, if the current frequency is > max, it will be set to max.
375 Acceptable values at present include:
377 #### frequency in kilohertz
379 Low the lowest available frequency
381 High the highest available frequency
383 HighM1 (high minus one) will select the next highest
385 available frequency
386 Medium attempts to set a frequency in the middle of the
388 available range
389 Conservative attempts to use the Conservative CPU governor
391 OnDemand attempts to use the OnDemand CPU governor (the
393 default value)
394 Performance attempts to use the Performance CPU governor
396 PowerSave attempts to use the PowerSave CPU governor
398 UserSpace attempts to use the UserSpace CPU governor
400 The following informational environment variable is set
403 in the job
404 step when --cpu-freq option is requested.
405 SLURM_CPU_FREQ_REQ
406 This environment variable can also be used to supply the value
408 for the CPU frequency request if it is set when the 'srun' com‐
409 mand is issued. The --cpu-freq on the command line will over‐
410 ride the environment variable value. The form on the environ‐
411 ment variable is the same as the command line. See the ENVIRON‐
412 MENT VARIABLES section for a description of the
413 SLURM_CPU_FREQ_REQ variable.
414 NOTE: This parameter is treated as a request, not a requirement.
416 If the job step's node does not support setting the CPU fre‐
417 quency, or the requested value is outside the bounds of the
418 legal frequencies, an error is logged, but the job step is
419 allowed to continue.
420 NOTE: Setting the frequency for just the CPUs of the job step
422 implies that the tasks are confined to those CPUs. If task con‐
423 finement (i.e., TaskPlugin=task/affinity or TaskPlu‐
424 gin=task/cgroup with the "ConstrainCores" option) is not config‐
425 ured, this parameter is ignored.
426 NOTE: When the step completes, the frequency and governor of
428 each selected CPU is reset to the previous values.
429 NOTE: When submitting jobs with the --cpu-freq option with lin‐
431 uxproc as the ProctrackType can cause jobs to run too quickly
432 before Accounting is able to poll for job information. As a
433 result not all of accounting information will be present.
434 --cpus-per-gpu=<ncpus>
437 Advise Slurm that ensuing job steps will require ncpus proces‐
438 sors per allocated GPU. Requires the --gpus option. Not com‐
439 patible with the --cpus-per-task option.
440 -c, --cpus-per-task=<ncpus>
443 Advise the Slurm controller that ensuing job steps will require
444 ncpus number of processors per task. Without this option, the
445 controller will just try to allocate one processor per task.
446 For instance, consider an application that has 4 tasks, each
448 requiring 3 processors. If our cluster is comprised of
449 quad-processors nodes and we simply ask for 12 processors, the
450 controller might give us only 3 nodes. However, by using the
451 --cpus-per-task=3 options, the controller knows that each task
452 requires 3 processors on the same node, and the controller will
453 grant an allocation of 4 nodes, one for each of the 4 tasks.
454 --deadline=<OPT>
457 remove the job if no ending is possible before this deadline
458 (start > (deadline - time[-min])). Default is no deadline.
459 Valid time formats are:
460 HH:MM[:SS] [AM|PM]
461 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
462 MM/DD[/YY]-HH:MM[:SS]
463 YYYY-MM-DD[THH:MM[:SS]]]
464 --delay-boot=<minutes>
467 Do not reboot nodes in order to satisfied this job's feature
468 specification if the job has been eligible to run for less than
469 this time period. If the job has waited for less than the spec‐
470 ified period, it will use only nodes which already have the
471 specified features. The argument is in units of minutes. A
472 default value may be set by a system administrator using the
473 delay_boot option of the SchedulerParameters configuration
474 parameter in the slurm.conf file, otherwise the default value is
475 zero (no delay).
476 -d, --dependency=<dependency_list>
479 Defer the start of this job until the specified dependencies
480 have been satisfied completed. <dependency_list> is of the form
481 <type:job_id[:job_id][,type:job_id[:job_id]]> or
482 <type:job_id[:job_id][?type:job_id[:job_id]]>. All dependencies
483 must be satisfied if the "," separator is used. Any dependency
484 may be satisfied if the "?" separator is used. Many jobs can
485 share the same dependency and these jobs may even belong to dif‐
486 ferent users. The value may be changed after job submission
487 using the scontrol command. Once a job dependency fails due to
488 the termination state of a preceding job, the dependent job will
489 never be run, even if the preceding job is requeued and has a
490 different termination state in a subsequent execution.
491 after:job_id[:jobid...]
493 This job can begin execution after the specified jobs
494 have begun execution.
495 afterany:job_id[:jobid...]
497 This job can begin execution after the specified jobs
498 have terminated.
499 afterburstbuffer:job_id[:jobid...]
501 This job can begin execution after the specified jobs
502 have terminated and any associated burst buffer stage out
503 operations have completed.
504 aftercorr:job_id[:jobid...]
506 A task of this job array can begin execution after the
507 corresponding task ID in the specified job has completed
508 successfully (ran to completion with an exit code of
509 zero).
510 afternotok:job_id[:jobid...]
512 This job can begin execution after the specified jobs
513 have terminated in some failed state (non-zero exit code,
514 node failure, timed out, etc).
515 afterok:job_id[:jobid...]
517 This job can begin execution after the specified jobs
518 have successfully executed (ran to completion with an
519 exit code of zero).
520 expand:job_id
522 Resources allocated to this job should be used to expand
523 the specified job. The job to expand must share the same
524 QOS (Quality of Service) and partition. Gang scheduling
525 of resources in the partition is also not supported.
526 singleton
528 This job can begin execution after any previously
529 launched jobs sharing the same job name and user have
530 terminated. In other words, only one job by that name
531 and owned by that user can be running or suspended at any
532 point in time.
533 -D, --chdir=<directory>
536 Set the working directory of the batch script to directory
537 before it is executed. The path can be specified as full path or
538 relative path to the directory where the command is executed.
539 -e, --error=<filename pattern>
542 Instruct Slurm to connect the batch script's standard error
543 directly to the file name specified in the "filename pattern".
544 By default both standard output and standard error are directed
545 to the same file. For job arrays, the default file name is
546 "slurm-%A_%a.out", "%A" is replaced by the job ID and "%a" with
547 the array index. For other jobs, the default file name is
548 "slurm-%j.out", where the "%j" is replaced by the job ID. See
549 the filename pattern section below for filename specification
550 options.
551 --exclusive[=user|mcs]
554 The job allocation can not share nodes with other running jobs
555 (or just other users with the "=user" option or with the "=mcs"
556 option). The default shared/exclusive behavior depends on sys‐
557 tem configuration and the partition's OverSubscribe option takes
558 precedence over the job's option.
559 --export=<environment variables [ALL] | NONE>
562 Identify which environment variables from the submission envi‐
563 ronment are propagated to the launched application. By default,
564 all are propagated. Multiple environment variable names should
565 be comma separated. Environment variable names may be specified
566 to propagate the current value (e.g. "--export=EDITOR") or spe‐
567 cific values may be exported (e.g. "--export=EDI‐
568 TOR=/bin/emacs"). In these two examples, the propagated envi‐
569 ronment will only contain the variable EDITOR, along with
570 SLURM_* environment variables. However, Slurm will then implic‐
571 itly attempt to load the user's environment on the node where
572 the script is being executed, as if --get-user-env was speci‐
573 fied. This will happen whenever NONE or environment variables
574 are specified. If one desires to add to the submission environ‐
575 ment instead of replacing it, have the argument include ALL
576 (e.g. "--export=ALL,EDITOR=/bin/emacs"). Make sure ALL is speci‐
577 fied first, since sbatch applies the environment from left to
578 right, overwriting as necessary. Environment variables propa‐
579 gated from the submission environment will always overwrite
580 environment variables found in the user environment on the node.
581 If one desires no environment variables be propagated from the
582 submitting machine, use the argument NONE. Regardless of this
583 setting, the appropriate SLURM_* task environment variables are
584 always exported to the environment. This option is particularly
585 important for jobs that are submitted on one cluster and execute
586 on a different cluster (e.g. with different paths). To avoid
587 steps inheriting environment export settings (e.g. NONE) from
588 sbatch command, the environment variable SLURM_EXPORT_ENV should
589 be set to ALL in the job script.
590 --export-file=<filename | fd>
593 If a number between 3 and OPEN_MAX is specified as the argument
594 to this option, a readable file descriptor will be assumed
595 (STDIN and STDOUT are not supported as valid arguments). Other‐
596 wise a filename is assumed. Export environment variables
597 defined in <filename> or read from <fd> to the job's execution
598 environment. The content is one or more environment variable
599 definitions of the form NAME=value, each separated by a null
600 character. This allows the use of special characters in envi‐
601 ronment definitions.
602 -F, --nodefile=<node file>
605 Much like --nodelist, but the list is contained in a file of
606 name node file. The node names of the list may also span multi‐
607 ple lines in the file. Duplicate node names in the file will
608 be ignored. The order of the node names in the list is not
609 important; the node names will be sorted by Slurm.
610 --get-user-env[=timeout][mode]
613 This option will tell sbatch to retrieve the login environment
614 variables for the user specified in the --uid option. The envi‐
615 ronment variables are retrieved by running something of this
616 sort "su - <username> -c /usr/bin/env" and parsing the output.
617 Be aware that any environment variables already set in sbatch's
618 environment will take precedence over any environment variables
619 in the user's login environment. Clear any environment variables
620 before calling sbatch that you do not want propagated to the
621 spawned program. The optional timeout value is in seconds.
622 Default value is 8 seconds. The optional mode value control the
623 "su" options. With a mode value of "S", "su" is executed with‐
624 out the "-" option. With a mode value of "L", "su" is executed
625 with the "-" option, replicating the login environment. If mode
626 not specified, the mode established at Slurm build time is used.
627 Example of use include "--get-user-env", "--get-user-env=10"
628 "--get-user-env=10L", and "--get-user-env=S". This option was
629 originally created for use by Moab.
630 --gid=<group>
633 If sbatch is run as root, and the --gid option is used, submit
634 the job with group's group access permissions. group may be the
635 group name or the numerical group ID.
636 -G, --gpus=[<type>:]<number>
639 Specify the total number of GPUs required for the job. An
640 optional GPU type specification can be supplied. For example
641 "--gpus=volta:3". Multiple options can be requested in a comma
642 separated list, for example: "--gpus=volta:3,kepler:1". See
643 also the --gpus-per-node, --gpus-per-socket and --gpus-per-task
644 options.
645 --gpu-bind=<type>
648 Bind tasks to specific GPUs. By default every spawned task can
649 access every GPU allocated to the job.
650 Supported type options:
652 closest Bind each task to the GPU(s) which are closest. In a
654 NUMA environment, each task may be bound to more than
655 one GPU (i.e. all GPUs in that NUMA environment).
656 map_gpu:<list>
658 Bind by setting GPU masks on tasks (or ranks) as spec‐
659 ified where <list> is
660 <gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs
661 are interpreted as decimal values unless they are pre‐
662 ceded with '0x' in which case they interpreted as
663 hexadecimal values. If the number of tasks (or ranks)
664 exceeds the number of elements in this list, elements
665 in the list will be reused as needed starting from the
666 beginning of the list. To simplify support for large
667 task counts, the lists may follow a map with an aster‐
668 isk and repetition count. For example
669 "map_gpu:0*4,1*4". Not supported unless the entire
670 node is allocated to the job.
671 mask_gpu:<list>
673 Bind by setting GPU masks on tasks (or ranks) as spec‐
674 ified where <list> is
675 <gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The
676 mapping is specified for a node and identical mapping
677 is applied to the tasks on every node (i.e. the lowest
678 task ID on each node is mapped to the first mask spec‐
679 ified in the list, etc.). GPU masks are always inter‐
680 preted as hexadecimal values but can be preceded with
681 an optional '0x'. Not supported unless the entire node
682 is allocated to the job. To simplify support for large
683 task counts, the lists may follow a map with an aster‐
684 isk and repetition count. For example
685 "mask_gpu:0x0f*4,0xf0*4". Not supported unless the
686 entire node is allocated to the job.
687 --gpu-freq=[<type]=value>[,<type=value>][,verbose]
690 Request that GPUs allocated to the job are configured with spe‐
691 cific frequency values. This option can be used to indepen‐
692 dently configure the GPU and its memory frequencies. After the
693 job is completed, the frequencies of all affected GPUs will be
694 reset to the highest possible values. In some cases, system
695 power caps may override the requested values. The field type
696 can be "memory". If type is not specified, the GPU frequency is
697 implied. The value field can either be "low", "medium", "high",
698 "highm1" or a numeric value in megahertz (MHz). If the speci‐
699 fied numeric value is not possible, a value as close as possible
700 will be used. See below for definition of the values. The ver‐
701 bose option causes current GPU frequency information to be
702 logged. Examples of use include "--gpu-freq=medium,memory=high"
703 and "--gpu-freq=450".
704 Supported value definitions:
706 low the lowest available frequency.
708 medium attempts to set a frequency in the middle of the
710 available range.
711 high the highest available frequency.
713 highm1 (high minus one) will select the next highest avail‐
715 able frequency.
716 --gpus-per-node=[<type>:]<number>
719 Specify the number of GPUs required for the job on each node
720 included in the job's resource allocation. An optional GPU type
721 specification can be supplied. For example
722 "--gpus-per-node=volta:3". Multiple options can be requested in
723 a comma separated list, for example:
724 "--gpus-per-node=volta:3,kepler:1". See also the --gpus,
725 --gpus-per-socket and --gpus-per-task options.
726 --gpus-per-socket=[<type>:]<number>
729 Specify the number of GPUs required for the job on each socket
730 included in the job's resource allocation. An optional GPU type
731 specification can be supplied. For example
732 "--gpus-per-socket=volta:3". Multiple options can be requested
733 in a comma separated list, for example:
734 "--gpus-per-socket=volta:3,kepler:1". Requires job to specify a
735 sockets per node count ( --sockets-per-node). See also the
736 --gpus, --gpus-per-node and --gpus-per-task options.
737 --gpus-per-task=[<type>:]<number>
740 Specify the number of GPUs required for the job on each task to
741 be spawned in the job's resource allocation. An optional GPU
742 type specification can be supplied. This option requires the
743 specification of a task count. For example
744 "--gpus-per-task=volta:1". Multiple options can be requested in
745 a comma separated list, for example:
746 "--gpus-per-task=volta:3,kepler:1". Requires job to specify a
747 task count (--nodes). See also the --gpus, --gpus-per-socket
748 and --gpus-per-node options.
749 --gres=<list>
752 Specifies a comma delimited list of generic consumable
753 resources. The format of each entry on the list is
754 "name[[:type]:count]". The name is that of the consumable
755 resource. The count is the number of those resources with a
756 default value of 1. The count can have a suffix of "k" or "K"
757 (multiple of 1024), "m" or "M" (multiple of 1024 x 1024), "g" or
758 "G" (multiple of 1024 x 1024 x 1024), "t" or "T" (multiple of
759 1024 x 1024 x 1024 x 1024), "p" or "P" (multiple of 1024 x 1024
760 x 1024 x 1024 x 1024). The specified resources will be allo‐
761 cated to the job on each node. The available generic consumable
762 resources is configurable by the system administrator. A list
763 of available generic consumable resources will be printed and
764 the command will exit if the option argument is "help". Exam‐
765 ples of use include "--gres=gpu:2,mic:1", "--gres=gpu:kepler:2",
766 and "--gres=help".
767 --gres-flags=<type>
770 Specify generic resource task binding options.
771 disable-binding
773 Disable filtering of CPUs with respect to generic
774 resource locality. This option is currently required to
775 use more CPUs than are bound to a GRES (i.e. if a GPU is
776 bound to the CPUs on one socket, but resources on more
777 than one socket are required to run the job). This
778 option may permit a job to be allocated resources sooner
779 than otherwise possible, but may result in lower job per‐
780 formance.
781 enforce-binding
783 The only CPUs available to the job will be those bound to
784 the selected GRES (i.e. the CPUs identified in the
785 gres.conf file will be strictly enforced). This option
786 may result in delayed initiation of a job. For example a
787 job requiring two GPUs and one CPU will be delayed until
788 both GPUs on a single socket are available rather than
789 using GPUs bound to separate sockets, however the appli‐
790 cation performance may be improved due to improved commu‐
791 nication speed. Requires the node to be configured with
792 more than one socket and resource filtering will be per‐
793 formed on a per-socket basis.
794 -H, --hold
797 Specify the job is to be submitted in a held state (priority of
798 zero). A held job can now be released using scontrol to reset
799 its priority (e.g. "scontrol release <job_id>").
800 -h, --help
803 Display help information and exit.
804 --hint=<type>
807 Bind tasks according to application hints.
808 compute_bound
810 Select settings for compute bound applications: use all
811 cores in each socket, one thread per core.
812 memory_bound
814 Select settings for memory bound applications: use only
815 one core in each socket, one thread per core.
816 [no]multithread
818 [don't] use extra threads with in-core multi-threading
819 which can benefit communication intensive applications.
820 Only supported with the task/affinity plugin.
821 help show this help message
823 --ignore-pbs
826 Ignore any "#PBS" options specified in the batch script.
827 -i, --input=<filename pattern>
830 Instruct Slurm to connect the batch script's standard input
831 directly to the file name specified in the "filename pattern".
832 By default, "/dev/null" is open on the batch script's standard
834 input and both standard output and standard error are directed
835 to a file of the name "slurm-%j.out", where the "%j" is replaced
836 with the job allocation number, as described below in the file‐
837 name pattern section.
838 -J, --job-name=<jobname>
841 Specify a name for the job allocation. The specified name will
842 appear along with the job id number when querying running jobs
843 on the system. The default is the name of the batch script, or
844 just "sbatch" if the script is read on sbatch's standard input.
845 -k, --no-kill [=off]
848 Do not automatically terminate a job if one of the nodes it has
849 been allocated fails. The user will assume the responsibilities
850 for fault-tolerance should a node fail. When there is a node
851 failure, any active job steps (usually MPI jobs) on that node
852 will almost certainly suffer a fatal error, but with --no-kill,
853 the job allocation will not be revoked so the user may launch
854 new job steps on the remaining nodes in their allocation.
855 Specify an optional argument of "off" disable the effect of the
857 SBATCH_NO_KILL environment variable.
858 By default Slurm terminates the entire job allocation if any
860 node fails in its range of allocated nodes.
861 --kill-on-invalid-dep=<yes|no>
864 If a job has an invalid dependency and it can never run this
865 parameter tells Slurm to terminate it or not. A terminated job
866 state will be JOB_CANCELLED. If this option is not specified
867 the system wide behavior applies. By default the job stays
868 pending with reason DependencyNeverSatisfied or if the
869 kill_invalid_depend is specified in slurm.conf the job is termi‐
870 nated.
871 -L, --licenses=<license>
874 Specification of licenses (or other resources available on all
875 nodes of the cluster) which must be allocated to this job.
876 License names can be followed by a colon and count (the default
877 count is one). Multiple license names should be comma separated
878 (e.g. "--licenses=foo:4,bar"). To submit jobs using remote
879 licenses, those served by the slurmdbd, specify the name of the
880 server providing the licenses. For example "--license=nas‐
881 tran@slurmdb:12".
882 -M, --clusters=<string>
885 Clusters to issue commands to. Multiple cluster names may be
886 comma separated. The job will be submitted to the one cluster
887 providing the earliest expected job initiation time. The default
888 value is the current cluster. A value of 'all' will query to run
889 on all clusters. Note the --export option to control environ‐
890 ment variables exported between clusters. Note that the Slur‐
891 mDBD must be up for this option to work properly.
892 -m, --distribution=
895 arbitrary|<block|cyclic|plane=<options>[:block|cyclic|fcyclic]>
896 Specify alternate distribution methods for remote processes. In
898 sbatch, this only sets environment variables that will be used
899 by subsequent srun requests. This option controls the assign‐
900 ment of tasks to the nodes on which resources have been allo‐
901 cated, and the distribution of those resources to tasks for
902 binding (task affinity). The first distribution method (before
903 the ":") controls the distribution of resources across nodes.
904 The optional second distribution method (after the ":") controls
905 the distribution of resources across sockets within a node.
906 Note that with select/cons_res, the number of cpus allocated on
907 each socket and node may be different. Refer to
908 https://slurm.schedmd.com/mc_support.html for more information
909 on resource allocation, assignment of tasks to nodes, and bind‐
910 ing of tasks to CPUs.
911 First distribution method:
913 block The block distribution method will distribute tasks to a
915 node such that consecutive tasks share a node. For exam‐
916 ple, consider an allocation of three nodes each with two
917 cpus. A four-task block distribution request will dis‐
918 tribute those tasks to the nodes with tasks one and two
919 on the first node, task three on the second node, and
920 task four on the third node. Block distribution is the
921 default behavior if the number of tasks exceeds the num‐
922 ber of allocated nodes.
923 cyclic The cyclic distribution method will distribute tasks to a
925 node such that consecutive tasks are distributed over
926 consecutive nodes (in a round-robin fashion). For exam‐
927 ple, consider an allocation of three nodes each with two
928 cpus. A four-task cyclic distribution request will dis‐
929 tribute those tasks to the nodes with tasks one and four
930 on the first node, task two on the second node, and task
931 three on the third node. Note that when SelectType is
932 select/cons_res, the same number of CPUs may not be allo‐
933 cated on each node. Task distribution will be round-robin
934 among all the nodes with CPUs yet to be assigned to
935 tasks. Cyclic distribution is the default behavior if
936 the number of tasks is no larger than the number of allo‐
937 cated nodes.
938 plane The tasks are distributed in blocks of a specified size.
940 The options include a number representing the size of the
941 task block. This is followed by an optional specifica‐
942 tion of the task distribution scheme within a block of
943 tasks and between the blocks of tasks. The number of
944 tasks distributed to each node is the same as for cyclic
945 distribution, but the taskids assigned to each node
946 depend on the plane size. For more details (including
947 examples and diagrams), please see
948 https://slurm.schedmd.com/mc_support.html
949 and
950 https://slurm.schedmd.com/dist_plane.html
951 arbitrary
953 The arbitrary method of distribution will allocate pro‐
954 cesses in-order as listed in file designated by the envi‐
955 ronment variable SLURM_HOSTFILE. If this variable is
956 listed it will override any other method specified. If
957 not set the method will default to block. Inside the
958 hostfile must contain at minimum the number of hosts
959 requested and be one per line or comma separated. If
960 specifying a task count (-n, --ntasks=<number>), your
961 tasks will be laid out on the nodes in the order of the
962 file.
963 NOTE: The arbitrary distribution option on a job alloca‐
964 tion only controls the nodes to be allocated to the job
965 and not the allocation of CPUs on those nodes. This
966 option is meant primarily to control a job step's task
967 layout in an existing job allocation for the srun com‐
968 mand.
969 Second distribution method:
972 block The block distribution method will distribute tasks to
974 sockets such that consecutive tasks share a socket.
975 cyclic The cyclic distribution method will distribute tasks to
977 sockets such that consecutive tasks are distributed over
978 consecutive sockets (in a round-robin fashion). Tasks
979 requiring more than one CPU will have all of those CPUs
980 allocated on a single socket if possible.
981 fcyclic
983 The fcyclic distribution method will distribute tasks to
984 sockets such that consecutive tasks are distributed over
985 consecutive sockets (in a round-robin fashion). Tasks
986 requiring more than one CPU will have each CPUs allocated
987 in a cyclic fashion across sockets.
988 --mail-type=<type>
991 Notify user by email when certain event types occur. Valid type
992 values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
993 BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buf‐
994 fer stage out and teardown completed), TIME_LIMIT, TIME_LIMIT_90
995 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80
996 percent of time limit), TIME_LIMIT_50 (reached 50 percent of
997 time limit) and ARRAY_TASKS (send emails for each array task).
998 Multiple type values may be specified in a comma separated list.
999 The user to be notified is indicated with --mail-user. Unless
1000 the ARRAY_TASKS option is specified, mail notifications on job
1001 BEGIN, END and FAIL apply to a job array as a whole rather than
1002 generating individual email messages for each task in the job
1003 array.
1004 --mail-user=<user>
1007 User to receive email notification of state changes as defined
1008 by --mail-type. The default value is the submitting user.
1009 --mcs-label=<mcs>
1012 Used only when the mcs/group plugin is enabled. This parameter
1013 is a group among the groups of the user. Default value is cal‐
1014 culated by the Plugin mcs if it's enabled.
1015 --mem=<size[units]>
1018 Specify the real memory required per node. Default units are
1019 megabytes unless the SchedulerParameters configuration parameter
1020 includes the "default_gbytes" option for gigabytes. Different
1021 units can be specified using the suffix [K|M|G|T]. Default
1022 value is DefMemPerNode and the maximum value is MaxMemPerNode.
1023 If configured, both parameters can be seen using the scontrol
1024 show config command. This parameter would generally be used if
1025 whole nodes are allocated to jobs (SelectType=select/linear).
1026 Also see --mem-per-cpu and --mem-per-gpu. The --mem,
1027 --mem-per-cpu and --mem-per-gpu options are mutually exclusive.
1028 If --mem, --mem-per-cpu or --mem-per-gpu are specified as com‐
1029 mand line arguments, then they will take precedence over the
1030 environment.
1031 NOTE: A memory size specification of zero is treated as a spe‐
1033 cial case and grants the job access to all of the memory on each
1034 node. If the job is allocated multiple nodes in a heterogeneous
1035 cluster, the memory limit on each node will be that of the node
1036 in the allocation with the smallest memory size (same limit will
1037 apply to every node in the job's allocation).
1038 NOTE: Enforcement of memory limits currently relies upon the
1040 task/cgroup plugin or enabling of accounting, which samples mem‐
1041 ory use on a periodic basis (data need not be stored, just col‐
1042 lected). In both cases memory use is based upon the job's Resi‐
1043 dent Set Size (RSS). A task may exceed the memory limit until
1044 the next periodic accounting sample.
1045 --mem-per-cpu=<size[units]>
1048 Minimum memory required per allocated CPU. Default units are
1049 megabytes unless the SchedulerParameters configuration parameter
1050 includes the "default_gbytes" option for gigabytes. Default
1051 value is DefMemPerCPU and the maximum value is MaxMemPerCPU (see
1052 exception below). If configured, both parameters can be seen
1053 using the scontrol show config command. Note that if the job's
1054 --mem-per-cpu value exceeds the configured MaxMemPerCPU, then
1055 the user's limit will be treated as a memory limit per task;
1056 --mem-per-cpu will be reduced to a value no larger than MaxMem‐
1057 PerCPU; --cpus-per-task will be set and the value of
1058 --cpus-per-task multiplied by the new --mem-per-cpu value will
1059 equal the original --mem-per-cpu value specified by the user.
1060 This parameter would generally be used if individual processors
1061 are allocated to jobs (SelectType=select/cons_res). If
1062 resources are allocated by the core, socket or whole nodes; the
1063 number of CPUs allocated to a job may be higher than the task
1064 count and the value of --mem-per-cpu should be adjusted accord‐
1065 ingly. Also see --mem and --mem-per-gpu. The --mem,
1066 --mem-per-cpu and --mem-per-gpu options are mutually exclusive.
1067 NOTE:If the final amount of memory requested by job (eg.: when
1069 --mem-per-cpu use with --exclusive option) can't be satisfied by
1070 any of nodes configured in the partition, the job will be
1071 rejected.
1072 --mem-per-gpu=<size[units]>
1075 Minimum memory required per allocated GPU. Default units are
1076 megabytes unless the SchedulerParameters configuration parameter
1077 includes the "default_gbytes" option for gigabytes. Different
1078 units can be specified using the suffix [K|M|G|T]. Default
1079 value is DefMemPerGPU and is available on both a global and per
1080 partition basis. If configured, the parameters can be seen
1081 using the scontrol show config and scontrol show partition com‐
1082 mands. Also see --mem. The --mem, --mem-per-cpu and
1083 --mem-per-gpu options are mutually exclusive.
1084 --mem-bind=[{quiet,verbose},]type
1087 Bind tasks to memory. Used only when the task/affinity plugin is
1088 enabled and the NUMA memory functions are available. Note that
1089 the resolution of CPU and memory binding may differ on some
1090 architectures. For example, CPU binding may be performed at the
1091 level of the cores within a processor while memory binding will
1092 be performed at the level of nodes, where the definition of
1093 "nodes" may differ from system to system. By default no memory
1094 binding is performed; any task using any CPU can use any memory.
1095 This option is typically used to ensure that each task is bound
1096 to the memory closest to it's assigned CPU. The use of any type
1097 other than "none" or "local" is not recommended. If you want
1098 greater control, try running a simple test code with the options
1099 "--cpu-bind=verbose,none --mem-bind=verbose,none" to determine
1100 the specific configuration.
1101 NOTE: To have Slurm always report on the selected memory binding
1103 for all commands executed in a shell, you can enable verbose
1104 mode by setting the SLURM_MEM_BIND environment variable value to
1105 "verbose".
1106 The following informational environment variables are set when
1108 --mem-bind is in use:
1109 SLURM_MEM_BIND_LIST
1111 SLURM_MEM_BIND_PREFER
1112 SLURM_MEM_BIND_SORT
1113 SLURM_MEM_BIND_TYPE
1114 SLURM_MEM_BIND_VERBOSE
1115 See the ENVIRONMENT VARIABLES section for a more detailed
1117 description of the individual SLURM_MEM_BIND* variables.
1118 Supported options include:
1120 help show this help message
1122 local Use memory local to the processor in use
1124 map_mem:<list>
1126 Bind by setting memory masks on tasks (or ranks) as spec‐
1127 ified where <list> is
1128 <numa_id_for_task_0>,<numa_id_for_task_1>,... The map‐
1129 ping is specified for a node and identical mapping is
1130 applied to the tasks on every node (i.e. the lowest task
1131 ID on each node is mapped to the first ID specified in
1132 the list, etc.). NUMA IDs are interpreted as decimal
1133 values unless they are preceded with '0x' in which case
1134 they interpreted as hexadecimal values. If the number of
1135 tasks (or ranks) exceeds the number of elements in this
1136 list, elements in the list will be reused as needed
1137 starting from the beginning of the list. To simplify
1138 support for large task counts, the lists may follow a map
1139 with an asterisk and repetition count For example
1140 "map_mem:0x0f*4,0xf0*4". Not supported unless the entire
1141 node is allocated to the job.
1142 mask_mem:<list>
1144 Bind by setting memory masks on tasks (or ranks) as spec‐
1145 ified where <list> is
1146 <numa_mask_for_task_0>,<numa_mask_for_task_1>,... The
1147 mapping is specified for a node and identical mapping is
1148 applied to the tasks on every node (i.e. the lowest task
1149 ID on each node is mapped to the first mask specified in
1150 the list, etc.). NUMA masks are always interpreted as
1151 hexadecimal values. Note that masks must be preceded
1152 with a '0x' if they don't begin with [0-9] so they are
1153 seen as numerical values. If the number of tasks (or
1154 ranks) exceeds the number of elements in this list, ele‐
1155 ments in the list will be reused as needed starting from
1156 the beginning of the list. To simplify support for large
1157 task counts, the lists may follow a mask with an asterisk
1158 and repetition count For example "mask_mem:0*4,1*4". Not
1159 supported unless the entire node is allocated to the job.
1160 no[ne] don't bind tasks to memory (default)
1162 p[refer]
1164 Prefer use of first specified NUMA node, but permit
1165 use of other available NUMA nodes.
1166 q[uiet]
1168 quietly bind before task runs (default)
1169 rank bind by task rank (not recommended)
1171 sort sort free cache pages (run zonesort on Intel KNL nodes)
1173 v[erbose]
1175 verbosely report binding before task runs
1176 --mincpus=<n>
1179 Specify a minimum number of logical cpus/processors per node.
1180 -N, --nodes=<minnodes[-maxnodes]>
1183 Request that a minimum of minnodes nodes be allocated to this
1184 job. A maximum node count may also be specified with maxnodes.
1185 If only one number is specified, this is used as both the mini‐
1186 mum and maximum node count. The partition's node limits super‐
1187 sede those of the job. If a job's node limits are outside of
1188 the range permitted for its associated partition, the job will
1189 be left in a PENDING state. This permits possible execution at
1190 a later time, when the partition limit is changed. If a job
1191 node limit exceeds the number of nodes configured in the parti‐
1192 tion, the job will be rejected. Note that the environment vari‐
1193 able SLURM_JOB_NODES will be set to the count of nodes actually
1194 allocated to the job. See the ENVIRONMENT VARIABLES section for
1195 more information. If -N is not specified, the default behavior
1196 is to allocate enough nodes to satisfy the requirements of the
1197 -n and -c options. The job will be allocated as many nodes as
1198 possible within the range specified and without delaying the
1199 initiation of the job. The node count specification may include
1200 a numeric value followed by a suffix of "k" (multiplies numeric
1201 value by 1,024) or "m" (multiplies numeric value by 1,048,576).
1202 -n, --ntasks=<number>
1205 sbatch does not launch tasks, it requests an allocation of
1206 resources and submits a batch script. This option advises the
1207 Slurm controller that job steps run within the allocation will
1208 launch a maximum of number tasks and to provide for sufficient
1209 resources. The default is one task per node, but note that the
1210 --cpus-per-task option will change this default.
1211 --network=<type>
1214 Specify information pertaining to the switch or network. The
1215 interpretation of type is system dependent. This option is sup‐
1216 ported when running Slurm on a Cray natively. It is used to
1217 request using Network Performance Counters. Only one value per
1218 request is valid. All options are case in-sensitive. In this
1219 configuration supported values include:
1220 system
1222 Use the system-wide network performance counters. Only
1223 nodes requested will be marked in use for the job alloca‐
1224 tion. If the job does not fill up the entire system the
1225 rest of the nodes are not able to be used by other jobs
1226 using NPC, if idle their state will appear as PerfCnts.
1227 These nodes are still available for other jobs not using
1228 NPC.
1229 blade Use the blade network performance counters. Only nodes
1231 requested will be marked in use for the job allocation.
1232 If the job does not fill up the entire blade(s) allocated
1233 to the job those blade(s) are not able to be used by other
1234 jobs using NPC, if idle their state will appear as PerfC‐
1235 nts. These nodes are still available for other jobs not
1236 using NPC.
1237 In all cases the job allocation request must specify the
1240 --exclusive option. Otherwise the request will be denied.
1241 Also with any of these options steps are not allowed to share
1243 blades, so resources would remain idle inside an allocation if
1244 the step running on a blade does not take up all the nodes on
1245 the blade.
1246 The network option is also supported on systems with IBM's Par‐
1248 allel Environment (PE). See IBM's LoadLeveler job command key‐
1249 word documentation about the keyword "network" for more informa‐
1250 tion. Multiple values may be specified in a comma separated
1251 list. All options are case in-sensitive. Supported values
1252 include:
1253 BULK_XFER[=<resources>]
1255 Enable bulk transfer of data using Remote
1256 Direct-Memory Access (RDMA). The optional resources
1257 specification is a numeric value which can have a
1258 suffix of "k", "K", "m", "M", "g" or "G" for kilo‐
1259 bytes, megabytes or gigabytes. NOTE: The resources
1260 specification is not supported by the underlying IBM
1261 infrastructure as of Parallel Environment version
1262 2.2 and no value should be specified at this time.
1263 CAU=<count> Number of Collective Acceleration Units (CAU)
1265 required. Applies only to IBM Power7-IH processors.
1266 Default value is zero. Independent CAU will be
1267 allocated for each programming interface (MPI, LAPI,
1268 etc.)
1269 DEVNAME=<name>
1271 Specify the device name to use for communications
1272 (e.g. "eth0" or "mlx4_0").
1273 DEVTYPE=<type>
1275 Specify the device type to use for communications.
1276 The supported values of type are: "IB" (InfiniBand),
1277 "HFI" (P7 Host Fabric Interface), "IPONLY" (IP-Only
1278 interfaces), "HPCE" (HPC Ethernet), and "KMUX" (Ker‐
1279 nel Emulation of HPCE). The devices allocated to a
1280 job must all be of the same type. The default value
1281 depends upon depends upon what hardware is available
1282 and in order of preferences is IPONLY (which is not
1283 considered in User Space mode), HFI, IB, HPCE, and
1284 KMUX.
1285 IMMED =<count>
1287 Number of immediate send slots per window required.
1288 Applies only to IBM Power7-IH processors. Default
1289 value is zero.
1290 INSTANCES =<count>
1292 Specify number of network connections for each task
1293 on each network connection. The default instance
1294 count is 1.
1295 IPV4 Use Internet Protocol (IP) version 4 communications
1297 (default).
1298 IPV6 Use Internet Protocol (IP) version 6 communications.
1300 LAPI Use the LAPI programming interface.
1302 MPI Use the MPI programming interface. MPI is the
1304 default interface.
1305 PAMI Use the PAMI programming interface.
1307 SHMEM Use the OpenSHMEM programming interface.
1309 SN_ALL Use all available switch networks (default).
1311 SN_SINGLE Use one available switch network.
1313 UPC Use the UPC programming interface.
1315 US Use User Space communications.
1317 Some examples of network specifications:
1320 Instances=2,US,MPI,SN_ALL
1322 Create two user space connections for MPI communica‐
1323 tions on every switch network for each task.
1324 US,MPI,Instances=3,Devtype=IB
1326 Create three user space connections for MPI communi‐
1327 cations on every InfiniBand network for each task.
1328 IPV4,LAPI,SN_Single
1330 Create a IP version 4 connection for LAPI communica‐
1331 tions on one switch network for each task.
1332 Instances=2,US,LAPI,MPI
1334 Create two user space connections each for LAPI and
1335 MPI communications on every switch network for each
1336 task. Note that SN_ALL is the default option so
1337 every switch network is used. Also note that
1338 Instances=2 specifies that two connections are
1339 established for each protocol (LAPI and MPI) and
1340 each task. If there are two networks and four tasks
1341 on the node then a total of 32 connections are
1342 established (2 instances x 2 protocols x 2 networks
1343 x 4 tasks).
1344 --nice[=adjustment]
1347 Run the job with an adjusted scheduling priority within Slurm.
1348 With no adjustment value the scheduling priority is decreased by
1349 100. A negative nice value increases the priority, otherwise
1350 decreases it. The adjustment range is +/- 2147483645. Only priv‐
1351 ileged users can specify a negative adjustment.
1352 --no-requeue
1355 Specifies that the batch job should never be requeued under any
1356 circumstances. Setting this option will prevent system adminis‐
1357 trators from being able to restart the job (for example, after a
1358 scheduled downtime), recover from a node failure, or be requeued
1359 upon preemption by a higher priority job. When a job is
1360 requeued, the batch script is initiated from its beginning.
1361 Also see the --requeue option. The JobRequeue configuration
1362 parameter controls the default behavior on the cluster.
1363 --ntasks-per-core=<ntasks>
1366 Request the maximum ntasks be invoked on each core. Meant to be
1367 used with the --ntasks option. Related to --ntasks-per-node
1368 except at the core level instead of the node level. NOTE: This
1369 option is not supported unless SelectType=cons_res is configured
1370 (either directly or indirectly on Cray systems) along with the
1371 node's core count.
1372 --ntasks-per-node=<ntasks>
1375 Request that ntasks be invoked on each node. If used with the
1376 --ntasks option, the --ntasks option will take precedence and
1377 the --ntasks-per-node will be treated as a maximum count of
1378 tasks per node. Meant to be used with the --nodes option. This
1379 is related to --cpus-per-task=ncpus, but does not require knowl‐
1380 edge of the actual number of cpus on each node. In some cases,
1381 it is more convenient to be able to request that no more than a
1382 specific number of tasks be invoked on each node. Examples of
1383 this include submitting a hybrid MPI/OpenMP app where only one
1384 MPI "task/rank" should be assigned to each node while allowing
1385 the OpenMP portion to utilize all of the parallelism present in
1386 the node, or submitting a single setup/cleanup/monitoring job to
1387 each node of a pre-existing allocation as one step in a larger
1388 job script.
1389 --ntasks-per-socket=<ntasks>
1392 Request the maximum ntasks be invoked on each socket. Meant to
1393 be used with the --ntasks option. Related to --ntasks-per-node
1394 except at the socket level instead of the node level. NOTE:
1395 This option is not supported unless SelectType=cons_res is con‐
1396 figured (either directly or indirectly on Cray systems) along
1397 with the node's socket count.
1398 -O, --overcommit
1401 Overcommit resources. When applied to job allocation, only one
1402 CPU is allocated to the job per node and options used to specify
1403 the number of tasks per node, socket, core, etc. are ignored.
1404 When applied to job step allocations (the srun command when exe‐
1405 cuted within an existing job allocation), this option can be
1406 used to launch more than one task per CPU. Normally, srun will
1407 not allocate more than one process per CPU. By specifying
1408 --overcommit you are explicitly allowing more than one process
1409 per CPU. However no more than MAX_TASKS_PER_NODE tasks are per‐
1410 mitted to execute per node. NOTE: MAX_TASKS_PER_NODE is defined
1411 in the file slurm.h and is not a variable, it is set at Slurm
1412 build time.
1413 -o, --output=<filename pattern>
1416 Instruct Slurm to connect the batch script's standard output
1417 directly to the file name specified in the "filename pattern".
1418 By default both standard output and standard error are directed
1419 to the same file. For job arrays, the default file name is
1420 "slurm-%A_%a.out", "%A" is replaced by the job ID and "%a" with
1421 the array index. For other jobs, the default file name is
1422 "slurm-%j.out", where the "%j" is replaced by the job ID. See
1423 the filename pattern section below for filename specification
1424 options.
1425 --open-mode=append|truncate
1428 Open the output and error files using append or truncate mode as
1429 specified. The default value is specified by the system config‐
1430 uration parameter JobFileAppend.
1431 --parsable
1434 Outputs only the job id number and the cluster name if present.
1435 The values are separated by a semicolon. Errors will still be
1436 displayed.
1437 -p, --partition=<partition_names>
1440 Request a specific partition for the resource allocation. If
1441 not specified, the default behavior is to allow the slurm con‐
1442 troller to select the default partition as designated by the
1443 system administrator. If the job can use more than one parti‐
1444 tion, specify their names in a comma separate list and the one
1445 offering earliest initiation will be used with no regard given
1446 to the partition name ordering (although higher priority parti‐
1447 tions will be considered first). When the job is initiated, the
1448 name of the partition used will be placed first in the job
1449 record partition string.
1450 --power=<flags>
1453 Comma separated list of power management plugin options. Cur‐
1454 rently available flags include: level (all nodes allocated to
1455 the job should have identical power caps, may be disabled by the
1456 Slurm configuration option PowerParameters=job_no_level).
1457 --priority=<value>
1460 Request a specific job priority. May be subject to configura‐
1461 tion specific constraints. value should either be a numeric
1462 value or "TOP" (for highest possible value). Only Slurm opera‐
1463 tors and administrators can set the priority of a job.
1464 --profile=<all|none|[energy[,|task[,|lustre[,|network]]]]>
1467 enables detailed data collection by the acct_gather_profile
1468 plugin. Detailed data are typically time-series that are stored
1469 in an HDF5 file for the job or an InfluxDB database depending on
1470 the configured plugin.
1471 All All data types are collected. (Cannot be combined with
1474 other values.)
1475 None No data types are collected. This is the default.
1478 (Cannot be combined with other values.)
1479 Energy Energy data is collected.
1482 Task Task (I/O, Memory, ...) data is collected.
1485 Lustre Lustre data is collected.
1488 Network Network (InfiniBand) data is collected.
1491 --propagate[=rlimit[,rlimit...]]
1494 Allows users to specify which of the modifiable (soft) resource
1495 limits to propagate to the compute nodes and apply to their
1496 jobs. If no rlimit is specified, then all resource limits will
1497 be propagated. The following rlimit names are supported by
1498 Slurm (although some options may not be supported on some sys‐
1499 tems):
1500 ALL All limits listed below (default)
1502 NONE No limits listed below
1504 AS The maximum address space for a process
1506 CORE The maximum size of core file
1508 CPU The maximum amount of CPU time
1510 DATA The maximum size of a process's data segment
1512 FSIZE The maximum size of files created. Note that if the
1514 user sets FSIZE to less than the current size of the
1515 slurmd.log, job launches will fail with a 'File size
1516 limit exceeded' error.
1517 MEMLOCK The maximum size that may be locked into memory
1519 NOFILE The maximum number of open files
1521 NPROC The maximum number of processes available
1523 RSS The maximum resident set size
1525 STACK The maximum stack size
1527 -q, --qos=<qos>
1530 Request a quality of service for the job. QOS values can be
1531 defined for each user/cluster/account association in the Slurm
1532 database. Users will be limited to their association's defined
1533 set of qos's when the Slurm configuration parameter, Account‐
1534 ingStorageEnforce, includes "qos" in it's definition.
1535 -Q, --quiet
1538 Suppress informational messages from sbatch such as Job ID. Only
1539 errors will still be displayed.
1540 --reboot
1543 Force the allocated nodes to reboot before starting the job.
1544 This is only supported with some system configurations and will
1545 otherwise be silently ignored.
1546 --requeue
1549 Specifies that the batch job should eligible to being requeue.
1550 The job may be requeued explicitly by a system administrator,
1551 after node failure, or upon preemption by a higher priority job.
1552 When a job is requeued, the batch script is initiated from its
1553 beginning. Also see the --no-requeue option. The JobRequeue
1554 configuration parameter controls the default behavior on the
1555 cluster.
1556 --reservation=<name>
1559 Allocate resources for the job from the named reservation.
1560 -s, --oversubscribe
1563 The job allocation can over-subscribe resources with other run‐
1564 ning jobs. The resources to be over-subscribed can be nodes,
1565 sockets, cores, and/or hyperthreads depending upon configura‐
1566 tion. The default over-subscribe behavior depends on system
1567 configuration and the partition's OverSubscribe option takes
1568 precedence over the job's option. This option may result in the
1569 allocation being granted sooner than if the --oversubscribe
1570 option was not set and allow higher system utilization, but
1571 application performance will likely suffer due to competition
1572 for resources. Also see the --exclusive option.
1573 -S, --core-spec=<num>
1576 Count of specialized cores per node reserved by the job for sys‐
1577 tem operations and not used by the application. The application
1578 will not use these cores, but will be charged for their alloca‐
1579 tion. Default value is dependent upon the node's configured
1580 CoreSpecCount value. If a value of zero is designated and the
1581 Slurm configuration option AllowSpecResourcesUsage is enabled,
1582 the job will be allowed to override CoreSpecCount and use the
1583 specialized resources on nodes it is allocated. This option can
1584 not be used with the --thread-spec option.
1585 --signal=[B:]<sig_num>[@<sig_time>]
1588 When a job is within sig_time seconds of its end time, send it
1589 the signal sig_num. Due to the resolution of event handling by
1590 Slurm, the signal may be sent up to 60 seconds earlier than
1591 specified. sig_num may either be a signal number or name (e.g.
1592 "10" or "USR1"). sig_time must have an integer value between 0
1593 and 65535. By default, no signal is sent before the job's end
1594 time. If a sig_num is specified without any sig_time, the
1595 default time will be 60 seconds. Use the "B:" option to signal
1596 only the batch shell, none of the other processes will be sig‐
1597 naled. By default all job steps will be signaled, but not the
1598 batch shell itself. To have the signal sent at preemption time
1599 see the preempt_send_user_signal SlurmctldParameter.
1600 --sockets-per-node=<sockets>
1603 Restrict node selection to nodes with at least the specified
1604 number of sockets. See additional information under -B option
1605 above when task/affinity plugin is enabled.
1606 --spread-job
1609 Spread the job allocation over as many nodes as possible and
1610 attempt to evenly distribute tasks across the allocated nodes.
1611 This option disables the topology/tree plugin.
1612 --switches=<count>[@<max-time>]
1615 When a tree topology is used, this defines the maximum count of
1616 switches desired for the job allocation and optionally the maxi‐
1617 mum time to wait for that number of switches. If Slurm finds an
1618 allocation containing more switches than the count specified,
1619 the job remains pending until it either finds an allocation with
1620 desired switch count or the time limit expires. It there is no
1621 switch count limit, there is no delay in starting the job.
1622 Acceptable time formats include "minutes", "minutes:seconds",
1623 "hours:minutes:seconds", "days-hours", "days-hours:minutes" and
1624 "days-hours:minutes:seconds". The job's maximum time delay may
1625 be limited by the system administrator using the SchedulerParam‐
1626 eters configuration parameter with the max_switch_wait parameter
1627 option. On a dragonfly network the only switch count supported
1628 is 1 since communication performance will be highest when a job
1629 is allocate resources on one leaf switch or more than 2 leaf
1630 switches. The default max-time is the max_switch_wait Sched‐
1631 ulerParameters.
1632 -t, --time=<time>
1635 Set a limit on the total run time of the job allocation. If the
1636 requested time limit exceeds the partition's time limit, the job
1637 will be left in a PENDING state (possibly indefinitely). The
1638 default time limit is the partition's default time limit. When
1639 the time limit is reached, each task in each job step is sent
1640 SIGTERM followed by SIGKILL. The interval between signals is
1641 specified by the Slurm configuration parameter KillWait. The
1642 OverTimeLimit configuration parameter may permit the job to run
1643 longer than scheduled. Time resolution is one minute and second
1644 values are rounded up to the next minute.
1645 A time limit of zero requests that no time limit be imposed.
1647 Acceptable time formats include "minutes", "minutes:seconds",
1648 "hours:minutes:seconds", "days-hours", "days-hours:minutes" and
1649 "days-hours:minutes:seconds".
1650 --test-only
1653 Validate the batch script and return an estimate of when a job
1654 would be scheduled to run given the current job queue and all
1655 the other arguments specifying the job requirements. No job is
1656 actually submitted.
1657 --thread-spec=<num>
1660 Count of specialized threads per node reserved by the job for
1661 system operations and not used by the application. The applica‐
1662 tion will not use these threads, but will be charged for their
1663 allocation. This option can not be used with the --core-spec
1664 option.
1665 --threads-per-core=<threads>
1668 Restrict node selection to nodes with at least the specified
1669 number of threads per core. NOTE: "Threads" refers to the num‐
1670 ber of processing units on each core rather than the number of
1671 application tasks to be launched per core. See additional
1672 information under -B option above when task/affinity plugin is
1673 enabled.
1674 --time-min=<time>
1677 Set a minimum time limit on the job allocation. If specified,
1678 the job may have it's --time limit lowered to a value no lower
1679 than --time-min if doing so permits the job to begin execution
1680 earlier than otherwise possible. The job's time limit will not
1681 be changed after the job is allocated resources. This is per‐
1682 formed by a backfill scheduling algorithm to allocate resources
1683 otherwise reserved for higher priority jobs. Acceptable time
1684 formats include "minutes", "minutes:seconds", "hours:min‐
1685 utes:seconds", "days-hours", "days-hours:minutes" and
1686 "days-hours:minutes:seconds".
1687 --tmp=<size[units]>
1690 Specify a minimum amount of temporary disk space per node.
1691 Default units are megabytes unless the SchedulerParameters con‐
1692 figuration parameter includes the "default_gbytes" option for
1693 gigabytes. Different units can be specified using the suffix
1694 [K|M|G|T].
1695 --usage
1698 Display brief help message and exit.
1699 --uid=<user>
1702 Attempt to submit and/or run a job as user instead of the invok‐
1703 ing user id. The invoking user's credentials will be used to
1704 check access permissions for the target partition. User root may
1705 use this option to run jobs as a normal user in a RootOnly par‐
1706 tition for example. If run as root, sbatch will drop its permis‐
1707 sions to the uid specified after node allocation is successful.
1708 user may be the user name or numerical user ID.
1709 --use-min-nodes
1712 If a range of node counts is given, prefer the smaller count.
1713 -V, --version
1716 Display version information and exit.
1717 -v, --verbose
1720 Increase the verbosity of sbatch's informational messages. Mul‐
1721 tiple -v's will further increase sbatch's verbosity. By default
1722 only errors will be displayed.
1723 -w, --nodelist=<node name list>
1726 Request a specific list of hosts. The job will contain all of
1727 these hosts and possibly additional hosts as needed to satisfy
1728 resource requirements. The list may be specified as a
1729 comma-separated list of hosts, a range of hosts (host[1-5,7,...]
1730 for example), or a filename. The host list will be assumed to
1731 be a filename if it contains a "/" character. If you specify a
1732 minimum node or processor count larger than can be satisfied by
1733 the supplied host list, additional resources will be allocated
1734 on other nodes as needed. Duplicate node names in the list will
1735 be ignored. The order of the node names in the list is not
1736 important; the node names will be sorted by Slurm.
1737 -W, --wait
1740 Do not exit until the submitted job terminates. The exit code
1741 of the sbatch command will be the same as the exit code of the
1742 submitted job. If the job terminated due to a signal rather than
1743 a normal exit, the exit code will be set to 1. In the case of a
1744 job array, the exit code recorded will be the highest value for
1745 any task in the job array.
1746 --wait-all-nodes=<value>
1749 Controls when the execution of the command begins. By default
1750 the job will begin execution as soon as the allocation is made.
1751 0 Begin execution as soon as allocation can be made. Do not
1753 wait for all nodes to be ready for use (i.e. booted).
1754 1 Do not begin execution until all nodes are ready for use.
1756 --wckey=<wckey>
1759 Specify wckey to be used with job. If TrackWCKey=no (default)
1760 in the slurm.conf this value is ignored.
1761 --wrap=<command string>
1764 Sbatch will wrap the specified command string in a simple "sh"
1765 shell script, and submit that script to the slurm controller.
1766 When --wrap is used, a script name and arguments may not be
1767 specified on the command line; instead the sbatch-generated
1768 wrapper script is used.
1769 -x, --exclude=<node name list>
1772 Explicitly exclude certain nodes from the resources granted to
1773 the job.
1774
1777 sbatch allows for a filename pattern to contain one or more replacement
1778 symbols, which are a percent sign "%" followed by a letter (e.g. %j).
1779 \\ Do not process any of the replacement symbols.
1781 %% The character "%".
1783 %A Job array's master job allocation number.
1785 %a Job array ID (index) number.
1787 %J jobid.stepid of the running job. (e.g. "128.0")
1789 %j jobid of the running job.
1791 %N short hostname. This will create a separate IO file per node.
1793 %n Node identifier relative to current job (e.g. "0" is the first
1795 node of the running job) This will create a separate IO file per
1796 node.
1797 %s stepid of the running job.
1799 %t task identifier (rank) relative to current job. This will create
1801 a separate IO file per task.
1802 %u User name.
1804 %x Job name.
1806 A number placed between the percent character and format specifier may
1808 be used to zero-pad the result in the IO filename. This number is
1809 ignored if the format specifier corresponds to non-numeric data (%N
1810 for example).
1811 Some examples of how the format string may be used for a 4 task job
1813 step with a Job ID of 128 and step id of 0 are included below:
1814 job%J.out job128.0.out
1816 job%4j.out job0128.out
1818 job%j-%2t.out job128-00.out, job128-01.out, ...
1820
1822 Upon startup, sbatch will read and handle the options set in the fol‐
1823 lowing environment variables. Note that environment variables will
1824 override any options set in a batch script, and command line options
1825 will override any environment variables.
1826 SBATCH_ACCOUNT Same as -A, --account
1829 SBATCH_ACCTG_FREQ Same as --acctg-freq
1831 SBATCH_ARRAY_INX Same as -a, --array
1833 SBATCH_BATCH Same as --batch
1835 SBATCH_CHECKPOINT Same as --checkpoint
1837 SBATCH_CLUSTERS or SLURM_CLUSTERS
1839 Same as --clusters
1840 SBATCH_CONSTRAINT Same as -C, --constraint
1842 SBATCH_CORE_SPEC Same as --core-spec
1844 SBATCH_CPUS_PER_GPU Same as --cpus-per-gpu
1846 SBATCH_DEBUG Same as -v, --verbose
1848 SBATCH_DELAY_BOOT Same as --delay-boot
1850 SBATCH_DISTRIBUTION Same as -m, --distribution
1852 SBATCH_EXCLUSIVE Same as --exclusive
1854 SBATCH_EXPORT Same as --export
1856 SBATCH_GET_USER_ENV Same as --get-user-env
1858 SBATCH_GPUS Same as -G, --gpus
1860 SBATCH_GPU_BIND Same as --gpu-bind
1862 SBATCH_GPU_FREQ Same as --gpu-freq
1864 SBATCH_GPUS_PER_NODE Same as --gpus-per-node
1866 SBATCH_GPUS_PER_TASK Same as --gpus-per-task SBATCH_GRES Same as
1868 --gres
1869 SBATCH_GRES_FLAGS Same as --gres-flags
1871 SBATCH_HINT or SLURM_HINT
1873 Same as --hint
1874 SBATCH_IGNORE_PBS Same as --ignore-pbs
1876 SBATCH_JOB_NAME Same as -J, --job-name
1878 SBATCH_MEM_BIND Same as --mem-bind
1880 SBATCH_MEM_PER_GPU Same as --mem-per-gpu
1882 SBATCH_NETWORK Same as --network
1884 SBATCH_NO_KILL Same as -k, --no-kill
1886 SBATCH_NO_REQUEUE Same as --no-requeue
1888 SBATCH_OPEN_MODE Same as --open-mode
1890 SBATCH_OVERCOMMIT Same as -O, --overcommit
1892 SBATCH_PARTITION Same as -p, --partition
1894 SBATCH_POWER Same as --power
1896 SBATCH_PROFILE Same as --profile
1898 SBATCH_QOS Same as --qos
1900 SBATCH_RESERVATION Same as --reservation
1902 SBATCH_REQ_SWITCH When a tree topology is used, this defines the
1904 maximum count of switches desired for the job
1905 allocation and optionally the maximum time to
1906 wait for that number of switches. See --switches
1907 SBATCH_REQUEUE Same as --requeue
1909 SBATCH_SIGNAL Same as --signal
1911 SBATCH_SPREAD_JOB Same as --spread-job
1913 SBATCH_THREAD_SPEC Same as --thread-spec
1915 SBATCH_TIMELIMIT Same as -t, --time
1917 SBATCH_USE_MIN_NODES Same as --use-min-nodes
1919 SBATCH_WAIT Same as -W, --wait
1921 SBATCH_WAIT_ALL_NODES Same as --wait-all-nodes
1923 SBATCH_WAIT4SWITCH Max time waiting for requested switches. See
1925 --switches
1926 SBATCH_WCKEY Same as --wckey
1928 SLURM_CONF The location of the Slurm configuration file.
1930 SLURM_EXIT_ERROR Specifies the exit code generated when a Slurm
1932 error occurs (e.g. invalid options). This can be
1933 used by a script to distinguish application exit
1934 codes from various Slurm error conditions.
1935 SLURM_STEP_KILLED_MSG_NODE_ID=ID
1937 If set, only the specified node will log when the
1938 job or step are killed by a signal.
1939
1942 The Slurm controller will set the following variables in the environ‐
1943 ment of the batch script.
1944 SBATCH_MEM_BIND
1946 Set to value of the --mem-bind option.
1947 SBATCH_MEM_BIND_LIST
1949 Set to bit mask used for memory binding.
1950 SBATCH_MEM_BIND_PREFER
1952 Set to "prefer" if the --mem-bind option includes the prefer
1953 option.
1954 SBATCH_MEM_BIND_TYPE
1956 Set to the memory binding type specified with the --mem-bind
1957 option. Possible values are "none", "rank", "map_map",
1958 "mask_mem" and "local".
1959 SBATCH_MEM_BIND_VERBOSE
1961 Set to "verbose" if the --mem-bind option includes the verbose
1962 option. Set to "quiet" otherwise.
1963 SLURM_*_PACK_GROUP_#
1965 For a heterogeneous job allocation, the environment variables
1966 are set separately for each component.
1967 SLURM_ARRAY_TASK_COUNT
1969 Total number of tasks in a job array.
1970 SLURM_ARRAY_TASK_ID
1972 Job array ID (index) number.
1973 SLURM_ARRAY_TASK_MAX
1975 Job array's maximum ID (index) number.
1976 SLURM_ARRAY_TASK_MIN
1978 Job array's minimum ID (index) number.
1979 SLURM_ARRAY_TASK_STEP
1981 Job array's index step size.
1982 SLURM_ARRAY_JOB_ID
1984 Job array's master job ID number.
1985 SLURM_CLUSTER_NAME
1987 Name of the cluster on which the job is executing.
1988 SLURM_CPUS_ON_NODE
1990 Number of CPUS on the allocated node.
1991 SLURM_CPUS_PER_GPU
1993 Number of CPUs requested per allocated GPU. Only set if the
1994 --cpus-per-gpu option is specified.
1995 SLURM_CPUS_PER_TASK
1997 Number of cpus requested per task. Only set if the
1998 --cpus-per-task option is specified.
1999 SLURM_DISTRIBUTION
2001 Same as -m, --distribution
2002 SLURM_EXPORT_ENV
2004 Same as -e, --export.
2005 SLURM_GPUS
2007 Number of GPUs requested. Only set if the -G, --gpus option is
2008 specified.
2009 SLURM_GPU_BIND
2011 Requested binding of tasks to GPU. Only set if the --gpu-bind
2012 option is specified.
2013 SLURM_GPU_FREQ
2015 Requested GPU frequency. Only set if the --gpu-freq option is
2016 specified.
2017 SLURM_GPUS_PER_NODE
2019 Requested GPU count per allocated node. Only set if the
2020 --gpus-per-node option is specified.
2021 SLURM_GPUS_PER_SOCKET
2023 Requested GPU count per allocated socket. Only set if the
2024 --gpus-per-socket option is specified.
2025 SLURM_GPUS_PER_TASK
2027 Requested GPU count per allocated task. Only set if the
2028 --gpus-per-task option is specified.
2029 SLURM_GTIDS
2031 Global task IDs running on this node. Zero origin and comma
2032 separated.
2033 SLURM_JOB_ACCOUNT
2035 Account name associated of the job allocation.
2036 SLURM_JOB_ID (and SLURM_JOBID for backwards compatibility)
2038 The ID of the job allocation.
2039 SLURM_JOB_CPUS_PER_NODE
2041 Count of processors available to the job on this node. Note the
2042 select/linear plugin allocates entire nodes to jobs, so the
2043 value indicates the total count of CPUs on the node. The
2044 select/cons_res plugin allocates individual processors to jobs,
2045 so this number indicates the number of processors on this node
2046 allocated to the job.
2047 SLURM_JOB_DEPENDENCY
2049 Set to value of the --dependency option.
2050 SLURM_JOB_NAME
2052 Name of the job.
2053 SLURM_JOB_NODELIST (and SLURM_NODELIST for backwards compatibility)
2055 List of nodes allocated to the job.
2056 SLURM_JOB_NUM_NODES (and SLURM_NNODES for backwards compatibility)
2058 Total number of nodes in the job's resource allocation.
2059 SLURM_JOB_PARTITION
2061 Name of the partition in which the job is running.
2062 SLURM_JOB_QOS
2064 Quality Of Service (QOS) of the job allocation.
2065 SLURM_JOB_RESERVATION
2067 Advanced reservation containing the job allocation, if any.
2068 SLURM_LOCALID
2070 Node local task ID for the process within a job.
2071 SLURM_MEM_PER_CPU
2073 Same as --mem-per-cpu
2074 SLURM_MEM_PER_GPU
2076 Requested memory per allocated GPU. Only set if the
2077 --mem-per-gpu option is specified.
2078 SLURM_MEM_PER_NODE
2080 Same as --mem
2081 SLURM_NODE_ALIASES
2083 Sets of node name, communication address and hostname for nodes
2084 allocated to the job from the cloud. Each element in the set if
2085 colon separated and each set is comma separated. For example:
2086 SLURM_NODE_ALIASES=ec0:1.2.3.4:foo,ec1:1.2.3.5:bar
2087 SLURM_NODEID
2089 ID of the nodes allocated.
2090 SLURM_NTASKS (and SLURM_NPROCS for backwards compatibility)
2092 Same as -n, --ntasks
2093 SLURM_NTASKS_PER_CORE
2095 Number of tasks requested per core. Only set if the
2096 --ntasks-per-core option is specified.
2097 SLURM_NTASKS_PER_NODE
2099 Number of tasks requested per node. Only set if the
2100 --ntasks-per-node option is specified.
2101 SLURM_NTASKS_PER_SOCKET
2103 Number of tasks requested per socket. Only set if the
2104 --ntasks-per-socket option is specified.
2105 SLURM_PACK_SIZE
2107 Set to count of components in heterogeneous job.
2108 SLURM_PRIO_PROCESS
2110 The scheduling priority (nice value) at the time of job submis‐
2111 sion. This value is propagated to the spawned processes.
2112 SLURM_PROCID
2114 The MPI rank (or relative process ID) of the current process
2115 SLURM_PROFILE
2117 Same as --profile
2118 SLURM_RESTART_COUNT
2120 If the job has been restarted due to system failure or has been
2121 explicitly requeued, this will be sent to the number of times
2122 the job has been restarted.
2123 SLURM_SUBMIT_DIR
2125 The directory from which sbatch was invoked or, if applicable,
2126 the directory specified by the -D, --chdir option.
2127 SLURM_SUBMIT_HOST
2129 The hostname of the computer from which sbatch was invoked.
2130 SLURM_TASKS_PER_NODE
2132 Number of tasks to be initiated on each node. Values are comma
2133 separated and in the same order as SLURM_JOB_NODELIST. If two
2134 or more consecutive nodes are to have the same task count, that
2135 count is followed by "(x#)" where "#" is the repetition count.
2136 For example, "SLURM_TASKS_PER_NODE=2(x3),1" indicates that the
2137 first three nodes will each execute three tasks and the fourth
2138 node will execute one task.
2139 SLURM_TASK_PID
2141 The process ID of the task being started.
2142 SLURM_TOPOLOGY_ADDR
2144 This is set only if the system has the topology/tree plugin
2145 configured. The value will be set to the names network
2146 switches which may be involved in the job's communications
2147 from the system's top level switch down to the leaf switch and
2148 ending with node name. A period is used to separate each hard‐
2149 ware component name.
2150 SLURM_TOPOLOGY_ADDR_PATTERN
2152 This is set only if the system has the topology/tree plugin
2153 configured. The value will be set component types listed in
2154 SLURM_TOPOLOGY_ADDR. Each component will be identified as
2155 either "switch" or "node". A period is used to separate each
2156 hardware component type.
2157 SLURMD_NODENAME
2159 Name of the node running the job script.
2160
2163 Specify a batch script by filename on the command line. The batch
2164 script specifies a 1 minute time limit for the job.
2165 $ cat myscript
2167 #!/bin/sh
2168 #SBATCH --time=1
2169 srun hostname |sort
2170 $ sbatch -N4 myscript
2172 salloc: Granted job allocation 65537
2173 $ cat slurm-65537.out
2175 host1
2176 host2
2177 host3
2178 host4
2179 Pass a batch script to sbatch on standard input:
2182 $ sbatch -N4 <<EOF
2184 > #!/bin/sh
2185 > srun hostname |sort
2186 > EOF
2187 sbatch: Submitted batch job 65541
2188 $ cat slurm-65541.out
2190 host1
2191 host2
2192 host3
2193 host4
2194 To create a heterogeneous job with 3 components, each allocating a
2197 unique set of nodes:
2198 sbatch -w node[2-3] : -w node4 : -w node[5-7] work.bash
2200 Submitted batch job 34987
2201
2204 Copyright (C) 2006-2007 The Regents of the University of California.
2205 Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2206 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2207 Copyright (C) 2010-2017 SchedMD LLC.
2208 This file is part of Slurm, a resource management program. For
2210 details, see <https://slurm.schedmd.com/>.
2211 Slurm is free software; you can redistribute it and/or modify it under
2213 the terms of the GNU General Public License as published by the Free
2214 Software Foundation; either version 2 of the License, or (at your
2215 option) any later version.
2216 Slurm is distributed in the hope that it will be useful, but WITHOUT
2218 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2219 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2220 for more details.
2221
2224 sinfo(1), sattach(1), salloc(1), squeue(1), scancel(1), scontrol(1),
2225 slurm.conf(5), sched_setaffinity (2), numa (3)
2226December 2019 Slurm Commands sbatch(1)