1CONDOR_SUBMIT(1) HTCondor Manual CONDOR_SUBMIT(1)
2
6 condor_submit - HTCondor Manual
7 Queue jobs for execution under HTCondor
9
12 condor_submit [-terse ] [-verbose ] [-unused ] [-file submit_file]
13 [-name schedd_name] [-remote schedd_name] [-addr <ip:port>] [-pool
14 pool_name] [-disable ] [-password passphrase] [-debug ] [-append com‐
15 mand ...][-batch-name batch_name] [-spool ] [-dump filename] [-interac‐
16 tive ] [-factory ] [-allow-crlf-script ] [-dry-run ] [-maxjobs num‐
17 ber-of-jobs] [-single-cluster ] [<submit-variable>=<value> ] [submit
18 description file ] [-queue queue_arguments]
19
21 condor_submit is the program for submitting jobs for execution under
22 HTCondor. condor_submit requires one or more submit description com‐
23 mands to direct the queuing of jobs. These commands may come from a
24 file, standard input, the command line, or from some combination of
25 these. One submit description may contain specifications for the queu‐
26 ing of many HTCondor jobs at once. A single invocation of condor_submit
27 may cause one or more clusters. A cluster is a set of jobs specified in
28 the submit description between queue commands for which the exe‐
29 cutable is not changed. It is advantageous to submit multiple jobs as a
30 single cluster because the schedd uses much less memory to hold the
31 jobs.
32 Multiple clusters may be specified within a single submit description.
34 Each cluster must specify a single executable.
35 The job ClassAd attribute ClusterId identifies a cluster.
37 The submit description file argument is the path and file name of the
39 submit description file. If this optional argument is the dash charac‐
40 ter (-), then the commands are taken from standard input. If - is spec‐
41 ified for the submit description file, -verbose is implied; this can be
42 overridden by specifying -terse.
43 If no submit description file argument is given, and no -queue argument
45 is given, commands are taken automatically from standard input.
46 Note that submission of jobs from a Windows machine requires a stashed
48 password to allow HTCondor to impersonate the user submitting the job.
49 To stash a password, use the condor_store_cred command. See the manual
50 page for details.
51 For lengthy lines within the submit description file, the backslash (\)
53 is a line continuation character. Placing the backslash at the end of a
54 line causes the current line's command to be continued with the next
55 line of the file. Submit description files may contain comments. A com‐
56 ment is any line beginning with a pound character (#).
57
59 -terse Terse output - display JobId ranges only.
60 -verbose
62 Verbose output - display the created job ClassAd
63 -unused
65 As a default, causes no warnings to be issued about user-de‐
66 fined macros not being used within the submit description
67 file. The meaning reverses (toggles) when the configuration
68 variable WARN_ON_UNUSED_SUBMIT_FILE_MACROS
69 is set to the non default value of False. Printing the
70 warnings can help identify spelling errors of submit descrip‐
71 tion file commands. The warnings are sent to stderr.
72 -file submit_file
74 Use submit_file as the submit description file. This is
75 equivalent to providing submit_file as an argument without
76 the preceding -file.
77 -name schedd_name
79 Submit to the specified condor_schedd. Use this option to
80 submit to a condor_schedd other than the default local one.
81 schedd_name is the value of the Name ClassAd attribute on the
82 machine where the condor_schedd daemon runs.
83 -remote schedd_name
85 Submit to the specified condor_schedd, spooling all required
86 input files over the network connection. schedd_name is the
87 value of the Name ClassAd attribute on the machine where the
88 condor_schedd daemon runs. This option is equivalent to using
89 both -name and -spool.
90 -addr <ip:port>
92 Submit to the condor_schedd at the IP address and port given
93 by the sinful string argument <ip:port>.
94 -pool pool_name
96 Look in the specified pool for the condor_schedd to submit
97 to. This option is used with -name or -remote.
98 -disable
100 Disable file permission checks when submitting a job for read
101 permissions on all input files, such as those defined by com‐
102 mands input and transfer_input_files , as well as write
103 permission to output files, such as a log file defined by log
104 and output files defined with output or transfer_out‐
105 put_files .
106 -debug Cause debugging information to be sent to stderr, based on
108 the value of the configuration variable TOOL_DEBUG.
109 -append command
111 Augment the commands in the submit description file with the
112 given command. This command will be considered to immediately
113 precede the queue command within the submit description file,
114 and come after all other previous commands. If the command
115 specifies a queue command, as in the example
116 condor_submit mysubmitfile -append "queue input in A, B, C"
118 then the entire -append command line option and its arguments
120 are converted to
121 condor_submit mysubmitfile -queue input in A, B, C
123 The submit description file is not modified. Multiple com‐
125 mands are specified by using the -append option multiple
126 times. Each new command is given in a separate -append op‐
127 tion. Commands with spaces in them will need to be enclosed
128 in double quote marks.
129 -batch-name batch_name
131 Set the batch name for this submit. The batch name is dis‐
132 played by condor_q -batch. It is intended for use by users to
133 give meaningful names to their jobs and to influence how con‐
134 dor_q groups jobs for display. Use of this argument takes
135 precedence over a batch name specified in the submit descrip‐
136 tion file itself.
137 -spool Spool all required input files, job event log, and proxy over
139 the connection to the condor_schedd. After submission, modify
140 local copies of the files without affecting your jobs. Any
141 output files for completed jobs need to be retrieved with
142 condor_transfer_data.
143 -dump filename
145 Sends all ClassAds to the specified file, instead of to the
146 condor_schedd.
147 -interactive
149 Indicates that the user wants to run an interactive shell on
150 an execute machine in the pool. This is equivalent to creat‐
151 ing a submit description file of a vanilla universe sleep
152 job, and then running condor_ssh_to_job by hand. Without any
153 additional arguments, condor_submit with the -interactive
154 flag creates a dummy vanilla universe job that sleeps, sub‐
155 mits it to the local scheduler, waits for the job to run, and
156 then launches condor_ssh_to_job to run a shell. If the user
157 would like to run the shell on a machine that matches a par‐
158 ticular requirements expression, the submit description file
159 is specified, and it will contain the expression. Note that
160 all policy expressions specified in the submit description
161 file are honored, but any executable or universe commands
162 are overwritten to be sleep and vanilla. The job ClassAd at‐
163 tribute InteractiveJob is set to True to identify interactive
164 jobs for condor_startd policy usage.
165 -factory
167 Sends all of the jobs as a late materialization job factory.
168 A job factory consists of a single cluster classad and a di‐
169 gest containing the submit commands necessary to describe the
170 differences between jobs. If the Queue statement has item‐
171 data, then the itemdata will be sent. Using this option is
172 equivalent to using the max_materialize
173 submit command.
174 -allow-crlf-script
176 Changes the check for an invalid line ending on the exe‐
177 cutable script's #! line from an ERROR to a WARNING. The #!
178 line will be ignored by Windows, so it won't matter if it is
179 invalid; but Unix and Linux will not run a script that has a
180 Windows/DOS line ending on the first line of the script. So
181 condor_submit will not allow such a script to be submitted as
182 the job's executable unless this option is supplied.
183 -dry-run file
185 Parse the submit description file, sending the resulting job
186 ClassAd to the file given by file, but do not submit the
187 job(s). This permits observation of the job specification,
188 and it facilitates debugging the submit description file con‐
189 tents. If file is -, the output is written to stdout.
190 -maxjobs number-of-jobs
192 If the total number of jobs specified by the submit descrip‐
193 tion file is more than the integer value given by num‐
194 ber-of-jobs, then no jobs are submitted for execution and an
195 error message is generated. A 0 or negative value for the
196 number-of-jobs causes no limit to be imposed.
197 -single-cluster
199 If the jobs specified by the submit description file causes
200 more than a single cluster value to be assigned, then no jobs
201 are submitted for execution and an error message is gener‐
202 ated.
203 <submit-variable>=<value>
205 Defines a submit command or submit variable with a value, and
206 parses it as if it was placed at the beginning of the submit
207 description file. The submit description file is not changed.
208 To correctly parse the condor_submit command line, this op‐
209 tion must be specified without white space characters before
210 and after the equals sign (=), or the entire option must be
211 surrounded by double quote marks.
212 -queue queue_arguments
214 A command line specification of how many jobs to queue, which
215 is only permitted if the submit description file does not
216 have a queue command. The queue_arguments are the same as may
217 be within a submit description file. The parsing of the
218 queue_arguments finishes at the end of the line or when a
219 dash character (-) is encountered. Therefore, its best place‐
220 ment within the command line will be at the end of the com‐
221 mand line.
222 On a Unix command line, the shell expands file globs before
224 parsing occurs.
225
227 Note: more information on submitting HTCondor jobs can be found here:
228 Submitting a Job.
229 As of version 8.5.6, the condor_submit language supports multi-line
231 values in commands. The syntax is the same as the configuration lan‐
232 guage (see more details here: admin-manual/introduction-to-configura‐
233 tion:multi-line values).
234 Each submit description file describes one or more clusters of jobs to
236 be placed in the HTCondor execution pool. All jobs in a cluster must
237 share the same executable, but they may have different input and output
238 files, and different program arguments. The submit description file is
239 generally the last command-line argument to condor_submit. If the sub‐
240 mit description file argument is omitted, condor_submit will read the
241 submit description from standard input.
242 The submit description file must contain at least one executable com‐
244 mand and at least one queue command. All of the other commands have de‐
245 fault actions.
246 Note that a submit file that contains more than one executable command
248 will produce multiple clusters when submitted. This is not generally
249 recommended, and is not allowed for submit files that are run as DAG
250 node jobs by condor_dagman.
251 The commands which can appear in the submit description file are numer‐
253 ous. They are listed here in alphabetical order by category.
254 BASIC COMMANDS
256 arguments = <argument_list>
259 List of arguments to be supplied to the executable as part of
260 the command line.
261 In the java universe, the first argument must be the name of
263 the class containing main.
264 There are two permissible formats for specifying arguments,
266 identified as the old syntax and the new syntax. The old syn‐
267 tax supports white space characters within arguments only in
268 special circumstances; when used, the command line arguments
269 are represented in the job ClassAd attribute Args. The new
270 syntax supports uniform quoting of white space characters
271 within arguments; when used, the command line arguments are
272 represented in the job ClassAd attribute Arguments.
273 Old Syntax
275 In the old syntax, individual command line arguments are de‐
277 limited (separated) by space characters. To allow a double
278 quote mark in an argument, it is escaped with a backslash;
279 that is, the two character sequence \" becomes a single dou‐
280 ble quote mark within an argument.
281 Further interpretation of the argument string differs depend‐
283 ing on the operating system. On Windows, the entire argument
284 string is passed verbatim (other than the backslash in front
285 of double quote marks) to the Windows application. Most Win‐
286 dows applications will allow spaces within an argument value
287 by surrounding the argument with double quotes marks. In all
288 other cases, there is no further interpretation of the argu‐
289 ments.
290 Example:
292 arguments = one \"two\" 'three'
294 Produces in Unix vanilla universe:
296 argument 1: one
298 argument 2: "two"
299 argument 3: 'three'
300 New Syntax
302 Here are the rules for using the new syntax:
304 1. The entire string representing the command line arguments
306 is surrounded by double quote marks. This permits the
307 white space characters of spaces and tabs to potentially
308 be embedded within a single argument. Putting the double
309 quote mark within the arguments is accomplished by escap‐
310 ing it with another double quote mark.
311 2. The white space characters of spaces or tabs delimit argu‐
313 ments.
314 3. To embed white space characters of spaces or tabs within a
316 single argument, surround the entire argument with single
317 quote marks.
318 4. To insert a literal single quote mark, escape it within an
320 argument already delimited by single quote marks by adding
321 another single quote mark.
322 Example:
324 arguments = "3 simple arguments"
326 Produces:
328 argument 1: 3
330 argument 2: simple
331 argument 3: arguments
332 Another example:
334 arguments = "one 'two with spaces' 3"
336 Produces:
338 argument 1: one
340 argument 2: two with spaces
341 argument 3: 3
342 And yet another example:
344 arguments = "one ""two"" 'spacey ''quoted'' argument'"
346 Produces:
348 argument 1: one
350 argument 2: "two"
351 argument 3: spacey 'quoted' argument
352 Notice that in the new syntax, the backslash has no special
354 meaning. This is for the convenience of Windows users.
355 environment = <parameter_list>
360 List of environment variables.
361 There are two different formats for specifying the environ‐
363 ment variables: the old format and the new format. The old
364 format is retained for backward-compatibility. It suffers
365 from a platform-dependent syntax and the inability to insert
366 some special characters into the environment.
367 The new syntax for specifying environment values:
369 1. Put double quote marks around the entire argument string.
371 This distinguishes the new syntax from the old. The old
372 syntax does not have double quote marks around it. Any
373 literal double quote marks within the string must be es‐
374 caped by repeating the double quote mark.
375 2. Each environment entry has the form
377 <name>=<value>
379 3. Use white space (space or tab characters) to separate en‐
381 vironment entries.
382 4. To put any white space in an environment entry, surround
384 the space and as much of the surrounding entry as desired
385 with single quote marks.
386 5. To insert a literal single quote mark, repeat the single
388 quote mark anywhere inside of a section surrounded by sin‐
389 gle quote marks.
390 Example:
392 environment = "one=1 two=""2"" three='spacey ''quoted'' value'"
394 Produces the following environment entries:
396 one=1
398 two="2"
399 three=spacey 'quoted' value
400 Under the old syntax, there are no double quote marks sur‐
402 rounding the environment specification. Each environment en‐
403 try remains of the form
404 <name>=<value>
406 Under Unix, list multiple environment entries by separating
408 them with a semicolon (;). Under Windows, separate multiple
409 entries with a vertical bar (|). There is no way to insert a
410 literal semicolon under Unix or a literal vertical bar under
411 Windows. Note that spaces are accepted, but rarely desired,
412 characters within parameter names and values, because they
413 are treated as literal characters, not separators or ignored
414 white space. Place spaces within the parameter list only if
415 required.
416 A Unix example:
418 environment = one=1;two=2;three="quotes have no 'special' meaning"
420 This produces the following:
422 one=1
424 two=2
425 three="quotes have no 'special' meaning"
426 If the environment is set with the environment command and
428 getenv is also set, values specified with environment over‐
429 ride values in the submitter's environment (regardless of the
430 order of the environment and getenv commands).
431 error = <pathname>
435 A path and file name used by HTCondor to capture any error
436 messages the program would normally write to the screen (that
437 is, this file becomes stderr). A path is given with respect
438 to the file system of the machine on which the job is submit‐
439 ted. The file is written (by the job) in the remote scratch
440 directory of the machine where the job is executed. When the
441 job exits, the resulting file is transferred back to the ma‐
442 chine where the job was submitted, and the path is utilized
443 for file placement. If you specify a relative path, the fi‐
444 nal path will be relative to the job's initial working direc‐
445 tory, and HTCondor will create directories as necessary to
446 transfer the file. If not specified, the default value of
447 /dev/null is used for submission to a Unix machine. If not
448 specified, error messages are ignored for submission to a
449 Windows machine. More than one job should not use the same
450 error file, since this will cause one job to overwrite the
451 errors of another. If HTCondor detects that the error and
452 output files for a job are the same, it will run the job such
453 that the output and error data is merged.
454 executable = <pathname>
458 An optional path and a required file name of the executable
459 file for this job cluster. Only one executable command
460 within a submit description file is guaranteed to work prop‐
461 erly. More than one often works.
462 If no path or a relative path is used, then the executable
464 file is presumed to be relative to the current working direc‐
465 tory of the user as the condor_submit command is issued.
466 batch_name = <batch_name>
472 Set the batch name for this submit. The batch name is dis‐
473 played by condor_q -batch. It is intended for use by users to
474 give meaningful names to their jobs and to influence how con‐
475 dor_q groups jobs for display. This value in a submit file
476 can be overridden by specifying the -batch-name argument on
477 the condor_submit command line.
478 getenv = <<matchlist> | True | False>
480 If getenv is set to
481 True, then condor_submit will copy all of the user's current
482 shell environment variables at the time of job submission
483 into the job ClassAd. The job will therefore execute with the
484 same set of environment variables that the user had at submit
485 time. Defaults to False. A wholesale import of the user's
486 environment is very likely to lead to problems executing the
487 job on a remote machine unless there is a shared file system
488 for users' home directories between the access point and exe‐
489 cute machine. So rather than setting getenv to True, it is
490 much better to set it to a list of environment variables to
491 import.
492 Matchlist is a comma, semicolon or space separated list of
494 environment variable names and name patterns that match or
495 reject names. Matchlist members are matched case-insensi‐
496 tively to each name in the environment and those that match
497 are imported. Matchlist members can contain * as wildcard
498 character which matches anything at that position. Members
499 can have two * characters if one of them is at the end. Mem‐
500 bers can be prefixed with ! to force a matching environment
501 variable to not be imported. The order of members in the
502 Matchlist has no effect on the result. getenv = true is
503 equivalent to getenv = *
504 Prior to HTCondor 8.9.7 getenv allows only True or False as
506 values.
507 Examples:
509 # import everything except PATH and INCLUDE (also path, include and other case-variants)
511 getenv = !PATH, !INCLUDE
512 # import everything with CUDA in the name
514 getenv = *cuda*
515 # Import every environment variable that starts with P or Q, except PATH
517 getenv = !path, P*, Q*
518 If the environment is set with the environment command and
520 getenv is also set, values specified with environment over‐
521 ride values in the submitter's environment (regardless of the
522 order of the environment and getenv commands).
523 input = <pathname>
527 HTCondor assumes that its jobs are long-running, and that the
528 user will not wait at the terminal for their completion. Be‐
529 cause of this, the standard files which normally access the
530 terminal, (stdin, stdout, and stderr), must refer to files.
531 Thus, the file name specified with input should contain any
532 keyboard input the program requires (that is, this file be‐
533 comes stdin). A path is given with respect to the file system
534 of the machine on which the job is submitted. The file is
535 transferred before execution to the remote scratch directory
536 of the machine where the job is executed. If not specified,
537 the default value of /dev/null is used for submission to a
538 Unix machine. If not specified, input is ignored for submis‐
539 sion to a Windows machine.
540 Note that this command does not refer to the command-line ar‐
542 guments of the program. The command-line arguments are speci‐
543 fied by the arguments command.
544 log = <pathname>
548 Use log to specify a file name where HTCondor will write a
549 log file of what is happening with this job cluster, called a
550 job event log. For example, HTCondor will place a log entry
551 into this file when and where the job begins running, when it
552 transfers files, if the job is evicted, and when the job com‐
553 pletes. Most users find specifying a log file to be handy;
554 its use is recommended. If no log entry is specified, HTCon‐
555 dor does not create a log for this cluster. If a relative
556 path is specified, it is relative to the current working di‐
557 rectory as the job is submitted or the directory specified by
558 submit command initialdir on the access point.
559 notification = <Always | Complete | Error | Never>
564 Owners of HTCondor jobs are notified by e-mail when certain
565 events occur. If defined by Always or Complete, the owner
566 will be notified when the job terminates. If defined by Er‐
567 ror, the owner will only be notified if the job terminates
568 abnormally, (as defined by JobSuccessExitCode, if defined) or
569 if the job is placed on hold because of a failure, and not by
570 user request. If defined by Never (the default), the owner
571 will not receive e-mail, regardless to what happens to the
572 job. The HTCondor User's manual documents statistics included
573 in the e-mail.
574 notify_user = <email-address>
578 Used to specify the e-mail address to use when HTCondor sends
579 e-mail about a job. If not specified, HTCondor defaults to
580 using the e-mail address defined by
581 job-owner@UID_DOMAIN
583 where the configuration variable UID_DOMAIN
585 is specified by the HTCondor site administrator. If UID_DO‐
586 MAIN has not been specified, HTCondor sends the e-mail to:
587 job-owner@submit-machine-name
589 output = <pathname>
593 The output file captures any information the program would
594 ordinarily write to the screen (that is, this file becomes
595 stdout). A path is given with respect to the file system of
596 the machine on which the job is submitted. The file is writ‐
597 ten (by the job) in the remote scratch directory of the ma‐
598 chine where the job is executed. When the job exits, the re‐
599 sulting file is transferred back to the machine where the job
600 was submitted, and the path is utilized for file placement.
601 If you specify a relative path, the final path will be rela‐
602 tive to the job's initial working directory, and HTCondor
603 will create directories as necessary to transfer the file.
604 If not specified, the default value of /dev/null is used for
605 submission to a Unix machine. If not specified, output is ig‐
606 nored for submission to a Windows machine. Multiple jobs
607 should not use the same output file, since this will cause
608 one job to overwrite the output of another. If HTCondor de‐
609 tects that the error and output files for a job are the same,
610 it will run the job such that the output and error data is
611 merged.
612 Note that if a program explicitly opens and writes to a file,
614 that file should not be specified as the output file.
615 priority = <integer>
619 An HTCondor job priority can be any integer, with 0 being the
620 default. Jobs with higher numerical priority will run before
621 jobs with lower numerical priority. Note that this priority
622 is on a per user basis. One user with many jobs may use this
623 command to order his/her own jobs, and this will have no ef‐
624 fect on whether or not these jobs will run ahead of another
625 user's jobs.
626 Note that the priority setting in an HTCondor submit file
628 will be overridden by condor_dagman if the submit file is
629 used for a node in a DAG, and the priority of the node within
630 the DAG is non-zero (see automated-workflows/dagman-priori‐
631 ties:Setting Priorities for Nodes for more details).
632 queue [<int expr> ]
636 Places zero or more copies of the job into the HTCondor
637 queue.
638 queue [<int expr> ] [<varname> ] in [slice ] <list of items> Places
640 zero or more copies of the job in the queue based on items in
641 a <list of items>
642 queue [<int expr> ] [<varname> ] matching [files | dirs ] [slice ]
644 <list of items with file globbing>] Places zero or more
645 copies of the job in the queue based on files that match a
646 <list of items with file globbing>
647 queue [<int expr> ] [<list of varnames> ] from [slice ] <file name>
649 | <list of items>] Places zero or more copies of the job in
650 the queue based on lines from the submit file or from <file
651 name>
652 The optional argument <int expr> specifies how many times to
654 repeat the job submission for a given set of arguments. It
655 may be an integer or an expression that evaluates to an inte‐
656 ger, and it defaults to 1. All but the first form of this
657 command are various ways of specifying a list of items. When
658 these forms are used <int expr> jobs will be queued for each
659 item in the list. The in, matching and from keyword indicates
660 how the list will be specified.
661 • in The list of items is an explicit comma and/or space sep‐
663 arated <list of items>. If the <list of items> begins with
664 an open paren, and the close paren is not on the same line
665 as the open, then the list continues until a line that be‐
666 gins with a close paren is read from the submit file.
667 • matching Each item in the <list of items with file glob‐
669 bing> will be matched against the names of files and direc‐
670 tories relative to the current directory, the set of match‐
671 ing names is the resulting list of items.
672 • files Only filenames will matched.
674 • dirs Only directory names will be matched.
676 • from <file name> | <list of items> Each line from <file
678 name> or <list of items> is a single item, this allows for
679 multiple variables to be set for each item. Lines from
680 <file name> or <list of items> will be split on comma
681 and/or space until there are values for each of the vari‐
682 ables specified in <list of varnames>. The last variable
683 will contain the remainder of the line. When the <list of
684 items> form is used, the list continues until the first
685 line that begins with a close paren, and lines beginning
686 with pound sign ('#') will be skipped. When using the
687 <file name> form, if the <file name> ends with |, then it
688 will be executed as a script whatever the script writes to
689 stdout will be the list of items.
690 The optional argument <varname> or <list of varnames> is the
692 name or names of of variables that will be set to the value
693 of the current item when queuing the job. If no <varname> is
694 specified the variable ITEM will be used. Leading and trail‐
695 ing whitespace be trimmed. The optional argument <slice> is a
696 python style slice selecting only some of the items in the
697 list of items. Negative step values are not supported.
698 A submit file may contain more than one queue statement,
700 and if desired, any commands may be placed between subsequent
701 queue commands, such as new input , output , error ,
702 initialdir , or arguments commands. This is handy when
703 submitting multiple runs into one cluster with one submit de‐
704 scription file.
705 universe = <vanilla | scheduler | local | grid | java | vm | paral‐
709 lel | docker>
710 Specifies which HTCondor universe to use when running this
711 job. The HTCondor universe specifies an HTCondor execution
712 environment.
713 The vanilla universe is the default (except where the config‐
715 uration variable DEFAULT_UNIVERSE
716 defines it otherwise).
717 The scheduler universe is for a job that is to run on the ma‐
719 chine where the job is submitted. This universe is intended
720 for a job that acts as a metascheduler and will not be pre‐
721 empted.
722 The local universe is for a job that is to run on the machine
724 where the job is submitted. This universe runs the job imme‐
725 diately and will not preempt the job.
726 The grid universe forwards the job to an external job manage‐
728 ment system. Further specification of the grid universe is
729 done with the grid_resource command.
730 The java universe is for programs written to the Java Virtual
732 Machine.
733 The vm universe facilitates the execution of a virtual ma‐
735 chine.
736 The parallel universe is for parallel jobs (e.g. MPI) that
738 require multiple machines in order to run.
739 The docker universe runs a docker container as an HTCondor
741 job.
742 max_materialize = <limit>
746 Submit jobs as a late materialization factory and instruct
747 the condor_schedd to keep the given number of jobs material‐
748 ized. Use this option to reduce the load on the con‐
749 dor_schedd when submitting a large number of jobs. The limit
750 can be an expression but it must evaluate to a constant at
751 submit time. A limit less than 1 will be treated as unlim‐
752 ited. The condor_schedd can be configured to have a materi‐
753 alization limit as well, the lower of the two limits will be
754 used. (see users-manual/submitting-a-job:submitting lots of
755 jobs for more details).
756 max_idle = <limit>
760 Submit jobs as a late materialization factory and instruct
761 the condor_schedd to keep the given number of non-running
762 jobs materialized. Use this option to reduce the load on the
763 condor_schedd when submitting a large number of jobs. The
764 limit may be an expression but it must evaluate to a constant
765 at submit time. Jobs in the Held state are considered to be
766 Idle for this limit. A limit of less than 1 will prevent
767 jobs from being materialized although the factory will still
768 be submitted to the condor_schedd. (see users-manual/sub‐
769 mitting-a-job:submitting lots of jobs for more details).
770 COMMANDS FOR MATCHMAKING
772 rank = <ClassAd Float Expression>
775 A ClassAd Floating-Point expression that states how to rank
776 machines which have already met the requirements expression.
777 Essentially, rank expresses preference. A higher numeric
778 value equals better rank. HTCondor will give the job the ma‐
779 chine with the highest rank. For example,
780 request_memory = max({60, Target.TotalSlotMemory})
782 rank = Memory
783 asks HTCondor to find all available machines with more than
785 60 megabytes of memory and give to the job the machine with
786 the most amount of memory. The HTCondor User's Manual con‐
787 tains complete information on the syntax and available at‐
788 tributes that can be used in the ClassAd expression.
789 request_cpus = <num-cpus>
793 A requested number of CPUs (cores). If not specified, the
794 number requested will be 1. If specified, the expression
795 && (RequestCpus <= Target.Cpus)
797 is appended to the requirements expression for the job.
799 For pools that enable dynamic condor_startd provisioning,
801 specifies the minimum number of CPUs requested for this job,
802 resulting in a dynamic slot being created with this many
803 cores.
804 request_disk = <quantity>
808 The requested amount of disk space in KiB requested for this
809 job. If not specified, it will be set to the job ClassAd at‐
810 tribute DiskUsage. The expression
811 && (RequestDisk <= Target.Disk)
813 is appended to the requirements expression for the job.
815 For pools that enable dynamic condor_startd provisioning, a
817 dynamic slot will be created with at least this much disk
818 space.
819 Characters may be appended to a numerical value to indicate
821 units. K or KB indicates KiB, 210 numbers of bytes. M or MB
822 indicates MiB, 220 numbers of bytes. G or GB indicates GiB,
823 230 numbers of bytes. T or TB indicates TiB, 240 numbers of
824 bytes.
825 request_gpus = <num-gpus>
829 A requested number of GPUs. If not specified, no GPUs will be
830 requested. If specified and require_gpus is not also speci‐
831 fied, the expression
832 && (Target.GPUs >= RequestGPUs)
834 is appended to the requirements expression for the job.
836 For pools that enable dynamic condor_startd provisioning,
838 specifies the minimum number of GPUs requested for this job,
839 resulting in a dynamic slot being created with this many
840 GPUs.
841 require_gpus = <constraint-expression>
845 A constraint on the properties of GPUs when used with a
846 non-zero request_gpus value. If not specified, no constraint
847 on GPUs will be added to the job. If specified and re‐
848 quest_gpus is non-zero, the expression
849 && (countMatches(MY.RequireGPUs, TARGET.AvailableGPUs) >= RequestGPUs)
851 is appended to the requirements expression for the job. This
853 expression cannot be evaluated by HTCondor prior to version
854 9.8.0. A warning to this will effect will be printed when
855 condor_submit detects this condition.
856 For pools that enable dynamic condor_startd provisioning and
858 are at least version 9.8.0, the constraint will be tested
859 against the properties of AvailableGPUs and only those that
860 match will be assigned to the dynamic slot.
861 request_memory = <quantity>
865 The required amount of memory in MiB that this job needs to
866 avoid excessive swapping. If not specified and the submit
867 command vm_memory is specified, then the value specified
868 for vm_memory defines request_memory . If neither re‐
869 quest_memory nor vm_memory is specified, the value is set
870 by the configuration variable JOB_DEFAULT_REQUESTMEMORY
871 . The actual amount of memory used by a job is represented
872 by the job ClassAd attribute MemoryUsage.
873 For pools that enable dynamic condor_startd provisioning, a
875 dynamic slot will be created with at least this much RAM.
876 The expression
878 && (RequestMemory <= Target.Memory)
880 is appended to the requirements expression for the job.
882 Characters may be appended to a numerical value to indicate
884 units. K or KB indicates KiB, 210 numbers of bytes. M or MB
885 indicates MiB, 220 numbers of bytes. G or GB indicates GiB,
886 230 numbers of bytes. T or TB indicates TiB, 240 numbers of
887 bytes.
888 request_<name> = <quantity>
894 The required amount of the custom machine resource identified
895 by <name> that this job needs. The custom machine resource is
896 defined in the machine's configuration. Machines that have
897 available GPUs will define <name> to be GPUs. <name> must be
898 at least two characters, and must not begin with _. If
899 <name> is either Cpu or Gpu a warning will be printed since
900 these are common typos.
901 cuda_version = <version>
905 The version of the CUDA runtime, if any, used or required by
906 this job, specified as <major>.<minor> (for example, 9.1).
907 If the minor version number is zero, you may specify only the
908 major version number. A single version number of 1000 or
909 higher is assumed to be the integer-coded version number (ma‐
910 jor * 1000 + (minor % 100)).
911 This does not arrange for the CUDA runtime to be present,
913 only for the job to run on a machine whose driver supports
914 the specified version.
915 requirements = <ClassAd Boolean Expression>
919 The requirements command is a boolean ClassAd expression
920 which uses C-like operators. In order for any job in this
921 cluster to run on a given machine, this requirements expres‐
922 sion must evaluate to true on the given machine.
923 For scheduler and local universe jobs, the requirements ex‐
925 pression is evaluated against the Scheduler ClassAd which
926 represents the the condor_schedd daemon running on the access
927 point, rather than a remote machine. Like all commands in the
928 submit description file, if multiple requirements commands
929 are present, all but the last one are ignored. By default,
930 condor_submit appends the following clauses to the require‐
931 ments expression:
932 1. Arch and OpSys are set equal to the Arch and OpSys of the
934 submit machine. In other words: unless you request other‐
935 wise, HTCondor will give your job machines with the same
936 architecture and operating system version as the machine
937 running condor_submit.
938 2. Cpus >= RequestCpus, if the job ClassAd attribute Re‐
940 questCpus is defined.
941 3. Disk >= RequestDisk, if the job ClassAd attribute Request‐
943 Disk is defined. Otherwise, Disk >= DiskUsage is appended
944 to the requirements. The DiskUsage attribute is initial‐
945 ized to the size of the executable plus the size of any
946 files specified in a transfer_input_files command. It ex‐
947 ists to ensure there is enough disk space on the target
948 machine for HTCondor to copy over both the executable and
949 needed input files. The DiskUsage attribute represents the
950 maximum amount of total disk space required by the job in
951 kilobytes. HTCondor automatically updates the DiskUsage
952 attribute approximately every 20 minutes while the job
953 runs with the amount of space being used by the job on the
954 execute machine.
955 4. Memory >= RequestMemory, if the job ClassAd attribute Re‐
957 questMemory is defined.
958 5. If Universe is set to Vanilla, FileSystemDomain is set
960 equal to the access point's FileSystemDomain.
961 View the requirements of a job which has already been submit‐
963 ted (along with everything else about the job ClassAd) with
964 the command condor_q -l; see the command reference for
965 condor_q. Also, see the HTCondor Users Manual for complete
966 information on the syntax and available attributes that can
967 be used in the ClassAd expression.
968 FILE TRANSFER COMMANDS
970 dont_encrypt_input_files = < file1,file2,file... >
974 A comma and/or space separated list of input files that are
975 not to be network encrypted when transferred with the file
976 transfer mechanism. Specification of files in this manner
977 overrides configuration that would use encryption. Each input
978 file must also be in the list given by transfer_input_files
979 . When a path to an input file or directory is specified,
980 this specifies the path to the file on the submit side. A
981 single wild card character (*) may be used in each file name.
982 dont_encrypt_output_files = < file1,file2,file... >
987 A comma and/or space separated list of output files that are
988 not to be network encrypted when transferred back with the
989 file transfer mechanism. Specification of files in this man‐
990 ner overrides configuration that would use encryption. The
991 output file(s) must also either be in the list given by
992 transfer_output_files or be discovered and to be transferred
993 back with the file transfer mechanism. When a path to an out‐
994 put file or directory is specified, this specifies the path
995 to the file on the execute side. A single wild card character
996 (*) may be used in each file name.
997 encrypt_execute_directory = <True | False>
1001 Defaults to False. If set to True, HTCondor will encrypt the
1002 contents of the remote scratch directory of the machine where
1003 the job is executed. This encryption is transparent to the
1004 job itself, but ensures that files left behind on the local
1005 disk of the execute machine, perhaps due to a system crash,
1006 will remain private. In addition, condor_submit will append
1007 to the job's requirements expression
1008 && (TARGET.HasEncryptExecuteDirectory)
1010 to ensure the job is matched to a machine that is capable of
1012 encrypting the contents of the execute directory. This sup‐
1013 port is limited to Windows platforms that use the NTFS file
1014 system and Linux platforms with the ecryptfs-utils package
1015 installed.
1016 encrypt_input_files = < file1,file2,file... >
1021 A comma and/or space separated list of input files that are
1022 to be network encrypted when transferred with the file trans‐
1023 fer mechanism. Specification of files in this manner over‐
1024 rides configuration that would not use encryption. Each input
1025 file must also be in the list given by transfer_input_files
1026 . When a path to an input file or directory is specified,
1027 this specifies the path to the file on the submit side. A
1028 single wild card character (*) may be used in each file name.
1029 The method of encryption utilized will be as agreed upon in
1030 security negotiation; if that negotiation failed, then the
1031 file transfer mechanism must also fail for files to be net‐
1032 work encrypted.
1033 encrypt_output_files = < file1,file2,file... >
1038 A comma and/or space separated list of output files that are
1039 to be network encrypted when transferred back with the file
1040 transfer mechanism. Specification of files in this manner
1041 overrides configuration that would not use encryption. The
1042 output file(s) must also either be in the list given by
1043 transfer_output_files or be discovered and to be transferred
1044 back with the file transfer mechanism. When a path to an out‐
1045 put file or directory is specified, this specifies the path
1046 to the file on the execute side. A single wild card character
1047 (*) may be used in each file name. The method of encryption
1048 utilized will be as agreed upon in security negotiation; if
1049 that negotiation failed, then the file transfer mechanism
1050 must also fail for files to be network encrypted.
1051 erase_output_and_error_on_restart
1055 If false, and when_to_transfer_output is ON_EXIT_OR_EVICT,
1056 HTCondor will append to the output and error logs rather than
1057 erase (truncate) them when the job restarts.
1058 max_transfer_input_mb = <ClassAd Integer Expression>
1062 This integer expression specifies the maximum allowed total
1063 size in MiB of the input files that are transferred for a
1064 job. This expression does not apply to grid universe or files
1065 transferred via file transfer plug-ins. The expression may
1066 refer to attributes of the job. The special value -1 indi‐
1067 cates no limit. If not defined, the value set by configura‐
1068 tion variable MAX_TRANSFER_INPUT_MB is used. If the ob‐
1069 served size of all input files at submit time is larger than
1070 the limit, the job will be immediately placed on hold with a
1071 HoldReasonCode value of 32. If the job passes this initial
1072 test, but the size of the input files increases or the limit
1073 decreases so that the limit is violated, the job will be
1074 placed on hold at the time when the file transfer is at‐
1075 tempted.
1076 max_transfer_output_mb = <ClassAd Integer Expression>
1080 This integer expression specifies the maximum allowed total
1081 size in MiB of the output files that are transferred for a
1082 job. This expression does not apply to grid universe or files
1083 transferred via file transfer plug-ins. The expression may
1084 refer to attributes of the job. The special value -1 indi‐
1085 cates no limit. If not set, the value set by configuration
1086 variable MAX_TRANSFER_OUTPUT_MB is used. If the total size
1087 of the job's output files to be transferred is larger than
1088 the limit, the job will be placed on hold with a HoldReason‐
1089 Code value of 33. The output will be transferred up to the
1090 point when the limit is hit, so some files may be fully
1091 transferred, some partially, and some not at all.
1092 output_destination = <destination-URL>
1097 When present, defines a URL that specifies both a plug-in and
1098 a destination for the transfer of the entire output sandbox
1099 or a subset of output files as specified by the submit com‐
1100 mand transfer_output_files . The plug-in does the transfer
1101 of files, and no files are sent back to the access point. The
1102 HTCondor Administrator's manual has full details.
1103 should_transfer_files = <YES | NO | IF_NEEDED >
1107 The should_transfer_files setting is used to define if HTCon‐
1108 dor should transfer files to and from the remote machine
1109 where the job runs. The file transfer mechanism is used to
1110 run jobs on machines which do not have a shared file system
1111 with the submit machine. should_transfer_files equal to YES
1112 will cause HTCondor to always transfer files for the job. NO
1113 disables HTCondor's file transfer mechanism. IF_NEEDED will
1114 not transfer files for the job if it is matched with a re‐
1115 source in the same FileSystemDomain as the access point (and
1116 therefore, on a machine with the same shared file system). If
1117 the job is matched with a remote resource in a different
1118 FileSystemDomain, HTCondor will transfer the necessary files.
1119 For more information about this and other settings related to
1121 transferring files, see the HTCondor User's manual section on
1122 the file transfer mechanism.
1123 Note that should_transfer_files is not supported for jobs
1125 submitted to the grid universe.
1126 skip_filechecks = <True | False>
1130 When True, file permission checks for the submitted job are
1131 disabled. When False, file permissions are checked; this is
1132 the behavior when this command is not present in the submit
1133 description file. File permissions are checked for read per‐
1134 missions on all input files, such as those defined by com‐
1135 mands input and transfer_input_files , and for write per‐
1136 mission to output files, such as a log file defined by log
1137 and output files defined with output or transfer_out‐
1138 put_files .
1139 stream_error = <True | False>
1143 If True, then stderr is streamed back to the machine from
1144 which the job was submitted. If False, stderr is stored lo‐
1145 cally and transferred back when the job completes. This com‐
1146 mand is ignored if the job ClassAd attribute TransferErr is
1147 False. The default value is False. This command must be used
1148 in conjunction with error , otherwise stderr will sent to
1149 /dev/null on Unix machines and ignored on Windows machines.
1150 stream_input = <True | False>
1154 If True, then stdin is streamed from the machine on which the
1155 job was submitted. The default value is False. The command is
1156 only relevant for jobs submitted to the vanilla or java uni‐
1157 verses, and it is ignored by the grid universe. This command
1158 must be used in conjunction with input , otherwise stdin
1159 will be /dev/null on Unix machines and ignored on Windows ma‐
1160 chines.
1161 stream_output = <True | False>
1165 If True, then stdout is streamed back to the machine from
1166 which the job was submitted. If False, stdout is stored lo‐
1167 cally and transferred back when the job completes. This com‐
1168 mand is ignored if the job ClassAd attribute TransferOut is
1169 False. The default value is False. This command must be used
1170 in conjunction with output , otherwise stdout will sent to
1171 /dev/null on Unix machines and ignored on Windows machines.
1172 transfer_executable = <True | False>
1176 This command is applicable to jobs submitted to the grid and
1177 vanilla universes. If transfer_executable is set to False,
1178 then HTCondor looks for the executable on the remote machine,
1179 and does not transfer the executable over. This is useful for
1180 an already pre-staged executable; HTCondor behaves more like
1181 rsh. The default value is True.
1182 transfer_input_files = < file1,file2,file... >
1186 A comma-delimited list of all the files and directories to be
1187 transferred into the working directory for the job, before
1188 the job is started. By default, the file specified in the ex‐
1189 ecutable command and any file specified in the input com‐
1190 mand (for example, stdin) are transferred.
1191 When a path to an input file or directory is specified, this
1193 specifies the path to the file on the submit side. The file
1194 is placed in the job's temporary scratch directory on the ex‐
1195 ecute side, and it is named using the base name of the origi‐
1196 nal path. For example, /path/to/input_file becomes input_file
1197 in the job's scratch directory.
1198 When a directory is specified, the behavior depends on
1200 whether there is a trailing path separator character. When a
1201 directory is specified with a trailing path separator, it is
1202 as if each of the items within the directory were listed in
1203 the transfer list. Therefore, the contents are transferred,
1204 but the directory itself is not. When there is no trailing
1205 path separator, the directory itself is transferred with all
1206 of its contents inside it. On platforms such as Windows
1207 where the path separator is not a forward slash (/), a trail‐
1208 ing forward slash is treated as equivalent to a trailing path
1209 separator. An example of an input directory specified with a
1210 trailing forward slash is input_data/.
1211 For grid universe jobs other than HTCondor-C, the transfer of
1213 directories is not currently supported.
1214 Symbolic links to files are transferred as the files they
1216 point to. Transfer of symbolic links to directories is not
1217 currently supported.
1218 For vanilla and vm universe jobs only, a file may be speci‐
1220 fied by giving a URL, instead of a file name. The implementa‐
1221 tion for URL transfers requires both configuration and avail‐
1222 able plug-in.
1223 If you have a plugin which handles https:// URLs (and HTCon‐
1225 dor ships with one enabled), HTCondor supports pre-signing S3
1226 URLs. This allows you to specify S3 URLs for this command,
1227 for transfer_output_remaps, and for output_destination. By
1228 pre-signing the URLs on the submit node, HTCondor avoids
1229 transferring your S3 credentials to the execute node. You
1230 must specify aws_access_key_id_file and aws_secret_ac‐
1231 cess_key_file; you may specify aws_region, if necessary; see
1232 below. To use the S3 service provided by AWS, use S3 URLs of
1233 the following forms:
1234 # For older buckets that aren't region-specific.
1236 s3://<bucket>/<key>
1237 # For newer, region-specific buckets.
1239 s3://<bucket>.s3.<region>.amazonaws.com/<key>
1240 To use other S3 services, where <host> must contain a .:
1242 s3://<host>/<key>
1244 # If necessary
1246 aws_region = <region>
1247 You may specify the corresponding access key ID and secret
1249 access key with s3_access_key_id_file and s3_secret_ac‐
1250 cess_key_file if you prefer (which may reduce confusion, if
1251 you're not using AWS).
1252 If you must access S3 using temporary credentials, you may
1254 specify the temporary credentials using aws_ac‐
1255 cess_key_id_file and aws_secret_access_key_file for the files
1256 containing the corresponding temporary token, and +EC2Ses‐
1257 sionToken for the file containing the session token.
1258 Temporary credentials have a limited lifetime. If you are
1260 using S3 only to download input files, the job must start be‐
1261 fore the credentials expire. If you are using S3 to upload
1262 output files, the job must finish before the credentials ex‐
1263 pire. HTCondor does not know when the credentials will ex‐
1264 pire; if they do so before they are needed, file transfer
1265 will fail.
1266 HTCondor does not presently support transferring entire buck‐
1268 ets or directories from S3.
1269 HTCondor supports Google Cloud Storage URLs -- gs:// -- via
1271 Google's "interoperability" API. You may specify gs:// URLs
1272 as if they were s3:// URLs, and they work the same way. You
1273 may specify the corresponding access key ID and secret access
1274 key with gs_access_key_id_file and gs_secret_access_key_file
1275 if you prefer (which may reduce confusion).
1276 Note that (at present), you may not provide more than one set
1278 of credentials for s3:// or gs:// file transfer; this implies
1279 that all such URLs download from or upload to the same ser‐
1280 vice.
1281 transfer_output_files = < file1,file2,file... >
1285 This command forms an explicit list of output files and di‐
1286 rectories to be transferred back from the temporary working
1287 directory on the execute machine to the access point. If
1288 there are multiple files, they must be delimited with commas.
1289 Setting transfer_output_files to the empty string ("") means
1290 that no files are to be transferred.
1291 For HTCondor-C jobs and all other non-grid universe jobs, if
1293 transfer_output_files is not specified, HTCondor will auto‐
1294 matically transfer back all files in the job's temporary
1295 working directory which have been modified or created by the
1296 job. Subdirectories are not scanned for output, so if output
1297 from subdirectories is desired, the output list must be ex‐
1298 plicitly specified. For grid universe jobs other than HTCon‐
1299 dor-C, desired output files must also be explicitly listed.
1300 Another reason to explicitly list output files is for a job
1301 that creates many files, and the user wants only a subset
1302 transferred back.
1303 For grid universe jobs other than with grid type condor, to
1305 have files other than standard output and standard error
1306 transferred from the execute machine back to the access
1307 point, do use transfer_output_files, listing all files to be
1308 transferred. These files are found on the execute machine in
1309 the working directory of the job.
1310 When a path to an output file or directory is specified, it
1312 specifies the path to the file on the execute side. As a des‐
1313 tination on the submit side, the file is placed in the job's
1314 initial working directory, and it is named using the base
1315 name of the original path. For example, path/to/output_file
1316 becomes output_file in the job's initial working directory.
1317 The name and path of the file that is written on the submit
1318 side may be modified by using transfer_output_remaps . Note
1319 that this remap function only works with files but not with
1320 directories.
1321 When a directory is specified, the behavior depends on
1323 whether there is a trailing path separator character. When a
1324 directory is specified with a trailing path separator, it is
1325 as if each of the items within the directory were listed in
1326 the transfer list. Therefore, the contents are transferred,
1327 but the directory itself is not. When there is no trailing
1328 path separator, the directory itself is transferred with all
1329 of its contents inside it. On platforms such as Windows
1330 where the path separator is not a forward slash (/), a trail‐
1331 ing forward slash is treated as equivalent to a trailing path
1332 separator. An example of an input directory specified with a
1333 trailing forward slash is input_data/.
1334 For grid universe jobs other than HTCondor-C, the transfer of
1336 directories is not currently supported.
1337 Symbolic links to files are transferred as the files they
1339 point to. Transfer of symbolic links to directories is not
1340 currently supported.
1341 transfer_checkpoint_files = < file1,file2,file3... >
1345 If present, this command defines the list of files and/or di‐
1346 rectories which constitute the job's checkpoint. When the
1347 job successfully checkpoints -- see checkpoint_exit_code --
1348 these files will be transferred to the submit node's spool.
1349 If this command is absent, the output is transferred instead.
1351 If no files or directories are specified, nothing will be
1353 transferred. This is generally not useful.
1354 The list is interpreted like transfer_output_files, but there
1356 is no corresponding remaps command.
1357 preserve_relative_paths = < True | False >
1361 For vanilla and Docker -universe jobs (and others that use
1362 the shadow), this command modifies the behavior of the file
1363 transfer commands. When set to true, the destination for an
1364 entry that is a relative path in a file transfer list becomes
1365 its relative path, not its basename. For example, in‐
1366 put_data/b (and its contents, if it is a directory) will be
1367 transferred to input_data/b, not b. This applies to the in‐
1368 put, output, and checkpoint lists.
1369 Trailing slashes are ignored when preserve_relative_paths is
1371 set.
1372 transfer_output_remaps = < " name = newname ; name2 = newname2 ...
1376 ">
1377 This specifies the name (and optionally path) to use when
1378 downloading output files from the completed job. Normally,
1379 output files are transferred back to the initial working di‐
1380 rectory with the same name they had in the execution direc‐
1381 tory. This gives you the option to save them with a different
1382 path or name. If you specify a relative path, the final path
1383 will be relative to the job's initial working directory, and
1384 HTCondor will create directories as necessary to transfer the
1385 file.
1386 name describes an output file name produced by your job, and
1388 newname describes the file name it should be downloaded to.
1389 Multiple remaps can be specified by separating each with a
1390 semicolon. If you wish to remap file names that contain
1391 equals signs or semicolons, these special characters may be
1392 escaped with a backslash. You cannot specify directories to
1393 be remapped.
1394 Note that whether an output file is transferred is controlled
1396 by transfer_output_files. Listing a file in transfer_out‐
1397 put_remaps is not sufficient to cause it to be transferred.
1398 transfer_plugins = < tag=plugin ; tag2,tag3=plugin2 ... >
1402 Specifies the file transfer plugins (see Additional kinds of
1403 file and credential transfer) that should be transferred
1404 along with the input files prior to invoking file transfer
1405 plugins for files specified in transfer_input_files. tag
1406 should be a URL prefix that is used in transfer_input_files,
1407 and plugin is the path to a file transfer plugin that will
1408 handle that type of URL transfer.
1409 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT | ON_SUCCESS
1413 >
1414 Setting when_to_transfer_output to ON_EXIT will cause HTCon‐
1415 dor to transfer the job's output files back to the submitting
1416 machine when the job completes (exits on its own). If a job
1417 is evicted and started again, the subsequent execution will
1418 start with only the executable and input files in the scratch
1419 directory sandbox. If transfer_output_files is not set, HT‐
1420 Condor considers all new files in the sandbox's top-level di‐
1421 rectory to be the output; subdirectories and their contents
1422 will not be transferred.
1423 Setting when_to_transfer_output to ON_EXIT_OR_EVICT will
1425 cause HTCondor to transfer the job's output files when the
1426 job completes (exits on its own) and when the job is evicted.
1427 When the job is evicted, HTCondor will transfer the output
1428 files to a temporary directory on the submit node (determined
1429 by the SPOOL configuration variable). When the job restarts,
1430 these files will be transferred instead of the input files.
1431 If transfer_output_files is not set, HTCondor considers all
1432 files in the sandbox's top-level directory to be the output;
1433 subdirectories and their contents will not be transferred.
1434 Setting when_to_transfer_output to ON_SUCCESS will cause HT‐
1436 Condor to transfer the job's output files when the job com‐
1437 pletes successfully. Success is defined by the suc‐
1438 cess_exit_code command, which must be set, even if the suc‐
1439 cessful value is the default 0. If transfer_output_files is
1440 not set, HTCondor considers all new files in the sandbox's
1441 top-level directory to be the output; subdirectories and
1442 their contents will not be transferred.
1443 In all three cases, the job will go on hold if transfer_out‐
1445 put_files specifies a file which does not exist at transfer
1446 time.
1447 aws_access_key_id_file, s3_access_key_id_file
1452 One of these commands is required if you specify an s3://
1453 URL; they specify the file containing the access key ID (and
1454 only the access key ID) used to pre-sign the URLs. Use only
1455 one.
1456 aws_secret_access_key_file, s3_secret_access_key_file
1461 One of these commands is required if you specify an s3://
1462 URL; they specify the file containing the secret access key
1463 (and only the secret access key) used to pre-sign the URLs.
1464 Use only one.
1465 aws_region
1469 Optional if you specify an S3 URL (and ignored otherwise),
1470 this command specifies the region to use if one is not speci‐
1471 fied in the URL.
1472 gs_access_key_id_file
1476 Required if you specify a gs:// URLs, this command specifies
1477 the file containing the access key ID (and only the access
1478 key ID) used to pre-sign the URLs.
1479 gs_secret_access_key_file
1483 Required if you specify a gs:// URLs, this command specifies
1484 the file containing the secret access key (and only the se‐
1485 cret access key) used to pre-sign the URLs.
1486 POLICY COMMANDS
1488 allowed_execute_duration = <integer>
1491 The longest time for which a job may be executing. Jobs
1492 which exceed this duration will go on hold. This time does
1493 not include file-transfer time. Jobs which self-checkpoint
1494 have this long to write out each checkpoint.
1495 This attribute is intended to help minimize the time wasted
1497 by jobs which may erroneously run forever.
1498 allowed_job_duration = <integer>
1502 The longest time for which a job may continuously be in the
1503 running state. Jobs which exceed this duration will go on
1504 hold. Exiting the running state resets the job duration used
1505 by this command.
1506 This command is intended to help minimize the time wasted by
1508 jobs which may erroneously run forever.
1509 max_retries = <integer>
1513 The maximum number of retries allowed for this job (must be
1514 non-negative). If the job fails (does not exit with the suc‐
1515 cess_exit_code exit code) it will be retried up to max_re‐
1516 tries times (unless retries are ceased because of the
1517 retry_until command). If max_retries is not defined, and ei‐
1518 ther retry_until or success_exit_code is, the value of DE‐
1519 FAULT_JOB_MAX_RETRIES will be used for the maximum number of
1520 retries.
1521 The combination of the max_retries, retry_until, and suc‐
1523 cess_exit_code commands causes an appropriate OnExitRemove
1524 expression to be automatically generated. If retry command(s)
1525 and on_exit_remove are both defined, the OnExitRemove expres‐
1526 sion will be generated by OR'ing the expression specified in
1527 OnExitRemove and the expression generated by the retry com‐
1528 mands.
1529 retry_until <Integer | ClassAd Boolean Expression>
1533 An integer value or boolean expression that prevents further
1534 retries from taking place, even if max_retries have not been
1535 exhausted. If retry_until is an integer, the job exiting
1536 with that exit code will cause retries to cease. If retry_un‐
1537 til is a ClassAd expression, the expression evaluating to
1538 True will cause retries to cease. For example, if you only
1539 want to retry exit codes 17, 34, and 81:
1540 max_retries = 5
1542 retry_until = !member( ExitCode, {17, 34, 81} )
1543 success_exit_code = <integer>
1547 The exit code that is considered successful for this job. De‐
1548 faults to 0 if not defined.
1549 Note: non-zero values of success_exit_code should generally
1551 not be used for DAG node jobs, unless when_to_transfer_output
1552 is set to ON_SUCCESS in order to avoid failed jobs going on
1553 hold.
1554 At the present time, condor_dagman does not take into account
1556 the value of success_exit_code. This means that, if suc‐
1557 cess_exit_code is set to a non-zero value, condor_dagman will
1558 consider the job failed when it actually succeeds. For sin‐
1559 gle-proc DAG node jobs, this can be overcome by using a POST
1560 script that takes into account the value of success_exit_code
1561 (although this is not recommended). For multi-proc DAG node
1562 jobs, there is currently no way to overcome this limitation.
1563 checkpoint_exit_code = <integer>
1567 The exit code which indicates that the executable has exited
1568 after successfully taking a checkpoint. The checkpoint will
1569 transferred and the executable restarted. See users-man‐
1570 ual/self-checkpointing-applications:Self-Checkpointing Appli‐
1571 cations for details.
1572 hold = <True | False>
1576 If hold is set to True, then the submitted job will be placed
1577 into the Hold state. Jobs in the Hold state will not run un‐
1578 til released by condor_release. Defaults to False.
1579 keep_claim_idle = <integer>
1583 An integer number of seconds that a job requests the con‐
1584 dor_schedd to wait before releasing its claim after the job
1585 exits or after the job is removed.
1586 The process by which the condor_schedd claims a condor_startd
1588 is somewhat time-consuming. To amortize this cost, the con‐
1589 dor_schedd tries to reuse claims to run subsequent jobs, af‐
1590 ter a job using a claim is done. However, it can only do this
1591 if there is an idle job in the queue at the moment the previ‐
1592 ous job completes. Sometimes, and especially for the node
1593 jobs when using DAGMan, there is a subsequent job about to be
1594 submitted, but it has not yet arrived in the queue when the
1595 previous job completes. As a result, the condor_schedd re‐
1596 leases the claim, and the next job must wait an entire nego‐
1597 tiation cycle to start. When this submit command is defined
1598 with a non-negative integer, when the job exits, the con‐
1599 dor_schedd tries as usual to reuse the claim. If it cannot,
1600 instead of releasing the claim, the condor_schedd keeps the
1601 claim until either the number of seconds given as a parame‐
1602 ter, or a new job which matches that claim arrives, whichever
1603 comes first. The condor_startd in question will remain in the
1604 Claimed/Idle state, and the original job will be "charged"
1605 (in terms of priority) for the time in this state.
1606 leave_in_queue = <ClassAd Boolean Expression>
1610 When the ClassAd Expression evaluates to True, the job is not
1611 removed from the queue upon completion. This allows the user
1612 of a remotely spooled job to retrieve output files in cases
1613 where HTCondor would have removed them as part of the cleanup
1614 associated with completion. The job will only exit the queue
1615 once it has been marked for removal (via condor_rm, for exam‐
1616 ple) and the leave_in_queue expression has become False.
1617 leave_in_queue defaults to False.
1618 As an example, if the job is to be removed once the output is
1620 retrieved with condor_transfer_data, then use
1621 leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED) ||\
1623 (StageOutFinish == 0))
1624 next_job_start_delay = <ClassAd Boolean Expression>
1628 This expression specifies the number of seconds to delay af‐
1629 ter starting up this job before the next job is started. The
1630 maximum allowed delay is specified by the HTCondor configura‐
1631 tion variable MAX_NEXT_JOB_START_DELAY
1632 , which defaults to 10 minutes. This command does not apply
1633 to scheduler or local universe jobs.
1634 This command has been historically used to implement a form
1636 of job start throttling from the job submitter's perspective.
1637 It was effective for the case of multiple job submission
1638 where the transfer of extremely large input data sets to the
1639 execute machine caused machine performance to suffer. This
1640 command is no longer useful, as throttling should be accom‐
1641 plished through configuration of the condor_schedd daemon.
1642 on_exit_hold = <ClassAd Boolean Expression>
1646 The ClassAd expression is checked when the job exits, and if
1647 True, places the job into the Hold state. If False (the de‐
1648 fault value when not defined), then nothing happens and the
1649 on_exit_remove expression is checked to determine if that
1650 needs to be applied.
1651 For example: Suppose a job is known to run for a minimum of
1653 an hour. If the job exits after less than an hour, the job
1654 should be placed on hold and an e-mail notification sent, in‐
1655 stead of being allowed to leave the queue.
1656 on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1658 This expression places the job on hold if it exits for any
1660 reason before running for an hour. An e-mail will be sent to
1661 the user explaining that the job was placed on hold because
1662 this expression became True.
1663 periodic_* expressions take precedence over on_exit_* expres‐
1665 sions, and *_hold expressions take precedence over a *_remove
1666 expressions.
1667 Only job ClassAd attributes will be defined for use by this
1669 ClassAd expression. This expression is available for the
1670 vanilla, java, parallel, grid, local and scheduler universes.
1671 on_exit_hold_reason = <ClassAd String Expression>
1675 When the job is placed on hold due to the on_exit_hold ex‐
1676 pression becoming True, this expression is evaluated to set
1677 the value of HoldReason in the job ClassAd. If this expres‐
1678 sion is UNDEFINED or produces an empty or invalid string, a
1679 default description is used.
1680 on_exit_hold_subcode = <ClassAd Integer Expression>
1684 When the job is placed on hold due to the on_exit_hold ex‐
1685 pression becoming True, this expression is evaluated to set
1686 the value of HoldReasonSubCode in the job ClassAd. The de‐
1687 fault subcode is 0. The HoldReasonCode will be set to 3,
1688 which indicates that the job went on hold due to a job policy
1689 expression.
1690 on_exit_remove = <ClassAd Boolean Expression>
1694 The ClassAd expression is checked when the job exits, and if
1695 True (the default value when undefined), then it allows the
1696 job to leave the queue normally. If False, then the job is
1697 placed back into the Idle state. If the user job runs under
1698 the vanilla universe, then the job restarts from the begin‐
1699 ning.
1700 For example, suppose a job occasionally segfaults, but
1702 chances are that the job will finish successfully if the job
1703 is run again with the same data. The on_exit_remove expres‐
1704 sion can cause the job to run again with the following com‐
1705 mand. Assume that the signal identifier for the segmentation
1706 fault is 11 on the platform where the job will be running.
1707 on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)
1709 This expression lets the job leave the queue if the job was
1711 not killed by a signal or if it was killed by a signal other
1712 than 11, representing segmentation fault in this example. So,
1713 if the exited due to signal 11, it will stay in the job
1714 queue. In any other case of the job exiting, the job will
1715 leave the queue as it normally would have done.
1716 As another example, if the job should only leave the queue if
1718 it exited on its own with status 0, this on_exit_remove ex‐
1719 pression works well:
1720 on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
1722 If the job was killed by a signal or exited with a non-zero
1724 exit status, HTCondor would leave the job in the queue to run
1725 again.
1726 periodic_* expressions take precedence over on_exit_* expres‐
1728 sions, and *_hold expressions take precedence over a *_remove
1729 expressions.
1730 Only job ClassAd attributes will be defined for use by this
1732 ClassAd expression.
1733 periodic_hold = <ClassAd Boolean Expression>
1737 This expression is checked periodically when the job is not
1738 in the Held state. If it becomes True, the job will be placed
1739 on hold. If unspecified, the default value is False.
1740 periodic_* expressions take precedence over on_exit_* expres‐
1742 sions, and *_hold expressions take precedence over a *_remove
1743 expressions.
1744 Only job ClassAd attributes will be defined for use by this
1746 ClassAd expression. Note that, by default, this expression is
1747 only checked once every 60 seconds. The period of these eval‐
1748 uations can be adjusted by setting the PERIODIC_EXPR_INTER‐
1749 VAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1750 configuration macros.
1751 periodic_hold_reason = <ClassAd String Expression>
1755 When the job is placed on hold due to the periodic_hold ex‐
1756 pression becoming True, this expression is evaluated to set
1757 the value of HoldReason in the job ClassAd. If this expres‐
1758 sion is UNDEFINED or produces an empty or invalid string, a
1759 default description is used.
1760 periodic_hold_subcode = <ClassAd Integer Expression>
1764 When the job is placed on hold due to the periodic_hold ex‐
1765 pression becoming true, this expression is evaluated to set
1766 the value of HoldReasonSubCode in the job ClassAd. The de‐
1767 fault subcode is 0. The HoldReasonCode will be set to 3,
1768 which indicates that the job went on hold due to a job policy
1769 expression.
1770 periodic_release = <ClassAd Boolean Expression>
1774 This expression is checked periodically when the job is in
1775 the Held state. If the expression becomes True, the job will
1776 be released. If the job was held via condor_hold (i.e. Hol‐
1777 dReasonCode is 1), then this expression is ignored.
1778 Only job ClassAd attributes will be defined for use by this
1780 ClassAd expression. Note that, by default, this expression is
1781 only checked once every 60 seconds. The period of these eval‐
1782 uations can be adjusted by setting the PERIODIC_EXPR_INTER‐
1783 VAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1784 configuration macros.
1785 periodic_remove = <ClassAd Boolean Expression>
1789 This expression is checked periodically. If it becomes True,
1790 the job is removed from the queue. If unspecified, the de‐
1791 fault value is False.
1792 See the Examples section of this manual page for an example
1794 of a periodic_remove expression.
1795 periodic_* expressions take precedence over on_exit_* expres‐
1797 sions, and *_hold expressions take precedence over a *_remove
1798 expressions. So, the periodic_remove expression takes prece‐
1799 dent over the on_exit_remove expression, if the two describe
1800 conflicting actions.
1801 Only job ClassAd attributes will be defined for use by this
1803 ClassAd expression. Note that, by default, this expression is
1804 only checked once every 60 seconds. The period of these eval‐
1805 uations can be adjusted by setting the PERIODIC_EXPR_INTER‐
1806 VAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1807 configuration macros.
1808 COMMANDS FOR THE GRID
1810 arc_application = <XML-string>
1813 For grid universe jobs of type arc, provides additional XML
1814 attributes under the <Application> section of the ARC ADL job
1815 description which are not covered by regular submit descrip‐
1816 tion file parameters.
1817 arc_resources = <XML-string>
1821 For grid universe jobs of type arc, provides additional XML
1822 attributes under the <Resources> section of the ARC ADL job
1823 description which are not covered by regular submit descrip‐
1824 tion file parameters.
1825 arc_rte = < rte1 option,rte2 >
1829 For grid universe jobs of type arc, provides a list of Run‐
1830 time Environment names that the job requires on the ARC sys‐
1831 tem. The list is comma-delimited. If a Runtime Environment
1832 name supports options, those can be provided after the name,
1833 separated by spaces. Runtime Environment names are defined
1834 by the ARC server.
1835 azure_admin_key = <pathname>
1839 For grid type azure jobs, specifies the path and file name of
1840 a file that contains an SSH public key. This key can be used
1841 to log into the administrator account of the instance via
1842 SSH.
1843 azure_admin_username = <account name>
1847 For grid type azure jobs, specifies the name of an adminis‐
1848 trator account to be created in the instance. This account
1849 can be logged into via SSH.
1850 azure_auth_file = <pathname>
1854 For grid type azure jobs, specifies a path and file name of
1855 the authorization file that grants permission for HTCondor to
1856 use the Azure account. If it's not defined, then HTCondor
1857 will attempt to use the default credentials of the Azure CLI
1858 tools.
1859 azure_image = <image id>
1863 For grid type azure jobs, identifies the disk image to be
1864 used for the boot disk of the instance. This image must al‐
1865 ready be registered within Azure.
1866 azure_location = <image id>
1870 For grid type azure jobs, identifies the location within
1871 Azure where the instance should be run. As an example, one
1872 current location is centralus.
1873 azure_size = <machine type>
1877 For grid type azure jobs, the hardware configuration that the
1878 virtual machine instance is to run on.
1879 batch_extra_submit_args = <command-line arguments>
1883 Used for batch grid universe jobs. Specifies additional com‐
1884 mand-line arguments to be given to the target batch system's
1885 job submission command.
1886 batch_project = <projectname>
1890 Used for batch grid universe jobs. Specifies the name of the
1891 PBS/LSF/SGE/SLURM project, account, or allocation that should
1892 be charged for the resources used by the job.
1893 batch_queue = <queuename>
1897 Used for batch grid universe jobs. Specifies the name of the
1898 PBS/LSF/SGE/SLURM job queue into which the job should be sub‐
1899 mitted. If not specified, the default queue is used. For a
1900 multi-cluster SLURM configuration, which cluster to use can
1901 be specified by supplying the name after an @ symbol. For
1902 example, to submit a job to the debug queue on cluster foo,
1903 you would use the value debug@foo.
1904 batch_runtime = <seconds>
1908 Used for batch grid universe jobs. Specifies a limit in sec‐
1909 onds on the execution time of the job. This limit is en‐
1910 forced by the PBS/LSF/SGE/SLURM scheduler.
1911 cloud_label_names = <name0,name1,name...>
1915 For grid type gce jobs, specifies the case of tag names that
1916 will be associated with the running instance. This is only
1917 necessary if a tag name case matters. By default the list
1918 will be automatically generated.
1919 cloud_label_<name> = <value>
1923 For grid type gce jobs, specifies a label and value to be as‐
1924 sociated with the running instance. The label name will be
1925 lower-cased; use cloud_label_names to change the case.
1926 delegate_job_GSI_credentials_lifetime = <seconds>
1930 Specifies the maximum number of seconds for which delegated
1931 proxies should be valid. The default behavior when this com‐
1932 mand is not specified is determined by the configuration
1933 variable DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME
1934 , which defaults to one day. A value of 0 indicates that the
1935 delegated proxy should be valid for as long as allowed by the
1936 credential used to create the proxy. This setting currently
1937 only applies to proxies delegated for non-grid jobs and for
1938 HTCondor-C jobs. This variable has no effect if the configu‐
1939 ration variable DELEGATE_JOB_GSI_CREDENTIALS
1940 is False, because in that case the job proxy is copied
1941 rather than delegated.
1942 ec2_access_key_id = <pathname>
1946 For grid type ec2 jobs, identifies the file containing the
1947 access key.
1948 ec2_ami_id = <EC2 xMI ID>
1952 For grid type ec2 jobs, identifies the machine image. Ser‐
1953 vices compatible with the EC2 Query API may refer to these
1954 with abbreviations other than AMI, for example EMI is valid
1955 for Eucalyptus.
1956 ec2_availability_zone = <zone name>
1960 For grid type ec2 jobs, specifies the Availability Zone that
1961 the instance should be run in. This command is optional, un‐
1962 less ec2_ebs_volumes is set. As an example, one current zone
1963 is us-east-1b.
1964 ec2_block_device_mapping = <block-device>:<kernel-device>,<block-de‐
1968 vice>:<kernel-device>, ...
1969 For grid type ec2 jobs, specifies the block device to kernel
1970 device mapping. This command is optional.
1971 ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device
1975 name>,...
1976 For grid type ec2 jobs, optionally specifies a list of Elas‐
1977 tic Block Store (EBS) volumes to be made available to the in‐
1978 stance and the device names they should have in the instance.
1979 ec2_elastic_ip = <elastic IP address>
1983 For grid type ec2 jobs, and optional specification of an
1984 Elastic IP address that should be assigned to this instance.
1985 ec2_iam_profile_arn = <IAM profile ARN>
1989 For grid type ec2 jobs, an Amazon Resource Name (ARN) identi‐
1990 fying which Identity and Access Management (IAM) (instance)
1991 profile to associate with the instance.
1992 ec2_iam_profile_name= <IAM profile name>
1996 For grid type ec2 jobs, a name identifying which Identity and
1997 Access Management (IAM) (instance) profile to associate with
1998 the instance.
1999 ec2_instance_type = <instance type>
2003 For grid type ec2 jobs, identifies the instance type. Differ‐
2004 ent services may offer different instance types, so no de‐
2005 fault value is set.
2006 ec2_keypair = <ssh key-pair name>
2010 For grid type ec2 jobs, specifies the name of an SSH key-pair
2011 that is already registered with the EC2 service. The associ‐
2012 ated private key can be used to ssh into the virtual machine
2013 once it is running.
2014 ec2_keypair_file = <pathname>
2018 For grid type ec2 jobs, specifies the complete path and file
2019 name of a file into which HTCondor will write an SSH key for
2020 use with ec2 jobs. The key can be used to ssh into the vir‐
2021 tual machine once it is running. If ec2_keypair is speci‐
2022 fied for a job, ec2_keypair_file is ignored.
2023 ec2_parameter_names = ParameterName1, ParameterName2, ...
2027 For grid type ec2 jobs, a space or comma separated list of
2028 the names of additional parameters to pass when instantiating
2029 an instance.
2030 ec2_parameter_<name> = <value>
2034 For grid type ec2 jobs, specifies the value for the corre‐
2035 spondingly named (instance instantiation) parameter. <name>
2036 is the parameter name specified in the submit command ec2_pa‐
2037 rameter_names , but with any periods replaced by under‐
2038 scores.
2039 ec2_secret_access_key = <pathname>
2043 For grid type ec2 jobs, specifies the path and file name con‐
2044 taining the secret access key.
2045 ec2_security_groups = group1, group2, ...
2049 For grid type ec2 jobs, defines the list of EC2 security
2050 groups which should be associated with the job.
2051 ec2_security_ids = id1, id2, ...
2055 For grid type ec2 jobs, defines the list of EC2 security
2056 group IDs which should be associated with the job.
2057 ec2_spot_price = <bid>
2061 For grid type ec2 jobs, specifies the spot instance bid,
2062 which is the most that the job submitter is willing to pay
2063 per hour to run this job.
2064 ec2_tag_names = <name0,name1,name...>
2068 For grid type ec2 jobs, specifies the case of tag names that
2069 will be associated with the running instance. This is only
2070 necessary if a tag name case matters. By default the list
2071 will be automatically generated.
2072 ec2_tag_<name> = <value>
2076 For grid type ec2 jobs, specifies a tag to be associated with
2077 the running instance. The tag name will be lower-cased; use
2078 ec2_tag_names to change the case.
2079 WantNameTag = <True | False>
2083 For grid type ec2 jobs, a job may request that its 'name' tag
2084 be (not) set by HTCondor. If the job does not otherwise spec‐
2085 ify any tags, not setting its name tag will eliminate a call
2086 by the EC2 GAHP, improving performance.
2087 ec2_user_data = <data>
2091 For grid type ec2 jobs, provides a block of data that can be
2092 accessed by the virtual machine. If both ec2_user_data and
2093 ec2_user_data_file are specified for a job, the two blocks of
2094 data are concatenated, with the data from this ec2_user_data
2095 submit command occurring first.
2096 ec2_user_data_file = <pathname>
2100 For grid type ec2 jobs, specifies a path and file name whose
2101 contents can be accessed by the virtual machine. If both
2102 ec2_user_data and ec2_user_data_file are specified for a job,
2103 the two blocks of data are concatenated, with the data from
2104 that ec2_user_data submit command occurring first.
2105 ec2_vpc_ip = <a.b.c.d>
2109 For grid type ec2 jobs, that are part of a Virtual Private
2110 Cloud (VPC), an optional specification of the IP address that
2111 this instance should have within the VPC.
2112 ec2_vpc_subnet = <subnet specification string>
2116 For grid type ec2 jobs, an optional specification of the Vir‐
2117 tual Private Cloud (VPC) that this instance should be a part
2118 of.
2119 gce_account = <account name>
2123 For grid type gce jobs, specifies the Google cloud services
2124 account to use. If this submit command isn't specified, then
2125 a random account from the authorization file given by
2126 gce_auth_file will be used.
2127 gce_auth_file = <pathname>
2131 For grid type gce jobs, specifies a path and file name of the
2132 authorization file that grants permission for HTCondor to use
2133 the Google account. If this command is not specified, then
2134 the default file of the Google command-line tools will be
2135 used.
2136 gce_image = <image id>
2140 For grid type gce jobs, the identifier of the virtual machine
2141 image representing the HTCondor job to be run. This virtual
2142 machine image must already be register with GCE and reside in
2143 Google's Cloud Storage service.
2144 gce_json_file = <pathname>
2148 For grid type gce jobs, specifies the path and file name of a
2149 file that contains JSON elements that should be added to the
2150 instance description submitted to the GCE service.
2151 gce_machine_type = <machine type>
2155 For grid type gce jobs, the long form of the URL that de‐
2156 scribes the machine configuration that the virtual machine
2157 instance is to run on.
2158 gce_metadata = <name=value,...,name=value>
2162 For grid type gce jobs, a comma separated list of name and
2163 value pairs that define metadata for a virtual machine in‐
2164 stance that is an HTCondor job.
2165 gce_metadata_file = <pathname>
2169 For grid type gce jobs, specifies a path and file name of the
2170 file that contains metadata for a virtual machine instance
2171 that is an HTCondor job. Within the file, each name and value
2172 pair is on its own line; so, the pairs are separated by the
2173 newline character.
2174 gce_preemptible = <True | False>
2178 For grid type gce jobs, specifies whether the virtual machine
2179 instance should be preemptible. The default is for the in‐
2180 stance to not be preemptible.
2181 grid_resource = <grid-type-string> <grid-specific-parameter-list>
2185 For each grid-type-string value, there are further type-spe‐
2186 cific values that must specified. This submit description
2187 file command allows each to be given in a space-separated
2188 list. Allowable grid-type-string values are arc, azure,
2189 batch, condor, ec2, and gce. The HTCondor manual chapter on
2190 Grid Computing details the variety of grid types.
2191 For a grid-type-string of batch, the single parameter is the
2193 name of the local batch system, and will be one of pbs, lsf,
2194 slurm, or sge.
2195 For a grid-type-string of condor, the first parameter is the
2197 name of the remote condor_schedd daemon. The second parameter
2198 is the name of the pool to which the remote condor_schedd
2199 daemon belongs.
2200 For a grid-type-string of ec2, one additional parameter spec‐
2202 ifies the EC2 URL.
2203 For a grid-type-string of arc, the single parameter is the
2205 name of the ARC resource to be used.
2206 transfer_error = <True | False>
2210 For jobs submitted to the grid universe only. If True, then
2211 the error output (from stderr) from the job is transferred
2212 from the remote machine back to the access point. The name of
2213 the file after transfer is given by the error command. If
2214 False, no transfer takes place (from the remote machine to
2215 access point), and the name of the file is given by the error
2216 command. The default value is True.
2217 transfer_input = <True | False>
2221 For jobs submitted to the grid universe only. If True, then
2222 the job input (stdin) is transferred from the machine where
2223 the job was submitted to the remote machine. The name of the
2224 file that is transferred is given by the input command. If
2225 False, then the job's input is taken from a pre-staged file
2226 on the remote machine, and the name of the file is given by
2227 the input command. The default value is True.
2228 For transferring files other than stdin, see transfer_in‐
2230 put_files .
2231 transfer_output = <True | False>
2235 For jobs submitted to the grid universe only. If True, then
2236 the output (from stdout) from the job is transferred from the
2237 remote machine back to the access point. The name of the file
2238 after transfer is given by the output command. If False, no
2239 transfer takes place (from the remote machine to access
2240 point), and the name of the file is given by the output
2241 command. The default value is True.
2242 For transferring files other than stdout, see transfer_out‐
2244 put_files .
2245 use_x509userproxy = <True | False>
2249 Set this command to True to indicate that the job requires an
2250 X.509 user proxy. If x509userproxy is set, then that file is
2251 used for the proxy. Otherwise, the proxy is looked for in the
2252 standard locations. If x509userproxy is set or if the job is
2253 a grid universe job of grid type arc, then the value of
2254 use_x509userproxy is forced to True. Defaults to False.
2255 x509userproxy = <full-pathname>
2259 Used to override the default path name for X.509 user cer‐
2260 tificates. The default location for X.509 proxies is the
2261 /tmp directory, which is generally a local file system. Set‐
2262 ting this value would allow HTCondor to access the proxy in a
2263 shared file system (for example, AFS). HTCondor will use the
2264 proxy specified in the submit description file first. If
2265 nothing is specified in the submit description file, it will
2266 use the environment variable X509_USER_PROXY. If that vari‐
2267 able is not present, it will search in the default location.
2268 Note that proxies are only valid for a limited time. Con‐
2269 dor_submit will not submit a job with an expired proxy, it
2270 will return an error. Also, if the configuration parameter
2271 CRED_MIN_TIME_LEFT is set to some number of seconds, and if
2272 the proxy will expire before that many seconds, condor_submit
2273 will also refuse to submit the job. That is, if
2274 CRED_MIN_TIME_LEFT is set to 60, condor_submit will refuse to
2275 submit a job whose proxy will expire 60 seconds from the time
2276 of submission.
2277 x509userproxy is relevant when the universe is vanilla, or
2279 when the universe is grid and the type of grid system is one
2280 of condor, or arc. Defining a value causes the proxy to be
2281 delegated to the execute machine. Further, VOMS attributes
2282 defined in the proxy will appear in the job ClassAd.
2283 use_scitokens = <True | False | Auto>
2287 Set this command to True to indicate that the job requires a
2288 scitoken. If scitokens_file is set, then that file is used
2289 for the scitoken filename. Otherwise, the the scitoken file‐
2290 name is looked for in the BEARER_TOKEN_FILE environment vari‐
2291 able. If scitokens_file is set then the value of use_scito‐
2292 kens defaults to True. If the filename is not defined in on
2293 one of these two places, then condor_submit will fail with an
2294 error message. Set this command to Auto to indicate that the
2295 job will use a scitoken if scitokens_file or the BEARER_TO‐
2296 KEN_FILE environment variable is set, but it will not be an
2297 error if no file is specified.
2298 scitokens_file = <full-pathname>
2302 Used to set the path to the file containing the scitoken that
2303 the job needs, or to override the path to the scitoken con‐
2304 tained in the BEARER_TOKEN_FILE environment variable.
2305 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2307 hold_kill_sig = <signal-number>
2310 For the scheduler universe only, signal-number is the sig‐
2311 nal delivered to the job when the job is put on hold with
2312 condor_hold. signal-number may be either the platform-spe‐
2313 cific name or value of the signal. If this command is not
2314 present, the value of kill_sig is used.
2315 jar_files = <file_list>
2319 Specifies a list of additional JAR files to include when us‐
2320 ing the Java universe. JAR files will be transferred along
2321 with the executable and automatically added to the classpath.
2322 java_vm_args = <argument_list>
2326 Specifies a list of additional arguments to the Java VM it‐
2327 self, When HTCondor runs the Java program, these are the ar‐
2328 guments that go before the class name. This can be used to
2329 set VM-specific arguments like stack size, garbage-collector
2330 arguments and initial property values.
2331 machine_count = <max>
2333 For the parallel universe, a single value (max) is required.
2334 It is neither a maximum or minimum, but the number of ma‐
2335 chines to be dedicated toward running the job.
2336 remove_kill_sig = <signal-number>
2340 For the scheduler universe only, signal-number is the sig‐
2341 nal delivered to the job when the job is removed with con‐
2342 dor_rm. signal-number may be either the platform-specific
2343 name or value of the signal. This example shows it both ways
2344 for a Linux signal:
2345 remove_kill_sig = SIGUSR1
2347 remove_kill_sig = 10
2348 If this command is not present, the value of kill_sig is
2350 used.
2351 COMMANDS FOR THE VM UNIVERSE
2353 vm_disk = file1:device1:permission1, file2:device2:permission2:for‐
2356 mat2, ...
2357 A list of comma separated disk files. Each disk file is spec‐
2358 ified by 4 colon separated fields. The first field is the
2359 path and file name of the disk file. The second field speci‐
2360 fies the device. The third field specifies permissions, and
2361 the optional fourth field specifies the image format. If a
2362 disk file will be transferred by HTCondor, then the first
2363 field should just be the simple file name (no path informa‐
2364 tion).
2365 An example that specifies two disk files:
2367 vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2369 vm_checkpoint = <True | False>
2373 A boolean value specifying whether or not to take check‐
2374 points. If not specified, the default value is False. In the
2375 current implementation, setting both vm_checkpoint and
2376 vm_networking to True does not yet work in all cases. Net‐
2377 working cannot be used if a vm universe job uses a checkpoint
2378 in order to continue execution after migration to another ma‐
2379 chine.
2380 vm_macaddr = <MACAddr>
2384 Defines that MAC address that the virtual machine's network
2385 interface should have, in the standard format of six groups
2386 of two hexadecimal digits separated by colons.
2387 vm_memory = <MBytes-of-memory>
2391 The amount of memory in MBytes that a vm universe job re‐
2392 quires.
2393 vm_networking = <True | False>
2397 Specifies whether to use networking or not. In the current
2398 implementation, setting both vm_checkpoint and vm_networking
2399 to True does not yet work in all cases. Networking cannot be
2400 used if a vm universe job uses a checkpoint in order to con‐
2401 tinue execution after migration to another machine.
2402 vm_networking_type = <nat | bridge >
2406 When vm_networking is True, this definition augments the
2407 job's requirements to match only machines with the specified
2408 networking. If not specified, then either networking type
2409 matches.
2410 vm_no_output_vm = <True | False>
2414 When True, prevents HTCondor from transferring output files
2415 back to the machine from which the vm universe job was sub‐
2416 mitted. If not specified, the default value is False.
2417 vm_type = <xen | kvm>
2421 Specifies the underlying virtual machine software that this
2422 job expects.
2423 xen_initrd = <image-file>
2427 When xen_kernel gives a file name for the kernel image to
2428 use, this optional command may specify a path to a ramdisk
2429 (initrd) image file. If the image file will be transferred by
2430 HTCondor, then the value should just be the simple file name
2431 (no path information).
2432 xen_kernel = <included | path-to-kernel>
2436 A value of included specifies that the kernel is included in
2437 the disk file. If not one of these values, then the value is
2438 a path and file name of the kernel to be used. If a kernel
2439 file will be transferred by HTCondor, then the value should
2440 just be the simple file name (no path information).
2441 xen_kernel_params = <string>
2445 A string that is appended to the Xen kernel command line.
2446 xen_root = <string>
2450 A string that is appended to the Xen kernel command line to
2451 specify the root device. This string is required when
2452 xen_kernel gives a path to a kernel. Omission for this re‐
2453 quired case results in an error message during submission.
2454 COMMANDS FOR THE DOCKER UNIVERSE
2456 docker_image = < image-name >
2459 Defines the name of the Docker image that is the basis for
2460 the docker container.
2461 docker_network_type = < host | none | custom_admin_defined_value>
2465 If docker_network_type is set to the string host, then the
2466 job is run using the host's network. If docker_network_type
2467 is set to the string none, then the job is run with no net‐
2468 work. If this is not set, each job gets a private network in‐
2469 terface. Some administrators may define site specific docker
2470 networks on a given worker node. When this is the case, ad‐
2471 ditional values may be valid here.
2472 docker_pull_policy = < always >
2476 if docker_pull_policy is set to always, when a docker uni‐
2477 verse job starts on a worker node, the option "--pull always"
2478 will be passed to the docker run command. This only impacts
2479 worker nodes which already have a locally cached version of
2480 the image. With this option, docker will always check with
2481 the repo to see if the cached version is out of date. This
2482 requires more network connectivity, and may cause docker hub
2483 to throttle future pull requests. It is generally recommened
2484 to never mutate docker image tag name, and avoid needing this
2485 option.
2486 container_service_names = <service-name>[, <service-name>]*
2490 A string- or comma- separated list of service names. Each
2491 service-name must have a corresponding <service-name>_con‐
2492 tainer_port command specifying a port number (an integer from
2493 0 to 65535). HTCondor will ask Docker to forward from a host
2494 port to the specified port inside the container. When Docker
2495 has done so, HTCondor will add an attribute to the job ad for
2496 each service, <service-name>HostPort, which contains the port
2497 number on the host forwarding to the corresponding service.
2498 COMMANDS FOR THE CONTAINER UNIVERSE
2500 container_image = < image-name >
2503 Defines the name of the container image. Can be a singularity
2504 .sif file, a singularity exploded directory, or a path to an
2505 image in a docker style repository
2506 container_target_dir = < path-to-directory-inside-container >
2510 Defines the working directory of the job inside the con‐
2511 tainer. Will be mapped to the scratch directory on the
2512 worker node.
2513 ADVANCED COMMANDS
2515 accounting_group = <accounting-group-name>
2518 Causes jobs to negotiate under the given accounting group.
2519 This value is advertised in the job ClassAd as AcctGroup. The
2520 HTCondor Administrator's manual contains more information
2521 about accounting groups.
2522 accounting_group_user = <accounting-group-user-name>
2526 Sets the name associated with this job to be used for re‐
2527 source usage accounting purposes, such as computation of
2528 fair-share priority and reporting via condor_userprio. If
2529 not set, defaults to the value of the job ClassAd attribute
2530 User. This value is advertised in the job ClassAd as Acct‐
2531 GroupUser.
2532 concurrency_limits = <string-list>
2536 A list of resources that this job needs. The resources are
2537 presumed to have concurrency limits placed upon them, thereby
2538 limiting the number of concurrent jobs in execution which
2539 need the named resource. Commas and space characters delimit
2540 the items in the list. Each item in the list is a string
2541 that identifies the limit, or it is a ClassAd expression that
2542 evaluates to a string, and it is evaluated in the context of
2543 machine ClassAd being considered as a match. Each item in the
2544 list also may specify a numerical value identifying the inte‐
2545 ger number of resources required for the job. The syntax
2546 follows the resource name by a colon character (:) and the
2547 numerical value. Details on concurrency limits are in the HT‐
2548 Condor Administrator's manual.
2549 concurrency_limits_expr = <ClassAd String Expression>
2553 A ClassAd expression that represents the list of resources
2554 that this job needs after evaluation. The ClassAd expression
2555 may specify machine ClassAd attributes that are evaluated
2556 against a matched machine. After evaluation, the list sets
2557 concurrency_limits.
2558 copy_to_spool = <True | False>
2562 If copy_to_spool is True, then condor_submit copies the exe‐
2563 cutable to the local spool directory before running it on a
2564 remote host. As copying can be quite time consuming and un‐
2565 necessary, the default value is False for all job universes.
2566 When False, condor_submit does not copy the executable to a
2567 local spool directory.
2568 coresize = <size>
2572 Should the user's program abort and produce a core file,
2573 coresize specifies the maximum size in bytes of the core file
2574 which the user wishes to keep. If coresize is not specified
2575 in the command file, this is set to 0 (meaning no core will
2576 be generated).
2577 cron_day_of_month = <Cron-evaluated Day>
2581 The set of days of the month for which a deferral time ap‐
2582 plies. The HTCondor User's manual section on Time Scheduling
2583 for Job Execution has further details.
2584 cron_day_of_week = <Cron-evaluated Day>
2588 The set of days of the week for which a deferral time ap‐
2589 plies. The HTCondor User's manual section on Time Scheduling
2590 for Job Execution has further details.
2591 cron_hour = <Cron-evaluated Hour>
2595 The set of hours of the day for which a deferral time ap‐
2596 plies. The HTCondor User's manual section on Time Scheduling
2597 for Job Execution has further details.
2598 cron_minute = <Cron-evaluated Minute>
2602 The set of minutes within an hour for which a deferral time
2603 applies. The HTCondor User's manual section on Time Schedul‐
2604 ing for Job Execution has further details.
2605 cron_month = <Cron-evaluated Month>
2609 The set of months within a year for which a deferral time ap‐
2610 plies. The HTCondor User's manual section on Time Scheduling
2611 for Job Execution has further details.
2612 cron_prep_time = <ClassAd Integer Expression>
2616 Analogous to deferral_prep_time . The number of seconds
2617 prior to a job's deferral time that the job may be matched
2618 and sent to an execution machine.
2619 cron_window = <ClassAd Integer Expression>
2623 Analogous to the submit command deferral_window . It allows
2624 cron jobs that miss their deferral time to begin execution.
2625 The HTCondor User's manual section on Time Scheduling for Job
2627 Execution has further details.
2628 dagman_log = <pathname>
2632 DAGMan inserts this command to specify an event log that it
2633 watches to maintain the state of the DAG. If the log com‐
2634 mand is not specified in the submit file, DAGMan uses the log
2635 command to specify the event log.
2636 deferral_prep_time = <ClassAd Integer Expression>
2640 The number of seconds prior to a job's deferral time that the
2641 job may be matched and sent to an execution machine.
2642 The HTCondor User's manual section on Time Scheduling for Job
2644 Execution has further details.
2645 deferral_time = <ClassAd Integer Expression>
2649 Allows a job to specify the time at which its execution is to
2650 begin, instead of beginning execution as soon as it arrives
2651 at the execution machine. The deferral time is an expression
2652 that evaluates to a Unix Epoch timestamp (the number of sec‐
2653 onds elapsed since 00:00:00 on January 1, 1970, Coordinated
2654 Universal Time). Deferral time is evaluated with respect to
2655 the execution machine. This option delays the start of execu‐
2656 tion, but not the matching and claiming of a machine for the
2657 job. If the job is not available and ready to begin execution
2658 at the deferral time, it has missed its deferral time. A job
2659 that misses its deferral time will be put on hold in the
2660 queue.
2661 The HTCondor User's manual section on Time Scheduling for Job
2663 Execution has further details.
2664 Due to implementation details, a deferral time may not be
2666 used for scheduler universe jobs.
2667 deferral_window = <ClassAd Integer Expression>
2671 The deferral window is used in conjunction with the defer‐
2672 ral_time command to allow jobs that miss their deferral time
2673 to begin execution.
2674 The HTCondor User's manual section on Time Scheduling for Job
2676 Execution has further details.
2677 description = <string>
2681 A string that sets the value of the job ClassAd attribute
2682 JobDescription. When set, tools which display the executable
2683 such as condor_q will instead use this string.
2684 email_attributes = <list-of-job-ad-attributes>
2688 A comma-separated list of attributes from the job ClassAd.
2689 These attributes and their values will be included in the
2690 e-mail notification of job completion.
2691 image_size = <size>
2695 Advice to HTCondor specifying the maximum virtual image size
2696 to which the job will grow during its execution. HTCondor
2697 will then execute the job only on machines which have enough
2698 resources, (such as virtual memory), to support executing the
2699 job. If not specified, HTCondor will automatically make a
2700 (reasonably accurate) estimate about the job's size and ad‐
2701 just this estimate as the program runs. If specified and un‐
2702 derestimated, the job may crash due to the inability to ac‐
2703 quire more address space; for example, if malloc() fails. If
2704 the image size is overestimated, HTCondor may have difficulty
2705 finding machines which have the required resources. size is
2706 specified in KiB. For example, for an image size of 8 MiB,
2707 size should be 8000.
2708 initialdir = <directory-path>
2712 Used to give jobs a directory with respect to file input and
2713 output. Also provides a directory (on the machine from which
2714 the job is submitted) for the job event log, when a full path
2715 is not specified.
2716 For vanilla universe jobs where there is a shared file sys‐
2718 tem, it is the current working directory on the machine where
2719 the job is executed.
2720 For vanilla or grid universe jobs where file transfer mecha‐
2722 nisms are utilized (there is not a shared file system), it is
2723 the directory on the machine from which the job is submitted
2724 where the input files come from, and where the job's output
2725 files go to.
2726 For scheduler universe jobs, it is the directory on the ma‐
2728 chine from which the job is submitted where the job runs; the
2729 current working directory for file input and output with re‐
2730 spect to relative path names.
2731 Note that the path to the executable is not relative to ini‐
2733 tialdir ; if it is a relative path, it is relative to the
2734 directory in which the condor_submit command is run.
2735 job_ad_information_attrs = <attribute-list>
2739 A comma-separated list of job ClassAd attribute names. The
2740 named attributes and their values are written to the job
2741 event log whenever any event is being written to the log.
2742 This implements the same thing as the configuration variable
2743 EVENT_LOG_INFORMATION_ATTRS (see the admin-manual/configura‐
2744 tion-macros:daemon logging configuration file entries page),
2745 but it applies to the job event log, instead of the system
2746 event log.
2747 job_lease_duration = <number-of-seconds>
2751 For vanilla, parallel, VM, and java universe jobs only, the
2752 duration in seconds of a job lease. The default value is
2753 2,400, or forty minutes. If a job lease is not desired, the
2754 value can be explicitly set to 0 to disable the job lease se‐
2755 mantics. The value can also be a ClassAd expression that
2756 evaluates to an integer. The HTCondor User's manual section
2757 on Special Environment Considerations has further details.
2758 job_machine_attrs = <attr1, attr2, ...>
2763 A comma and/or space separated list of machine attribute
2764 names that should be recorded in the job ClassAd in addition
2765 to the ones specified by the condor_schedd daemon's system
2766 configuration variable SYSTEM_JOB_MACHINE_ATTRS
2767 . When there are multiple run attempts, history of machine
2768 attributes from previous run attempts may be kept. The number
2769 of run attempts to store may be extended beyond the sys‐
2770 tem-specified history length by using the submit file command
2771 job_machine_attrs_history_length . A machine attribute
2772 named X will be inserted into the job ClassAd as an attribute
2773 named MachineAttrX0. The previous value of this attribute
2774 will be named MachineAttrX1, the previous to that will be
2775 named MachineAttrX2, and so on, up to the specified history
2776 length. A history of length 1 means that only MachineAttrX0
2777 will be recorded. The value recorded in the job ClassAd is
2778 the evaluation of the machine attribute in the context of the
2779 job ClassAd when the condor_schedd daemon initiates the start
2780 up of the job. If the evaluation results in an Undefined or
2781 Error result, the value recorded in the job ad will be Unde‐
2782 fined or Error, respectively.
2783 want_graceful_removal = <boolean expression>
2787 If true, this job will be given a chance to shut down cleanly
2788 when removed. The job will be given as much time as the ad‐
2789 ministrator of the execute resource allows, which may be
2790 none. The default is false. For details, see the configura‐
2791 tion setting GRACEFULLY_REMOVE_JOBS.
2792 kill_sig = <signal-number>
2796 When HTCondor needs to kick a job off of a machine, it will
2797 send the job the signal specified by signal-number . sig‐
2798 nal-number needs to be an integer which represents a valid
2799 signal on the execution machine. The default value is
2800 SIGTERM, which is the standard way to terminate a program in
2801 Unix.
2802 kill_sig_timeout = <seconds>
2806 This submit command should no longer be used as of HTCondor
2807 version 7.7.3; use job_max_vacate_time instead. If
2808 job_max_vacate_time is not defined, this defines the number
2809 of seconds that HTCondor should wait following the sending of
2810 the kill signal defined by kill_sig and forcibly killing
2811 the job. The actual amount of time between sending the signal
2812 and forcibly killing the job is the smallest of this value
2813 and the configuration variable KILLING_TIMEOUT
2814 , as defined on the execute machine.
2815 load_profile = <True | False>
2819 When True, loads the account profile of the dedicated run ac‐
2820 count for Windows jobs. May not be used with run_as_owner .
2821 log_xml = <True | False>
2825 If log_xml is True, then the job event log file will be
2826 written in ClassAd XML. If not specified, XML is not used.
2827 Note that the file is an XML fragment; it is missing the file
2828 header and footer. Do not mix XML and non-XML within a single
2829 file. If multiple jobs write to a single job event log file,
2830 ensure that all of the jobs specify this option in the same
2831 way.
2832 match_list_length = <integer value>
2836 Defaults to the value zero (0). When match_list_length is de‐
2837 fined with an integer value greater than zero (0), attributes
2838 are inserted into the job ClassAd. The maximum number of at‐
2839 tributes defined is given by the integer value. The job Clas‐
2840 sAds introduced are given as
2841 LastMatchName0 = "most-recent-Name"
2843 LastMatchName1 = "next-most-recent-Name"
2844 The value for each introduced ClassAd is given by the value
2846 of the Name attribute from the machine ClassAd of a previous
2847 execution (match). As a job is matched, the definitions for
2848 these attributes will roll, with LastMatchName1 becoming
2849 LastMatchName2, LastMatchName0 becoming LastMatchName1, and
2850 LastMatchName0 being set by the most recent value of the Name
2851 attribute.
2852 An intended use of these job attributes is in the require‐
2854 ments expression. The requirements can allow a job to prefer
2855 a match with either the same or a different resource than a
2856 previous match.
2857 job_max_vacate_time = <integer expression>
2861 An integer-valued expression (in seconds) that may be used to
2862 adjust the time given to an evicted job for gracefully shut‐
2863 ting down. If the job's setting is less than the machine's,
2864 the job's is used. If the job's setting is larger than the
2865 machine's, the result depends on whether the job has any ex‐
2866 cess retirement time. If the job has more retirement time
2867 left than the machine's max vacate time setting, then retire‐
2868 ment time will be converted into vacating time, up to the
2869 amount requested by the job.
2870 Setting this expression does not affect the job's resource
2872 requirements or preferences. For a job to only run on a ma‐
2873 chine with a minimum MachineMaxVacateTime, or to preferen‐
2874 tially run on such machines, explicitly specify this in the
2875 requirements and/or rank expressions.
2876 manifest = <True | False>
2880 For vanilla and Docker -universe jobs (and others that use
2881 the shadow), specifies if HTCondor (the starter) should pro‐
2882 duce a "manifest", which is directory containing three files:
2883 the list of files and directories at the top level of the
2884 sandbox when file transfer in completes (in), the same when
2885 file transfer out begins (out), and a dump of the environment
2886 set for the job (env).
2887 This feature is not presently available for Windows.
2889 manifest_dir = <directory name>
2893 For vanilla and Docker -universe jobs (and others that use
2894 the shadow), specifies the directory in which to record the
2895 manifest. Specifying this enables the creation of a mani‐
2896 fest. By default, the manifest directory is named <clus‐
2897 ter>_<proc>_manifest, to avoid conflicts.
2898 This feature is not presently available for Windows.
2900 max_job_retirement_time = <integer expression>
2904 An integer-valued expression (in seconds) that does nothing
2905 unless the machine that runs the job has been configured to
2906 provide retirement time. Retirement time is a grace period
2907 given to a job to finish when a resource claim is about to be
2908 preempted. The default behavior in many cases is to take as
2909 much retirement time as the machine offers, so this command
2910 will rarely appear in a submit description file.
2911 When a resource claim is to be preempted, this expression in
2913 the submit file specifies the maximum run time of the job (in
2914 seconds, since the job started). This expression has no ef‐
2915 fect, if it is greater than the maximum retirement time pro‐
2916 vided by the machine policy. If the resource claim is not
2917 preempted, this expression and the machine retirement policy
2918 are irrelevant. If the resource claim is preempted the job
2919 will be allowed to run until the retirement time expires, at
2920 which point it is hard-killed. The job will be soft-killed
2921 when it is getting close to the end of retirement in order to
2922 give it time to gracefully shut down. The amount of lead-time
2923 for soft-killing is determined by the maximum vacating time
2924 granted to the job.
2925 Any jobs running with nice_user priority have a default
2927 max_job_retirement_time of 0, so no retirement time is uti‐
2928 lized by default. In all other cases, no default value is
2929 provided, so the maximum amount of retirement time is uti‐
2930 lized by default.
2931 Setting this expression does not affect the job's resource
2933 requirements or preferences. For a job to only run on a ma‐
2934 chine with a minimum MaxJobRetirementTime, or to preferen‐
2935 tially run on such machines, explicitly specify this in the
2936 requirements and/or rank expressions.
2937 nice_user = <True | False>
2941 Normally, when a machine becomes available to HTCondor, HT‐
2942 Condor decides which job to run based upon user and job pri‐
2943 orities. Setting nice_user equal to True tells HTCondor not
2944 to use your regular user priority, but that this job should
2945 have last priority among all users and all jobs. So jobs sub‐
2946 mitted in this fashion run only on machines which no other
2947 non-nice_user job wants - a true bottom-feeder job! This is
2948 very handy if a user has some jobs they wish to run, but do
2949 not wish to use resources that could instead be used to run
2950 other people's HTCondor jobs. Jobs submitted in this fashion
2951 have an accounting group. The accounting group is config‐
2952 urable by setting NICE_USER_ACCOUNTING_GROUP_NAME which de‐
2953 faults to nice-user The default value is False.
2954 noop_job = <ClassAd Boolean Expression>
2958 When this boolean expression is True, the job is immediately
2959 removed from the queue, and HTCondor makes no attempt at run‐
2960 ning the job. The log file for the job will show a job sub‐
2961 mitted event and a job terminated event, along with an exit
2962 code of 0, unless the user specifies a different signal or
2963 exit code.
2964 noop_job_exit_code = <return value>
2968 When noop_job is in the submit description file and evalu‐
2969 ates to True, this command allows the job to specify the re‐
2970 turn value as shown in the job's log file job terminated
2971 event. If not specified, the job will show as having termi‐
2972 nated with status 0. This overrides any value specified with
2973 noop_job_exit_signal .
2974 noop_job_exit_signal = <signal number>
2978 When noop_job is in the submit description file and evalu‐
2979 ates to True, this command allows the job to specify the sig‐
2980 nal number that the job's log event will show the job having
2981 terminated with.
2982 remote_initialdir = <directory-path>
2986 The path specifies the directory in which the job is to be
2987 executed on the remote machine.
2988 rendezvousdir = <directory-path>
2992 Used to specify the shared file system directory to be used
2993 for file system authentication when submitting to a remote
2994 scheduler. Should be a path to a preexisting directory.
2995 run_as_owner = <True | False>
2999 A boolean value that causes the job to be run under the login
3000 of the submitter, if supported by the joint configuration of
3001 the submit and execute machines. On Unix platforms, this de‐
3002 faults to True, and on Windows platforms, it defaults to
3003 False. May not be used with load_profile . See the HTCondor
3004 manual Platform-Specific Information chapter for administra‐
3005 tive details on configuring Windows to support this option.
3006 stack_size = <size in bytes>
3010 This command applies only to Linux platforms. An integer
3011 number of bytes, representing the amount of stack space to be
3012 allocated for the job. This value replaces the default allo‐
3013 cation of stack space, which is unlimited in size.
3014 submit_event_notes = <note>
3018 A string that is appended to the submit event in the job's
3019 log file. For DAGMan jobs, the string DAG Node: and the
3020 node's name is automatically defined for submit_event_notes,
3021 causing the logged submit event to identify the DAG node job
3022 submitted.
3023 ulog_execute_attrs = <attribute-list>
3027 A comma-separated list of machine ClassAd attribute names.
3028 The named attributes and their values are written as part of
3029 the execution event in the job event log.
3030 use_oauth_services = <list of credential service names>
3034 A comma-separated list of credential-providing service names
3035 for which the job should be provided credentials for the job
3036 execution environment. The credential service providers must
3037 be configured by the pool admin.
3038 <credential_service_name>_oauth_permissions[_<handle>] = <scope>
3040 A string containing the scope(s) that should be requested for
3041 the credential named <credential_service_name>[_<handle>],
3042 where <handle> is optionally provided to differentiate be‐
3043 tween multiple credentials from the same credential service
3044 provider.
3045 <credential_service_name>_oauth_resource[_<handle>] = <resource>
3047 A string containing the resource (or "audience") that should
3048 be requested for the credential named <credential_ser‐
3049 vice_name>[_<handle>], where <handle> is optionally provided
3050 to differentiate between multiple credentials from the same
3051 credential service provider.
3052 MY.<attribute> = <value> or +<attribute> = <value>
3054 A macro that begins with MY. or a line that begins with a '+'
3055 (plus) character instructs condor_submit to insert the given
3056 attribute (without + or MY.) into the job ClassAd with the
3057 given value. The macro can be referenced in other submit
3058 statements by using $(MY.<attribute>). A +<attribute> is con‐
3059 verted to MY.<attribute> when the file is read.
3060 Note that setting an job attribute in this way should not be
3062 used in place of one of the specific commands listed above.
3063 Often, the command name does not directly correspond to an
3064 attribute name; furthermore, many submit commands result in
3065 actions more complex than simply setting an attribute or at‐
3066 tributes. See Job ClassAd Attributes for a list of HTCondor
3067 job attributes.
3068 MACROS AND COMMENTS
3070 In addition to commands, the submit description file can contain macros
3073 and comments.
3074 Macros Parameterless macros in the form of $(macro_name:default ini‐
3076 tial value) may be used anywhere in HTCondor submit descrip‐
3077 tion files to provide textual substitution at submit time.
3078 Macros can be defined by lines in the form of
3079 <macro_name> = <string>
3081 Several pre-defined macros are supplied by the submit de‐
3083 scription file parser. The $(Cluster) or $(ClusterId) macro
3084 supplies the value of the
3085 ClusterId job ClassAd attribute, and the $(Process) or
3087 $(ProcId) macro supplies the value of the ProcId job ClassAd
3088 attribute. The $(JobId) macro supplies the full job id. It
3089 is equivalent to $(ClusterId).$(ProcId). These macros are
3090 intended to aid in the specification of input/output files,
3091 arguments, etc., for clusters with lots of jobs, and/or could
3092 be used to supply an HTCondor process with its own cluster
3093 and process numbers on the command line.
3094 The $(Node) macro is defined for parallel universe jobs, and
3096 is especially relevant for MPI applications. It is a unique
3097 value assigned for the duration of the job that essentially
3098 identifies the machine (slot) on which a program is execut‐
3099 ing. Values assigned start at 0 and increase monotonically.
3100 The values are assigned as the parallel job is about to
3101 start.
3102 Recursive definition of macros is permitted. An example of a
3104 construction that works is the following:
3105 foo = bar
3107 foo = snap $(foo)
3108 As a result, foo = snap bar.
3110 Note that both left- and right- recursion works, so
3112 foo = bar
3114 foo = $(foo) snap
3115 has as its result foo = bar snap.
3117 The construction
3119 foo = $(foo) bar
3121 by itself will not work, as it does not have an initial base
3123 case. Mutually recursive constructions such as:
3124 B = bar
3126 C = $(B)
3127 B = $(C) boo
3128 will not work, and will fill memory with expansions.
3130 A default value may be specified, for use if the macro has no
3132 definition. Consider the example
3133 D = $(E:24)
3135 Where E is not defined within the submit description file,
3137 the default value 24 is used, resulting in
3138 D = 24
3140 This is useful for creating submit templates where values can
3142 be passed on the condor_submit command line, but that have a
3143 default value as well. In the above example, if you give a
3144 value for E on the command line like this
3145 condor_submit E=99 <submit-file>
3147 The value of 99 is used for E, resulting in
3149 D = 99
3151 To use the dollar sign character ($) as a literal, without
3155 macro expansion, use
3156 $(DOLLAR)
3158 In addition to the normal macro, there is also a special kind
3160 of macro called a substitution macro
3161 that allows the substitution of a machine ClassAd attribute
3162 value defined on the resource machine itself (gotten after a
3163 match to the machine has been made) into specific commands
3164 within the submit description file. The substitution macro is
3165 of the form:
3166 $$(attribute)
3168 As this form of the substitution macro is only evaluated
3170 within the context of the machine ClassAd, use of a scope
3171 resolution prefix TARGET. or MY. is not allowed.
3172 A common use of this form of the substitution macro is for
3174 the heterogeneous submission of an executable:
3175 executable = povray.$$(OpSys).$$(Arch)
3177 Values for the OpSys and Arch attributes are substituted at
3179 match time for any given resource. This example allows HTCon‐
3180 dor to automatically choose the correct executable for the
3181 matched machine.
3182 An extension to the syntax of the substitution macro provides
3184 an alternative string to use if the machine attribute within
3185 the substitution macro is undefined. The syntax appears as:
3186 $$(attribute:string_if_attribute_undefined)
3188 An example using this extended syntax provides a path name to
3190 a required input file. Since the file can be placed in dif‐
3191 ferent locations on different machines, the file's path name
3192 is given as an argument to the program.
3193 arguments = $$(input_file_path:/usr/foo)
3195 On the machine, if the attribute input_file_path is not de‐
3197 fined, then the path /usr/foo is used instead.
3198 As a special case that only works within the submit file en‐
3200 vironment command, the string $$(CondorScratchDir) is ex‐
3201 panded to the value of the job's scratch directory. This
3202 does not work for scheduler universe or grid universe jobs.
3203 For example, to set PYTHONPATH to a subdirectory of the job
3205 scratch dir, one could set
3206 environment = PYTHONPATH=$$(CondorScratchDir)/some/directory
3208 A further extension to the syntax of the substitution macro
3210 allows the evaluation of a ClassAd expression to define the
3211 value. In this form, the expression may refer to machine at‐
3212 tributes by prefacing them with the TARGET. scope resolution
3213 prefix. To place a ClassAd expression into the substitution
3214 macro, square brackets are added to delimit the expression.
3215 The syntax appears as:
3216 $$([ClassAd expression])
3218 An example of a job that uses this syntax may be one that
3220 wants to know how much memory it can use. The application
3221 cannot detect this itself, as it would potentially use all of
3222 the memory on a multi-slot machine. So the job determines the
3223 memory per slot, reducing it by 10% to account for miscella‐
3224 neous overhead, and passes this as a command line argument to
3225 the application. In the submit description file will be
3226 arguments = --memory $$([TARGET.Memory * 0.9])
3228 To insert two dollar sign characters ($$) as literals into a
3232 ClassAd string, use
3233 $$(DOLLARDOLLAR)
3235 The environment macro, $ENV, allows the evaluation of an en‐
3241 vironment variable to be used in setting a submit description
3242 file command. The syntax used is
3243 $ENV(variable)
3245 An example submit description file command that uses this
3247 functionality evaluates the submitter's home directory in or‐
3248 der to set the path and file name of a log file:
3249 log = $ENV(HOME)/jobs/logfile
3251 The environment variable is evaluated when the submit de‐
3253 scription file is processed.
3254 The $RANDOM_CHOICE macro allows a random choice to be made
3259 from a given list of parameters at submission time. For an
3260 expression, if some randomness needs to be generated, the
3261 macro may appear as
3262 $RANDOM_CHOICE(0,1,2,3,4,5,6)
3264 When evaluated, one of the parameters values will be chosen.
3266 Comments
3268 Blank lines and lines beginning with a pound sign ('#') char‐
3269 acter are ignored by the submit description file parser.
3270
3272 While processing the queue command in a submit file or from the command
3273 line, condor_submit will set the values of several automatic submit
3274 variables so that they can be referred to by statements in the submit
3275 file. With the exception of Cluster and Process, if these variables are
3276 set by the submit file, they will not be modified during queue pro‐
3277 cessing.
3278 ClusterId
3280 Set to the integer value that the ClusterId attribute that
3281 the job ClassAd will have when the job is submitted. All jobs
3282 in a single submit will normally have the same value for the
3283 ClusterId. If the -dry-run argument is specified, The value
3284 will be 1.
3285 Cluster
3287 Alternate name for the ClusterId submit variable. Before HT‐
3288 Condor version 8.4 this was the only name.
3289 ProcId Set to the integer value that the ProcId attribute of the job
3291 ClassAd will have when the job is submitted. The value will
3292 start at 0 and increment by 1 for each job submitted.
3293 Process
3295 Alternate name for the ProcId submit variable. Before HTCon‐
3296 dor version 8.4 this was the only name.
3297 JobId Set to $(ClusterId).$(ProcId) so that it will expand to the
3299 full id of the job.
3300 Node For parallel universes, set to the value #pArAlLeLnOdE# or
3302 #MpInOdE# depending on the parallel universe type For other
3303 universes it is set to nothing.
3304 Step Set to the step value as it varies from 0 to N-1 where N is
3306 the number provided on the queue argument. This variable
3307 changes at the same rate as ProcId when it changes at all.
3308 For submit files that don't make use of the queue number op‐
3309 tion, Step will always be 0. For submit files that don't make
3310 use of any of the foreach options, Step and ProcId will al‐
3311 ways be the same.
3312 ItemIndex
3314 Set to the index within the item list being processed by the
3315 various queue foreach options. For submit files that don't
3316 make use of any queue foreach list, ItemIndex will always be
3317 0 For submit files that make use of a slice to select only
3318 some items in a foreach list, ItemIndex will only be set to
3319 selected values.
3320 Row Alternate name for ItemIndex.
3322 Item when a queue foreach option is used and no variable list is
3324 supplied, this variable will be set to the value of the cur‐
3325 rent item.
3326 The automatic variables below are set before parsing the submit file,
3328 and will not vary during processing unless the submit file itself sets
3329 them.
3330 ARCH Set to the CPU architecture of the machine running con‐
3332 dor_submit. The value will be the same as the automatic con‐
3333 figuration variable of the same name.
3334 OPSYS Set to the name of the operating system on the machine run‐
3336 ning condor_submit. The value will be the same as the auto‐
3337 matic configuration variable of the same name.
3338 OPSYSANDVER
3340 Set to the name and major version of the operating system on
3341 the machine running condor_submit. The value will be the same
3342 as the automatic configuration variable of the same name.
3343 OPSYSMAJORVER
3345 Set to the major version of the operating system on the ma‐
3346 chine running condor_submit. The value will be the same as
3347 the automatic configuration variable of the same name.
3348 OPSYSVER
3350 Set to the version of the operating system on the machine
3351 running condor_submit. The value will be the same as the au‐
3352 tomatic configuration variable of the same name.
3353 SPOOL Set to the full path of the HTCondor spool directory. The
3355 value will be the same as the automatic configuration vari‐
3356 able of the same name.
3357 IsLinux
3359 Set to true if the operating system of the machine running
3360 condor_submit is a Linux variant. Set to false otherwise.
3361 IsWindows
3363 Set to true if the operating system of the machine running
3364 condor_submit is a Microsoft Windows variant. Set to false
3365 otherwise.
3366 SUBMIT_FILE
3368 Set to the full pathname of the submit file being processed
3369 by condor_submit. If submit statements are read from standard
3370 input, it is set to nothing.
3371 SUBMIT_TIME
3373 Set to the unix timestamp of the current time when the job is
3374 submitted.
3375 YEAR Set to the 4 digit year when the job is submitted.
3377 MONTH Set to the 2 digit month when the job is submitted.
3379 DAY Set to the 2 digit day when the job is submitted.
3381
3383 condor_submit will exit with a status value of 0 (zero) upon success,
3384 and a non-zero value upon failure.
3385
3387 • Submit Description File Example 1: This example queues three jobs for
3388 execution by HTCondor. The first will be given command line arguments
3389 of 15 and 2000, and it will write its standard output to foo.out1.
3390 The second will be given command line arguments of 30 and 2000, and
3391 it will write its standard output to foo.out2. Similarly the third
3392 will have arguments of 45 and 6000, and it will use foo.out3 for its
3393 standard output. Standard error output (if any) from all three pro‐
3394 grams will appear in foo.error.
3395 ####################
3397 #
3398 # submit description file
3399 # Example 1: queuing multiple jobs with differing
3400 # command line arguments and output files.
3401 #
3402 ####################
3403 Executable = foo
3405 Universe = vanilla
3406 Arguments = 15 2000
3408 Output = foo.out0
3409 Error = foo.err0
3410 Queue
3411 Arguments = 30 2000
3413 Output = foo.out1
3414 Error = foo.err1
3415 Queue
3416 Arguments = 45 6000
3418 Output = foo.out2
3419 Error = foo.err2
3420 Queue
3421 Or you can get the same results as the above submit file by using a
3423 list of arguments with the Queue statement
3424 ####################
3426 #
3427 # submit description file
3428 # Example 1b: queuing multiple jobs with differing
3429 # command line arguments and output files, alternate syntax
3430 #
3431 ####################
3432 Executable = foo
3434 Universe = vanilla
3435 # generate different output and error filenames for each process
3437 Output = foo.out$(Process)
3438 Error = foo.err$(Process)
3439 Queue Arguments From (
3441 15 2000
3442 30 2000
3443 45 6000
3444 )
3445 • Submit Description File Example 2: This submit description file exam‐
3447 ple queues 150 runs of program foo which must have been compiled and
3448 linked for an Intel x86 processor running RHEL 3. HTCondor will not
3449 attempt to run the processes on machines which have less than 32
3450 Megabytes of physical memory, and it will run them on machines which
3451 have at least 64 Megabytes, if such machines are available. Stdin,
3452 stdout, and stderr will refer to in.0, out.0, and err.0 for the first
3453 run of this program (process 0). Stdin, stdout, and stderr will refer
3454 to in.1, out.1, and err.1 for process 1, and so forth. A log file
3455 containing entries about where and when HTCondor runs, transfers
3456 file, if it's evicted, and when it terminates, among other things,
3457 the various processes in this cluster will be written into file
3458 foo.log.
3459 ####################
3461 #
3462 # Example 2: Show off some fancy features including
3463 # use of pre-defined macros and logging.
3464 #
3465 ####################
3466 Executable = foo
3468 Universe = vanilla
3469 Requirements = OpSys == "LINUX" && Arch =="INTEL"
3470 Rank = Memory >= 64
3471 Request_Memory = 32 Mb
3472 Image_Size = 28 Mb
3473 Error = err.$(Process)
3475 Input = in.$(Process)
3476 Output = out.$(Process)
3477 Log = foo.log
3478 Queue 150
3479 • Submit Description File Example 3: This example targets the
3481 /bin/sleep program to run only on a platform running a RHEL 6 operat‐
3482 ing system. The example presumes that the pool contains machines run‐
3483 ning more than one version of Linux, and this job needs the particu‐
3484 lar operating system to run correctly.
3485 ####################
3487 #
3488 # Example 3: Run on a RedHat 6 machine
3489 #
3490 ####################
3491 Universe = vanilla
3492 Executable = /bin/sleep
3493 Arguments = 30
3494 Requirements = (OpSysAndVer == "RedHat6")
3495 Error = err.$(Process)
3497 Input = in.$(Process)
3498 Output = out.$(Process)
3499 Log = sleep.log
3500 Queue
3501 • Command Line example: The following command uses the -append option
3503 to add two commands before the job(s) is queued. A log file and an
3504 error log file are specified. The submit description file is un‐
3505 changed.
3506 $ condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
3508 Note that each of the added commands is contained within quote marks
3510 because there are space characters within the command.
3511 • periodic_remove example: A job should be removed from the queue, if
3513 the total suspension time of the job is more than half of the run
3514 time of the job.
3515 Including the command
3517 periodic_remove = CumulativeSuspensionTime >
3519 ((RemoteWallClockTime - CumulativeSuspensionTime) / 2.0)
3520 in the submit description file causes this to happen.
3522
3524 • For security reasons, HTCondor will refuse to run any jobs submitted
3525 by user root (UID = 0) or by a user whose default group is group
3526 wheel (GID = 0). Jobs submitted by user root or a user with a default
3527 group of wheel will appear to sit forever in the queue in an idle
3528 state.
3529 • All path names specified in the submit description file must be less
3531 than 256 characters in length, and command line arguments must be
3532 less than 4096 characters in length; otherwise, condor_submit gives a
3533 warning message but the jobs will not execute properly.
3534 • Somewhat understandably, behavior gets bizarre if the user makes the
3536 mistake of requesting multiple HTCondor jobs to write to the same
3537 file, and/or if the user alters any files that need to be accessed by
3538 an HTCondor job which is still in the queue. For example, the com‐
3539 pressing of data or output files before an HTCondor job has completed
3540 is a common mistake.
3541
3543 HTCondor User Manual
3544
3546 HTCondor Team
3547
3549 1990-2023, Center for High Throughput Computing, Computer Sciences De‐
3550 partment, University of Wisconsin-Madison, Madison, WI, US. Licensed
3551 under the Apache License, Version 2.0.
3552 Oct 02, 2023 CONDOR_SUBMIT(1)