1CONDOR_SUBMIT(1) HTCondor Manual CONDOR_SUBMIT(1)
2
6 condor_submit - HTCondor Manual
7 Queue jobs for execution under HTCondor
9
12 condor_submit [-terse ] [-verbose ] [-unused ] [-file submit_file]
13 [-name schedd_name] [-remote schedd_name] [-addr <ip:port>] [-pool
14 pool_name] [-disable ] [-password passphrase] [-debug ] [-append com‐
15 mand ...][-batch-name batch_name] [-spool ] [-dump filename] [-interac‐
16 tive ] [-allow-crlf-script ] [-dry-run ] [-maxjobs number-of-jobs]
17 [-single-cluster ] [-stm method] [<submit-variable>=<value> ] [submit
18 description file ] [-queue queue_arguments]
19
21 condor_submit is the program for submitting jobs for execution under
22 HTCondor. condor_submit requires one or more submit description com‐
23 mands to direct the queuing of jobs. These commands may come from a
24 file, standard input, the command line, or from some combination of
25 these. One submit description may contain specifications for the queu‐
26 ing of many HTCondor jobs at once. A single invocation of condor_submit
27 may cause one or more clusters. A cluster is a set of jobs specified in
28 the submit description between queue commands for which the exe‐
29 cutable is not changed. It is advantageous to submit multiple jobs as a
30 single cluster because:
31 • Much less memory is used by the scheduler to hold the same number of
33 jobs.
34 • Only one copy of the checkpoint file is needed to represent all jobs
36 in a cluster until they begin execution.
37 • There is much less overhead involved for HTCondor to start the next
39 job in a cluster than for HTCondor to start a new cluster. This can
40 make a big difference when submitting lots of short jobs.
41 Multiple clusters may be specified within a single submit description.
43 Each cluster must specify a single executable.
44 The job ClassAd attribute ClusterId identifies a cluster.
46 The submit description file argument is the path and file name of the
48 submit description file. If this optional argument is the dash charac‐
49 ter (-), then the commands are taken from standard input. If - is spec‐
50 ified for the submit description file, -verbose is implied; this can be
51 overridden by specifying -terse.
52 If no submit discription file argument is given, and no -queue argument
54 is given, commands are taken automatically from standard input.
55 Note that submission of jobs from a Windows machine requires a stashed
57 password to allow HTCondor to impersonate the user submitting the job.
58 To stash a password, use the condor_store_cred command. See the manual
59 page for details.
60 For lengthy lines within the submit description file, the backslash (\)
62 is a line continuation character. Placing the backslash at the end of a
63 line causes the current line's command to be continued with the next
64 line of the file. Submit description files may contain comments. A com‐
65 ment is any line beginning with a pound character (#).
66
68 -terse Terse output - display JobId ranges only.
69 -verbose
71 Verbose output - display the created job ClassAd
72 -unused
74 As a default, causes no warnings to be issued about user-de‐
75 fined macros not being used within the submit description
76 file. The meaning reverses (toggles) when the configuration
77 variable WARN_ON_UNUSED_SUBMIT_FILE_MACROS
78 is set to the non default value of False. Printing the
79 warnings can help identify spelling errors of submit descrip‐
80 tion file commands. The warnings are sent to stderr.
81 -file submit_file
83 Use submit_file as the submit discription file. This is
84 equivalent to providing submit_file as an argument without
85 the preceeding -file.
86 -name schedd_name
88 Submit to the specified condor_schedd. Use this option to
89 submit to a condor_schedd other than the default local one.
90 schedd_name is the value of the Name ClassAd attribute on the
91 machine where the condor_schedd daemon runs.
92 -remote schedd_name
94 Submit to the specified condor_schedd, spooling all required
95 input files over the network connection. schedd_name is the
96 value of the Name ClassAd attribute on the machine where the
97 condor_schedd daemon runs. This option is equivalent to using
98 both -name and -spool.
99 -addr <ip:port>
101 Submit to the condor_schedd at the IP address and port given
102 by the sinful string argument <ip:port>.
103 -pool pool_name
105 Look in the specified pool for the condor_schedd to submit
106 to. This option is used with -name or -remote.
107 -disable
109 Disable file permission checks when submitting a job for read
110 permissions on all input files, such as those defined by com‐
111 mands input and transfer_input_files , as well as write
112 permission to output files, such as a log file defined by log
113 and output files defined with output or transfer_out‐
114 put_files .
115 -password passphrase
117 Specify a password to the MyProxy server.
118 -debug Cause debugging information to be sent to stderr, based on
120 the value of the configuration variable TOOL_DEBUG.
121 -append command
123 Augment the commands in the submit description file with the
124 given command. This command will be considered to immediately
125 precede the queue command within the submit description file,
126 and come after all other previous commands. If the command
127 specifies a queue command, as in the example
128 condor_submit mysubmitfile -append "queue input in A, B, C"
130 then the entire -append command line option and its arguments
132 are converted to
133 condor_submit mysubmitfile -queue input in A, B, C
135 The submit description file is not modified. Multiple com‐
137 mands are specified by using the -append option multiple
138 times. Each new command is given in a separate -append op‐
139 tion. Commands with spaces in them will need to be enclosed
140 in double quote marks.
141 -batch-name batch_name
143 Set the batch name for this submit. The batch name is dis‐
144 played by condor_q -batch. It is intended for use by users to
145 give meaningful names to their jobs and to influence how con‐
146 dor_q groups jobs for display. Use of this argument takes
147 precedence over a batch name specified in the submit descrip‐
148 tion file itself.
149 -spool Spool all required input files, job event log, and proxy over
151 the connection to the condor_schedd. After submission, modify
152 local copies of the files without affecting your jobs. Any
153 output files for completed jobs need to be retrieved with
154 condor_transfer_data.
155 -dump filename
157 Sends all ClassAds to the specified file, instead of to the
158 condor_schedd.
159 -interactive
161 Indicates that the user wants to run an interactive shell on
162 an execute machine in the pool. This is equivalent to creat‐
163 ing a submit description file of a vanilla universe sleep
164 job, and then running condor_ssh_to_job by hand. Without any
165 additional arguments, condor_submit with the -interactive
166 flag creates a dummy vanilla universe job that sleeps, sub‐
167 mits it to the local scheduler, waits for the job to run, and
168 then launches condor_ssh_to_job to run a shell. If the user
169 would like to run the shell on a machine that matches a par‐
170 ticular requirements expression, the submit description file
171 is specified, and it will contain the expression. Note that
172 all policy expressions specified in the submit description
173 file are honored, but any executable or universe commands
174 are overwritten to be sleep and vanilla. The job ClassAd at‐
175 tribute InteractiveJob is set to True to identify interactive
176 jobs for condor_startd policy usage.
177 -allow-crlf-script
179 Changes the check for an invalid line ending on the exe‐
180 cutable script's #! line from an ERROR to a WARNING. The #!
181 line will be ignored by Windows, so it won't matter if it is
182 invalid; but Unix and Linux will not run a script that has a
183 Windows/DOS line ending on the first line of the script. So
184 condor_submit will not allow such a script to be submitted as
185 the job's executable unless this option is supplied.
186 -dry-run file
188 Parse the submit description file, sending the resulting job
189 ClassAd to the file given by file, but do not submit the
190 job(s). This permits observation of the job specification,
191 and it facilitates debugging the submit description file con‐
192 tents. If file is -, the output is written to stdout.
193 -maxjobs number-of-jobs
195 If the total number of jobs specified by the submit descrip‐
196 tion file is more than the integer value given by num‐
197 ber-of-jobs, then no jobs are submitted for execution and an
198 error message is generated. A 0 or negative value for the
199 number-of-jobs causes no limit to be imposed.
200 -single-cluster
202 If the jobs specified by the submit description file causes
203 more than a single cluster value to be assigned, then no jobs
204 are submitted for execution and an error message is gener‐
205 ated.
206 -stm method
208 Specify the method use to move a sandbox into HTCondor.
209 method is one of stm_use_schedd_only or stm_use_transferd.
210 <submit-variable>=<value>
212 Defines a submit command or submit variable with a value, and
213 parses it as if it was placed at the beginning of the submit
214 description file. The submit description file is not changed.
215 To correctly parse the condor_submit command line, this op‐
216 tion must be specified without white space characters before
217 and after the equals sign (=), or the entire option must be
218 surrounded by double quote marks.
219 -queue queue_arguments
221 A command line specification of how many jobs to queue, which
222 is only permitted if the submit description file does not
223 have a queue command. The queue_arguments are the same as may
224 be within a submit description file. The parsing of the
225 queue_arguments finishes at the end of the line or when a
226 dash character (-) is encountered. Therefore, its best place‐
227 ment within the command line will be at the end of the com‐
228 mand line.
229 On a Unix command line, the shell expands file globs before
231 parsing occurs.
232
234 Note: more information on submitting HTCondor jobs can be found here:
235 Submitting a Job.
236 As of version 8.5.6, the condor_submit language supports multi-line
238 values in commands. The syntax is the same as the configuration lan‐
239 guage (see more details here: Multi-Line Values).
240 Each submit description file describes one or more clusters of jobs to
242 be placed in the HTCondor execution pool. All jobs in a cluster must
243 share the same executable, but they may have different input and output
244 files, and different program arguments. The submit description file is
245 generally the last command-line argument to condor_submit. If the sub‐
246 mit description file argument is omitted, condor_submit will read the
247 submit description from standard input.
248 The submit description file must contain at least one executable com‐
250 mand and at least one queue command. All of the other commands have de‐
251 fault actions.
252 Note that a submit file that contains more than one executable command
254 will produce multiple clusters when submitted. This is not generally
255 recommended, and is not allowed for submit files that are run as DAG
256 node jobs by condor_dagman.
257 The commands which can appear in the submit description file are numer‐
259 ous. They are listed here in alphabetical order by category.
260 BASIC COMMANDS
262 arguments = <argument_list>
264 List of arguments to be supplied to the executable as part of
265 the command line.
266 In the java universe, the first argument must be the name of
268 the class containing main.
269 There are two permissible formats for specifying arguments,
271 identified as the old syntax and the new syntax. The old syn‐
272 tax supports white space characters within arguments only in
273 special circumstances; when used, the command line arguments
274 are represented in the job ClassAd attribute Args. The new
275 syntax supports uniform quoting of white space characters
276 within arguments; when used, the command line arguments are
277 represented in the job ClassAd attribute Arguments.
278 Old Syntax
280 In the old syntax, individual command line arguments are de‐
282 limited (separated) by space characters. To allow a double
283 quote mark in an argument, it is escaped with a backslash;
284 that is, the two character sequence \" becomes a single dou‐
285 ble quote mark within an argument.
286 Further interpretation of the argument string differs depend‐
288 ing on the operating system. On Windows, the entire argument
289 string is passed verbatim (other than the backslash in front
290 of double quote marks) to the Windows application. Most Win‐
291 dows applications will allow spaces within an argument value
292 by surrounding the argument with double quotes marks. In all
293 other cases, there is no further interpretation of the argu‐
294 ments.
295 Example:
297 arguments = one \"two\" 'three'
299 Produces in Unix vanilla universe:
301 argument 1: one
303 argument 2: "two"
304 argument 3: 'three'
305 New Syntax
307 Here are the rules for using the new syntax:
309 1. The entire string representing the command line arguments
311 is surrounded by double quote marks. This permits the
312 white space characters of spaces and tabs to potentially
313 be embedded within a single argument. Putting the double
314 quote mark within the arguments is accomplished by escap‐
315 ing it with another double quote mark.
316 2. The white space characters of spaces or tabs delimit argu‐
318 ments.
319 3. To embed white space characters of spaces or tabs within a
321 single argument, surround the entire argument with single
322 quote marks.
323 4. To insert a literal single quote mark, escape it within an
325 argument already delimited by single quote marks by adding
326 another single quote mark.
327 Example:
329 arguments = "3 simple arguments"
331 Produces:
333 argument 1: 3
335 argument 2: simple
336 argument 3: arguments
337 Another example:
339 arguments = "one 'two with spaces' 3"
341 Produces:
343 argument 1: one
345 argument 2: two with spaces
346 argument 3: 3
347 And yet another example:
349 arguments = "one ""two"" 'spacey ''quoted'' argument'"
351 Produces:
353 argument 1: one
355 argument 2: "two"
356 argument 3: spacey 'quoted' argument
357 Notice that in the new syntax, the backslash has no special
359 meaning. This is for the convenience of Windows users.
360 environment = <parameter_list>
363 List of environment
364 variables.
365 There are two different formats for specifying the environ‐
367 ment variables: the old format and the new format. The old
368 format is retained for backward-compatibility. It suffers
369 from a platform-dependent syntax and the inability to insert
370 some special characters into the environment.
371 The new syntax for specifying environment values:
373 1. Put double quote marks around the entire argument string.
375 This distinguishes the new syntax from the old. The old
376 syntax does not have double quote marks around it. Any
377 literal double quote marks within the string must be es‐
378 caped by repeating the double quote mark.
379 2. Each environment entry has the form
381 <name>=<value>
383 3. Use white space (space or tab characters) to separate en‐
385 vironment entries.
386 4. To put any white space in an environment entry, surround
388 the space and as much of the surrounding entry as desired
389 with single quote marks.
390 5. To insert a literal single quote mark, repeat the single
392 quote mark anywhere inside of a section surrounded by sin‐
393 gle quote marks.
394 Example:
396 environment = "one=1 two=""2"" three='spacey ''quoted'' value'"
398 Produces the following environment entries:
400 one=1
402 two="2"
403 three=spacey 'quoted' value
404 Under the old syntax, there are no double quote marks sur‐
406 rounding the environment specification. Each environment en‐
407 try remains of the form
408 <name>=<value>
410 Under Unix, list multiple environment entries by separating
412 them with a semicolon (;). Under Windows, separate multiple
413 entries with a vertical bar (|). There is no way to insert a
414 literal semicolon under Unix or a literal vertical bar under
415 Windows. Note that spaces are accepted, but rarely desired,
416 characters within parameter names and values, because they
417 are treated as literal characters, not separators or ignored
418 white space. Place spaces within the parameter list only if
419 required.
420 A Unix example:
422 environment = one=1;two=2;three="quotes have no 'special' meaning"
424 This produces the following:
426 one=1
428 two=2
429 three="quotes have no 'special' meaning"
430 If the environment is set with the environment command and
432 getenv is also set to true, values specified with environ‐
433 ment override values in the submitter's environment (regard‐
434 less of the order of the environment and getenv commands).
435 error = <pathname>
438 A path and file name used by HTCondor to capture any error
439 messages the program would normally write to the screen (that
440 is, this file becomes stderr). A path is given with respect
441 to the file system of the machine on which the job is submit‐
442 ted. The file is written (by the job) in the remote scratch
443 directory of the machine where the job is executed. When the
444 job exits, the resulting file is transferred back to the ma‐
445 chine where the job was submitted, and the path is utilized
446 for file placement. If not specified, the default value of
447 /dev/null is used for submission to a Unix machine. If not
448 specified, error messages are ignored for submission to a
449 Windows machine. More than one job should not use the same
450 error file, since this will cause one job to overwrite the
451 errors of another. If HTCondor detects that the error and
452 output files for a job are the same, it will run the job such
453 that the output and error data is merged.
454 executable = <pathname>
456 An optional path and a required file name of the executable
457 file for this job cluster. Only one executable command
458 within a submit description file is guaranteed to work prop‐
459 erly. More than one often works.
460 If no path or a relative path is used, then the executable
462 file is presumed to be relative to the current working direc‐
463 tory of the user as the condor_submit command is issued.
464 If submitting into the standard universe, then the named exe‐
466 cutable must have been re-linked with the HTCondor libraries
467 (such as via the condor_compile command). If submitting into
468 the vanilla universe (the default), then the named executable
469 need not be re-linked and can be any process which can run in
470 the background (shell scripts work fine as well). If submit‐
471 ting into the Java universe, then the argument must be a com‐
472 piled .class file.
473 getenv = <True | False>
476 If getenv is set to
477 True, then condor_submit will copy all of the user's current
478 shell environment variables at the time of job submission
479 into the job ClassAd. The job will therefore execute with the
480 same set of environment variables that the user had at submit
481 time. Defaults to False.
482 If the environment is set with the environment command and
484 getenv is also set to true, values specified with environment
485 override values in the submitter's environment (regardless of
486 the order of the environment and getenv commands).
487 input = <pathname>
489 HTCondor assumes that its jobs are long-running, and that the
490 user will not wait at the terminal for their completion. Be‐
491 cause of this, the standard files which normally access the
492 terminal, (stdin, stdout, and stderr), must refer to files.
493 Thus, the file name specified with input should contain any
494 keyboard input the program requires (that is, this file be‐
495 comes stdin). A path is given with respect to the file system
496 of the machine on which the job is submitted. The file is
497 transferred before execution to the remote scratch directory
498 of the machine where the job is executed. If not specified,
499 the default value of /dev/null is used for submission to a
500 Unix machine. If not specified, input is ignored for submis‐
501 sion to a Windows machine. For grid universe jobs, input
502 may be a URL that the Globus tool globus_url_copy under‐
503 stands.
504 Note that this command does not refer to the command-line ar‐
506 guments of the program. The command-line arguments are speci‐
507 fied by the arguments command.
508 log = <pathname>
511 Use log to specify a file name where HTCondor will write a
512 log file of what is happening with this job cluster, called a
513 job event log. For example, HTCondor will place a log entry
514 into this file when and where the job begins running, when
515 the job produces a checkpoint, or moves (migrates) to another
516 machine, and when the job completes. Most users find specify‐
517 ing a log file to be handy; its use is recommended. If no log
518 entry is specified, HTCondor does not create a log for this
519 cluster. If a relative path is specified, it is relative to
520 the current working directory as the job is submitted or the
521 directory specified by submit command initialdir on the sub‐
522 mit machine.
523 log_xml = <True | False>
526 If log_xml is True, then the job event log file will be
527 written in ClassAd XML. If not specified, XML is not used.
528 Note that the file is an XML fragment; it is missing the file
529 header and footer. Do not mix XML and non-XML within a single
530 file. If multiple jobs write to a single job event log file,
531 ensure that all of the jobs specify this option in the same
532 way.
533 notification = <Always | Complete | Error | Never>
538 Owners of HTCondor jobs are notified by e-mail when certain
539 events occur. If defined by Always, the owner will be noti‐
540 fied whenever the job produces a checkpoint, as well as when
541 the job completes. If defined by Complete, the owner will be
542 notified when the job terminates. If defined by Error, the
543 owner will only be notified if the job terminates abnormally,
544 (as defined by JobSuccessExitCode, if defined) or if the job
545 is placed on hold because of a failure, and not by user re‐
546 quest. If defined by Never (the default), the owner will not
547 receive e-mail, regardless to what happens to the job. The
548 HTCondor User's manual documents statistics included in the
549 e-mail.
550 notify_user = <email-address>
552 Used to specify the e-mail address to use when HTCondor sends
553 e-mail about a job. If not specified, HTCondor defaults to
554 using the e-mail address defined by
555 job-owner@UID_DOMAIN
557 where the configuration variable UID_DOMAIN
559 is specified by the HTCondor site administrator. If UID_DO‐
560 MAIN has not been specified, HTCondor sends the e-mail to:
561 job-owner@submit-machine-name
563 output = <pathname>
567 The output file captures any information the program would
568 ordinarily write to the screen (that is, this file becomes
569 stdout). A path is given with respect to the file system of
570 the machine on which the job is submitted. The file is writ‐
571 ten (by the job) in the remote scratch directory of the ma‐
572 chine where the job is executed. When the job exits, the re‐
573 sulting file is transferred back to the machine where the job
574 was submitted, and the path is utilized for file placement.
575 If not specified, the default value of /dev/null is used for
576 submission to a Unix machine. If not specified, output is ig‐
577 nored for submission to a Windows machine. Multiple jobs
578 should not use the same output file, since this will cause
579 one job to overwrite the output of another. If HTCondor de‐
580 tects that the error and output files for a job are the same,
581 it will run the job such that the output and error data is
582 merged.
583 Note that if a program explicitly opens and writes to a file,
585 that file should not be specified as the output file.
586 priority = <integer>
589 An HTCondor job priority can be any integer, with 0 being the
590 default. Jobs with higher numerical priority will run before
591 jobs with lower numerical priority. Note that this priority
592 is on a per user basis. One user with many jobs may use this
593 command to order his/her own jobs, and this will have no ef‐
594 fect on whether or not these jobs will run ahead of another
595 user's jobs.
596 Note that the priority setting in an HTCondor submit file
598 will be overridden by condor_dagman if the submit file is
599 used for a node in a DAG, and the priority of the node within
600 the DAG is non-zero (see Advanced Features of DAGMan for
601 more details).
602 queue [<int expr> ]
604 Places zero or more copies of the job into the HTCondor
605 queue.
606 queue [<int expr> ] [<varname> ] in [slice ] <list of items> Places
608 zero or more copies of the job in the queue based on items in
609 a <list of items>
610 queue [<int expr> ] [<varname> ] matching [files | dirs ] [slice ]
612 <list of items with file globbing>] Places zero or more
613 copies of the job in the queue based on files that match a
614 <list of items with file globbing>
615 queue [<int expr> ] [<list of varnames> ] from [slice ] <file name>
617 | <list of items>] Places zero or more copies of the job in
618 the queue based on lines from the submit file or from <file
619 name>
620 The optional argument <int expr> specifies how many times to
622 repeat the job submission for a given set of arguments. It
623 may be an integer or an expression that evaluates to an inte‐
624 ger, and it defaults to 1. All but the first form of this
625 command are various ways of specifying a list of items. When
626 these forms are used <int expr> jobs will be queued for each
627 item in the list. The in, matching and from keyword indicates
628 how the list will be specified.
629 • in The list of items is an explicit comma and/or space sep‐
631 arated <list of items>. If the <list of items> begins with
632 an open paren, and the close paren is not on the same line
633 as the open, then the list continues until a line that be‐
634 gins with a close paren is read from the submit file.
635 • matching Each item in the <list of items with file glob‐
637 bing> will be matched against the names of files and direc‐
638 tories relative to the current directory, the set of match‐
639 ing names is the resulting list of items.
640 • files Only filenames will matched.
642 • dirs Only directory names will be matched.
644 • from <file name> | <list of items> Each line from <file
646 name> or <list of items> is a single item, this allows for
647 multiple variables to be set for each item. Lines from
648 <file name> or <list of items> will be split on comma
649 and/or space until there are values for each of the vari‐
650 ables specified in <list of varnames>. The last variable
651 will contain the remainder of the line. When the <list of
652 items> form is used, the list continues until the first
653 line that begins with a close paren, and lines beginning
654 with pound sign ('#') will be skipped. When using the
655 <file name> form, if the <file name> ends with |, then it
656 will be executed as a script whatever the script writes to
657 stdout will be the list of items.
658 The optional argument <varname> or <list of varnames> is the
660 name or names of of variables that will be set to the value
661 of the current item when queuing the job. If no <varname> is
662 specified the variable ITEM will be used. Leading and trail‐
663 ing whitespace be trimmed. The optional argument <slice> is a
664 python style slice selecting only some of the items in the
665 list of items. Negative step values are not supported.
666 A submit file may contain more than one queue statement,
668 and if desired, any commands may be placed between subsequent
669 queue commands, such as new input , output , error ,
670 initialdir , or arguments commands. This is handy when
671 submitting multiple runs into one cluster with one submit de‐
672 scription file.
673 universe = <vanilla | standard | scheduler | local | grid | java| vm
676 | parallel | docker>
677 Specifies which HTCondor universe to use when running this
678 job. The HTCondor universe specifies an HTCondor execution
679 environment.
680 The vanilla universe is the default (except where the config‐
682 uration variable DEFAULT_UNIVERSE
683 defines it otherwise), and is an execution environment for
684 jobs which do not use HTCondor's mechanisms for taking check‐
685 points; these are ones that have not been linked with the HT‐
686 Condor libraries. Use the vanilla universe to submit shell
687 scripts to HTCondor.
688 The standard universe tells HTCondor that this job has been
690 re-linked via condor_compile with the HTCondor libraries and
691 therefore supports taking checkpoints and remote system
692 calls.
693 The scheduler universe is for a job that is to run on the ma‐
695 chine where the job is submitted. This universe is intended
696 for a job that acts as a metascheduler and will not be pre‐
697 empted.
698 The local universe is for a job that is to run on the machine
700 where the job is submitted. This universe runs the job imme‐
701 diately and will not preempt the job.
702 The grid universe forwards the job to an external job manage‐
704 ment system. Further specification of the grid universe is
705 done with the grid_resource command.
706 The java universe is for programs written to the Java Virtual
708 Machine.
709 The vm universe facilitates the execution of a virtual ma‐
711 chine.
712 The parallel universe is for parallel jobs (e.g. MPI) that
714 require multiple machines in order to run.
715 The docker universe runs a docker container as an HTCondor
717 job.
718 COMMANDS FOR MATCHMAKING
720 rank = <ClassAd Float Expression>
722 A ClassAd Floating-Point expression that states how to rank
723 machines which have already met the requirements expression.
724 Essentially, rank expresses preference. A higher numeric
725 value equals better rank. HTCondor will give the job the ma‐
726 chine with the highest rank. For example,
727 request_memory = max({60, Target.TotalSlotMemory})
729 rank = Memory
730 asks HTCondor to find all available machines with more than
732 60 megabytes of memory and give to the job the machine with
733 the most amount of memory. The HTCondor User's Manual con‐
734 tains complete information on the syntax and available at‐
735 tributes that can be used in the ClassAd expression.
736 request_cpus = <num-cpus>
739 A requested number of CPUs (cores). If not specified, the
740 number requested will be 1. If specified, the expression
741 && (RequestCpus <= Target.Cpus)
743 is appended to the requirements expression for the job.
745 For pools that enable dynamic condor_startd provisioning,
747 specifies the minimum number of CPUs requested for this job,
748 resulting in a dynamic slot being created with this many
749 cores.
750 request_disk = <quantity>
753 The requested amount of disk space in KiB requested for this
754 job. If not specified, it will be set to the job ClassAd at‐
755 tribute DiskUsage. The expression
756 && (RequestDisk <= Target.Disk)
758 is appended to the requirements expression for the job.
760 For pools that enable dynamic condor_startd provisioning, a
762 dynamic slot will be created with at least this much disk
763 space.
764 Characters may be appended to a numerical value to indicate
766 units. K or KB indicates KiB, 210 numbers of bytes. M or MB
767 indicates MiB, 220 numbers of bytes. G or GB indicates GiB,
768 230 numbers of bytes. T or TB indicates TiB, 240 numbers of
769 bytes.
770 request_memory = <quantity>
773 The required amount of memory in MiB that this job needs to
774 avoid excessive swapping. If not specified and the submit
775 command vm_memory is specified, then the value specified
776 for vm_memory defines request_memory . If neither re‐
777 quest_memory nor vm_memory is specified, the value is set
778 by the configuration variable JOB_DEFAULT_REQUESTMEMORY
779 . The actual amount of memory used by a job is represented
780 by the job ClassAd attribute MemoryUsage.
781 For pools that enable dynamic condor_startd provisioning, a
783 dynamic slot will be created with at least this much RAM.
784 The expression
786 && (RequestMemory <= Target.Memory)
788 is appended to the requirements expression for the job.
790 Characters may be appended to a numerical value to indicate
792 units. K or KB indicates KiB, 210 numbers of bytes. M or MB
793 indicates MiB, 220 numbers of bytes. G or GB indicates GiB,
794 230 numbers of bytes. T or TB indicates TiB, 240 numbers of
795 bytes.
796 request_<name> = <quantity>
801 The required amount of the custom machine resource identified
802 by <name> that this job needs. The custom machine resource is
803 defined in the machine's configuration. Machines that have
804 available GPUs will define <name> to be GPUs.
805 requirements = <ClassAd Boolean Expression>
808 The requirements command is a boolean ClassAd expression
809 which uses C-like operators. In order for any job in this
810 cluster to run on a given machine, this requirements expres‐
811 sion must evaluate to true on the given machine.
812 For scheduler and local universe jobs, the requirements ex‐
814 pression is evaluated against the Scheduler ClassAd which
815 represents the the condor_schedd daemon running on the submit
816 machine, rather than a remote machine. Like all commands in
817 the submit description file, if multiple requirements com‐
818 mands are present, all but the last one are ignored. By de‐
819 fault, condor_submit appends the following clauses to the re‐
820 quirements expression:
821 1. Arch and OpSys are set equal to the Arch and OpSys of the
823 submit machine. In other words: unless you request other‐
824 wise, HTCondor will give your job machines with the same
825 architecture and operating system version as the machine
826 running condor_submit.
827 2. Cpus >= RequestCpus, if the job ClassAd attribute Re‐
829 questCpus is defined.
830 3. Disk >= RequestDisk, if the job ClassAd attribute Request‐
832 Disk is defined. Otherwise, Disk >= DiskUsage is appended
833 to the requirements. The DiskUsage attribute is initial‐
834 ized to the size of the executable plus the size of any
835 files specified in a transfer_input_files command. It ex‐
836 ists to ensure there is enough disk space on the target
837 machine for HTCondor to copy over both the executable and
838 needed input files. The DiskUsage attribute represents the
839 maximum amount of total disk space required by the job in
840 kilobytes. HTCondor automatically updates the DiskUsage
841 attribute approximately every 20 minutes while the job
842 runs with the amount of space being used by the job on the
843 execute machine.
844 4. Memory >= RequestMemory, if the job ClassAd attribute Re‐
846 questMemory is defined.
847 5. If Universe is set to Vanilla, FileSystemDomain is set
849 equal to the submit machine's FileSystemDomain.
850 View the requirements of a job which has already been submit‐
852 ted (along with everything else about the job ClassAd) with
853 the command condor_q -l; see the command reference for
854 condor_q. Also, see the HTCondor Users Manual for complete
855 information on the syntax and available attributes that can
856 be used in the ClassAd expression.
857 FILE TRANSFER COMMANDS
859 dont_encrypt_input_files = < file1,file2,file... >
863 A comma and/or space separated list of input files that are
864 not to be network encrypted when transferred with the file
865 transfer mechanism. Specification of files in this manner
866 overrides configuration that would use encryption. Each input
867 file must also be in the list given by transfer_input_files
868 . When a path to an input file or directory is specified,
869 this specifies the path to the file on the submit side. A
870 single wild card character (*) may be used in each file name.
871 dont_encrypt_output_files = < file1,file2,file... >
875 A comma and/or space separated list of output files that are
876 not to be network encrypted when transferred back with the
877 file transfer mechanism. Specification of files in this man‐
878 ner overrides configuration that would use encryption. The
879 output file(s) must also either be in the list given by
880 transfer_output_files or be discovered and to be transferred
881 back with the file transfer mechanism. When a path to an out‐
882 put file or directory is specified, this specifies the path
883 to the file on the execute side. A single wild card character
884 (*) may be used in each file name.
885 encrypt_execute_directory = <True | False>
888 Defaults to False. If set to True, HTCondor will encrypt the
889 contents of the remote scratch directory of the machine where
890 the job is executed. This encryption is transparent to the
891 job itself, but ensures that files left behind on the local
892 disk of the execute machine, perhaps due to a system crash,
893 will remain private. In addition, condor_submit will append
894 to the job's requirements expression
895 && (TARGET.HasEncryptExecuteDirectory)
897 to ensure the job is matched to a machine that is capable of
899 encrypting the contents of the execute directory. This sup‐
900 port is limited to Windows platforms that use the NTFS file
901 system and Linux platforms with the ecryptfs-utils package
902 installed.
903 encrypt_input_files = < file1,file2,file... >
907 A comma and/or space separated list of input files that are
908 to be network encrypted when transferred with the file trans‐
909 fer mechanism. Specification of files in this manner over‐
910 rides configuration that would not use encryption. Each input
911 file must also be in the list given by transfer_input_files
912 . When a path to an input file or directory is specified,
913 this specifies the path to the file on the submit side. A
914 single wild card character (*) may be used in each file name.
915 The method of encryption utilized will be as agreed upon in
916 security negotiation; if that negotiation failed, then the
917 file transfer mechanism must also fail for files to be net‐
918 work encrypted.
919 encrypt_output_files = < file1,file2,file... >
923 A comma and/or space separated list of output files that are
924 to be network encrypted when transferred back with the file
925 transfer mechanism. Specification of files in this manner
926 overrides configuration that would not use encryption. The
927 output file(s) must also either be in the list given by
928 transfer_output_files or be discovered and to be transferred
929 back with the file transfer mechanism. When a path to an out‐
930 put file or directory is specified, this specifies the path
931 to the file on the execute side. A single wild card character
932 (*) may be used in each file name. The method of encryption
933 utilized will be as agreed upon in security negotiation; if
934 that negotiation failed, then the file transfer mechanism
935 must also fail for files to be network encrypted.
936 max_transfer_input_mb = <ClassAd Integer Expression>
939 This integer expression specifies the maximum allowed total
940 size in MiB of the input files that are transferred for a
941 job. This expression does not apply to grid universe, stan‐
942 dard universe, or files transferred via file transfer
943 plug-ins. The expression may refer to attributes of the job.
944 The special value -1 indicates no limit. If not defined, the
945 value set by configuration variable MAX_TRANSFER_INPUT_MB
946 is used. If the observed size of all input files at submit
947 time is larger than the limit, the job will be immediately
948 placed on hold with a HoldReasonCode value of 32. If the job
949 passes this initial test, but the size of the input files in‐
950 creases or the limit decreases so that the limit is violated,
951 the job will be placed on hold at the time when the file
952 transfer is attempted.
953 max_transfer_output_mb = <ClassAd Integer Expression>
956 This integer expression specifies the maximum allowed total
957 size in MiB of the output files that are transferred for a
958 job. This expression does not apply to grid universe, stan‐
959 dard universe, or files transferred via file transfer
960 plug-ins. The expression may refer to attributes of the job.
961 The special value -1 indicates no limit. If not set, the
962 value set by configuration variable MAX_TRANSFER_OUTPUT_MB
963 is used. If the total size of the job's output files to be
964 transferred is larger than the limit, the job will be placed
965 on hold with a HoldReasonCode value of 33. The output will be
966 transferred up to the point when the limit is hit, so some
967 files may be fully transferred, some partially, and some not
968 at all.
969 output_destination = <destination-URL>
973 When present, defines a URL that specifies both a plug-in and
974 a destination for the transfer of the entire output sandbox
975 or a subset of output files as specified by the submit com‐
976 mand transfer_output_files . The plug-in does the transfer
977 of files, and no files are sent back to the submit machine.
978 The HTCondor Administrator's manual has full details.
979 should_transfer_files = <YES | NO | IF_NEEDED >
982 The should_transfer_files setting is used to define if HTCon‐
983 dor should transfer files to and from the remote machine
984 where the job runs. The file transfer mechanism is used to
985 run jobs which are not in the standard universe (and can
986 therefore use remote system calls for file access) on ma‐
987 chines which do not have a shared file system with the submit
988 machine. should_transfer_files equal to YES will cause HT‐
989 Condor to always transfer files for the job. NO disables HT‐
990 Condor's file transfer mechanism. IF_NEEDED will not transfer
991 files for the job if it is matched with a resource in the
992 same FileSystemDomain as the submit machine (and therefore,
993 on a machine with the same shared file system). If the job is
994 matched with a remote resource in a different FileSystemDo‐
995 main, HTCondor will transfer the necessary files.
996 For more information about this and other settings related to
998 transferring files, see the HTCondor User's manual section on
999 the file transfer mechanism.
1000 Note that should_transfer_files is not supported for jobs
1002 submitted to the grid universe.
1003 skip_filechecks = <True | False>
1006 When True, file permission checks for the submitted job are
1007 disabled. When False, file permissions are checked; this is
1008 the behavior when this command is not present in the submit
1009 description file. File permissions are checked for read per‐
1010 missions on all input files, such as those defined by com‐
1011 mands input and transfer_input_files , and for write per‐
1012 mission to output files, such as a log file defined by log
1013 and output files defined with output or transfer_out‐
1014 put_files .
1015 stream_error = <True | False>
1018 If True, then stderr is streamed back to the machine from
1019 which the job was submitted. If False, stderr is stored lo‐
1020 cally and transferred back when the job completes. This com‐
1021 mand is ignored if the job ClassAd attribute TransferErr is
1022 False. The default value is False. This command must be used
1023 in conjunction with error , otherwise stderr will sent to
1024 /dev/null on Unix machines and ignored on Windows machines.
1025 stream_input = <True | False>
1028 If True, then stdin is streamed from the machine on which the
1029 job was submitted. The default value is False. The command is
1030 only relevant for jobs submitted to the vanilla or java uni‐
1031 verses, and it is ignored by the grid universe. This command
1032 must be used in conjunction with input , otherwise stdin
1033 will be /dev/null on Unix machines and ignored on Windows ma‐
1034 chines.
1035 stream_output = <True | False>
1037 If True, then stdout is streamed back to the machine from
1038 which the job was submitted. If False, stdout is stored lo‐
1039 cally and transferred back when the job completes. This com‐
1040 mand is ignored if the job ClassAd attribute TransferOut is
1041 False. The default value is False. This command must be used
1042 in conjunction with output , otherwise stdout will sent to
1043 /dev/null on Unix machines and ignored on Windows machines.
1044 transfer_executable = <True | False>
1047 This command is applicable to jobs submitted to the grid and
1048 vanilla universes. If transfer_executable is set to False,
1049 then HTCondor looks for the executable on the remote machine,
1050 and does not transfer the executable over. This is useful for
1051 an already pre-staged executable; HTCondor behaves more like
1052 rsh. The default value is True.
1053 transfer_input_files = < file1,file2,file... >
1056 A comma-delimited list of all the files and directories to be
1057 transferred into the working directory for the job, before
1058 the job is started. By default, the file specified in the ex‐
1059 ecutable command and any file specified in the input com‐
1060 mand (for example, stdin) are transferred.
1061 When a path to an input file or directory is specified, this
1063 specifies the path to the file on the submit side. The file
1064 is placed in the job's temporary scratch directory on the ex‐
1065 ecute side, and it is named using the base name of the origi‐
1066 nal path. For example, /path/to/input_file becomes input_file
1067 in the job's scratch directory.
1068 A directory may be specified by appending the forward slash
1070 character (/) as a trailing path separator. This syntax is
1071 used for both Windows and Linux submit hosts. A directory ex‐
1072 ample using a trailing path separator is input_data/. When a
1073 directory is specified with the trailing path separator, the
1074 contents of the directory are transferred, but the directory
1075 itself is not transferred. It is as if each of the items
1076 within the directory were listed in the transfer list. When
1077 there is no trailing path separator, the directory is trans‐
1078 ferred, its contents are transferred, and these contents are
1079 placed inside the transferred directory.
1080 For grid universe jobs other than HTCondor-C, the transfer of
1082 directories is not currently supported.
1083 Symbolic links to files are transferred as the files they
1085 point to. Transfer of symbolic links to directories is not
1086 currently supported.
1087 For vanilla and vm universe jobs only, a file may be speci‐
1089 fied by giving a URL, instead of a file name. The implementa‐
1090 tion for URL transfers requires both configuration and avail‐
1091 able plug-in.
1092 transfer_output_files = < file1,file2,file... >
1095 This command forms an explicit list of output files and di‐
1096 rectories to be transferred back from the temporary working
1097 directory on the execute machine to the submit machine. If
1098 there are multiple files, they must be delimited with commas.
1099 Setting transfer_output_files to the empty string ("") means
1100 that no files are to be transferred.
1101 For HTCondor-C jobs and all other non-grid universe jobs, if
1103 transfer_output_files is not specified, HTCondor will auto‐
1104 matically transfer back all files in the job's temporary
1105 working directory which have been modified or created by the
1106 job. Subdirectories are not scanned for output, so if output
1107 from subdirectories is desired, the output list must be ex‐
1108 plicitly specified. For grid universe jobs other than HTCon‐
1109 dor-C, desired output files must also be explicitly listed.
1110 Another reason to explicitly list output files is for a job
1111 that creates many files, and the user wants only a subset
1112 transferred back.
1113 For grid universe jobs other than with grid type condor, to
1115 have files other than standard output and standard error
1116 transferred from the execute machine back to the submit ma‐
1117 chine, do use transfer_output_files, listing all files to be
1118 transferred. These files are found on the execute machine in
1119 the working directory of the job.
1120 When a path to an output file or directory is specified, it
1122 specifies the path to the file on the execute side. As a des‐
1123 tination on the submit side, the file is placed in the job's
1124 initial working directory, and it is named using the base
1125 name of the original path. For example, path/to/output_file
1126 becomes output_file in the job's initial working directory.
1127 The name and path of the file that is written on the submit
1128 side may be modified by using transfer_output_remaps . Note
1129 that this remap function only works with files but not with
1130 directories.
1131 A directory may be specified using a trailing path separator.
1133 An example of a trailing path separator is the slash charac‐
1134 ter on Unix platforms; a directory example using a trailing
1135 path separator is input_data/. When a directory is specified
1136 with a trailing path separator, the contents of the directory
1137 are transferred, but the directory itself is not transferred.
1138 It is as if each of the items within the directory were
1139 listed in the transfer list. When there is no trailing path
1140 separator, the directory is transferred, its contents are
1141 transferred, and these contents are placed inside the trans‐
1142 ferred directory.
1143 For grid universe jobs other than HTCondor-C, the transfer of
1145 directories is not currently supported.
1146 Symbolic links to files are transferred as the files they
1148 point to. Transfer of symbolic links to directories is not
1149 currently supported.
1150 transfer_output_remaps = < name = newname ; name2 = newname2 ... >
1152 This specifies the name (and optionally path) to use when
1153 downloading output files from the completed job. Normally,
1154 output files are transferred back to the initial working di‐
1155 rectory with the same name they had in the execution direc‐
1156 tory. This gives you the option to save them with a different
1157 path or name. If you specify a relative path, the final path
1158 will be relative to the job's initial working directory.
1159 name describes an output file name produced by your job, and
1161 newname describes the file name it should be downloaded to.
1162 Multiple remaps can be specified by separating each with a
1163 semicolon. If you wish to remap file names that contain
1164 equals signs or semicolons, these special characters may be
1165 escaped with a backslash. You cannot specify directories to
1166 be remapped.
1167 Note that whether an output file is transferred is controlled
1169 by transfer_output_files. Listing a file in transfer_out‐
1170 put_remaps is not sufficient to cause it to be transferred.
1171 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT >
1174 Setting when_to_transfer_output equal to ON_EXIT will cause
1175 HTCondor to transfer the job's output files back to the sub‐
1176 mitting machine only when the job completes (exits on its
1177 own).
1178 The ON_EXIT_OR_EVICT option is intended for fault tolerant
1180 jobs which periodically save their own state and can restart
1181 where they left off. In this case, files are spooled to the
1182 submit machine any time the job leaves a remote site, either
1183 because it exited on its own, or was evicted by the HTCondor
1184 system for any reason prior to job completion. The files
1185 spooled back are placed in a directory defined by the value
1186 of the SPOOL configuration variable. Any output files trans‐
1187 ferred back to the submit machine are automatically sent back
1188 out again as input files if the job restarts.
1189 POLICY COMMANDS
1191 max_retries = <integer>
1193 The maximum number of retries allowed for this job (must be
1194 non-negative). If the job fails (does not exit with the suc‐
1195 cess_exit_code exit code) it will be retried up to max_re‐
1196 tries times (unless retries are ceased because of the
1197 retry_until command). If max_retries is not defined, and ei‐
1198 ther retry_until or success_exit_code is, the value of DE‐
1199 FAULT_JOB_MAX_RETRIES will be used for the maximum number of
1200 retries.
1201 The combination of the max_retries, retry_until, and suc‐
1203 cess_exit_code commands causes an appropriate OnExitRemove
1204 expression to be automatically generated. If retry command(s)
1205 and on_exit_remove are both defined, the OnExitRemove expres‐
1206 sion will be generated by OR'ing the expression specified in
1207 OnExitRemove and the expression generated by the retry com‐
1208 mands.
1209 retry_until <Integer | ClassAd Boolean Expression>
1212 An integer value or boolean expression that prevents further
1213 retries from taking place, even if max_retries have not been
1214 exhausted. If retry_until is an integer, the job exiting
1215 with that exit code will cause retries to cease. If retry_un‐
1216 til is a ClassAd expression, the expression evaluating to
1217 True will cause retries to cease.
1218 success_exit_code = <integer>
1220 The exit code that is considered successful for this job. De‐
1221 faults to 0 if not defined.
1222 Note: non-zero values of success_exit_code should generally
1224 not be used for DAG node jobs. At the present time, con‐
1225 dor_dagman does not take into account the value of suc‐
1226 cess_exit_code. This means that, if success_exit_code is set
1227 to a non-zero value, condor_dagman will consider the job
1228 failed when it actually succeeds. For single-proc DAG node
1229 jobs, this can be overcome by using a POST script that takes
1230 into account the value of success_exit_code (although this is
1231 not recommended). For multi-proc DAG node jobs, there is cur‐
1232 rently no way to overcome this limitation.
1233 hold = <True | False>
1236 If hold is set to True, then the submitted job will be placed
1237 into the Hold state. Jobs in the Hold state will not run un‐
1238 til released by condor_release. Defaults to False.
1239 keep_claim_idle = <integer>
1242 An integer number of seconds that a job requests the con‐
1243 dor_schedd to wait before releasing its claim after the job
1244 exits or after the job is removed.
1245 The process by which the condor_schedd claims a condor_startd
1247 is somewhat time-consuming. To amortize this cost, the con‐
1248 dor_schedd tries to reuse claims to run subsequent jobs, af‐
1249 ter a job using a claim is done. However, it can only do this
1250 if there is an idle job in the queue at the moment the previ‐
1251 ous job completes. Sometimes, and especially for the node
1252 jobs when using DAGMan, there is a subsequent job about to be
1253 submitted, but it has not yet arrived in the queue when the
1254 previous job completes. As a result, the condor_schedd re‐
1255 leases the claim, and the next job must wait an entire nego‐
1256 tiation cycle to start. When this submit command is defined
1257 with a non-negative integer, when the job exits, the con‐
1258 dor_schedd tries as usual to reuse the claim. If it cannot,
1259 instead of releasing the claim, the condor_schedd keeps the
1260 claim until either the number of seconds given as a parame‐
1261 ter, or a new job which matches that claim arrives, whichever
1262 comes first. The condor_startd in question will remain in the
1263 Claimed/Idle state, and the original job will be "charged"
1264 (in terms of priority) for the time in this state.
1265 leave_in_queue = <ClassAd Boolean Expression>
1268 When the ClassAd Expression evaluates to True, the job is not
1269 removed from the queue upon completion. This allows the user
1270 of a remotely spooled job to retrieve output files in cases
1271 where HTCondor would have removed them as part of the cleanup
1272 associated with completion. The job will only exit the queue
1273 once it has been marked for removal (via condor_rm, for exam‐
1274 ple) and the leave_in_queue expression has become False.
1275 leave_in_queue defaults to False.
1276 As an example, if the job is to be removed once the output is
1278 retrieved with condor_transfer_data, then use
1279 leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED) ||\
1281 (StageOutFinish == 0))
1282 next_job_start_delay = <ClassAd Boolean Expression>
1286 This expression specifies the number of seconds to delay af‐
1287 ter starting up this job before the next job is started. The
1288 maximum allowed delay is specified by the HTCondor configura‐
1289 tion variable MAX_NEXT_JOB_START_DELAY
1290 , which defaults to 10 minutes. This command does not apply
1291 to scheduler or local universe jobs.
1292 This command has been historically used to implement a form
1294 of job start throttling from the job submitter's perspective.
1295 It was effective for the case of multiple job submission
1296 where the transfer of extremely large input data sets to the
1297 execute machine caused machine performance to suffer. This
1298 command is no longer useful, as throttling should be accom‐
1299 plished through configuration of the condor_schedd daemon.
1300 on_exit_hold = <ClassAd Boolean Expression>
1303 The ClassAd expression is checked when the job exits, and if
1304 True, places the job into the Hold state. If False (the de‐
1305 fault value when not defined), then nothing happens and the
1306 on_exit_remove expression is checked to determine if that
1307 needs to be applied.
1308 For example: Suppose a job is known to run for a minimum of
1310 an hour. If the job exits after less than an hour, the job
1311 should be placed on hold and an e-mail notification sent, in‐
1312 stead of being allowed to leave the queue.
1313 on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1315 This expression places the job on hold if it exits for any
1317 reason before running for an hour. An e-mail will be sent to
1318 the user explaining that the job was placed on hold because
1319 this expression became True.
1320 periodic_* expressions take precedence over on_exit_* expres‐
1322 sions, and *_hold expressions take precedence over a *_remove
1323 expressions.
1324 Only job ClassAd attributes will be defined for use by this
1326 ClassAd expression. This expression is available for the
1327 vanilla, java, parallel, grid, local and scheduler universes.
1328 It is additionally available, when submitted from a Unix ma‐
1329 chine, for the standard universe.
1330 on_exit_hold_reason = <ClassAd String Expression>
1332 When the job is placed on hold due to the on_exit_hold ex‐
1333 pression becoming True, this expression is evaluated to set
1334 the value of HoldReason in the job ClassAd. If this expres‐
1335 sion is UNDEFINED or produces an empty or invalid string, a
1336 default description is used.
1337 on_exit_hold_subcode = <ClassAd Integer Expression>
1340 When the job is placed on hold due to the on_exit_hold ex‐
1341 pression becoming True, this expression is evaluated to set
1342 the value of HoldReasonSubCode in the job ClassAd. The de‐
1343 fault subcode is 0. The HoldReasonCode will be set to 3,
1344 which indicates that the job went on hold due to a job policy
1345 expression.
1346 on_exit_remove = <ClassAd Boolean Expression>
1349 The ClassAd expression is checked when the job exits, and if
1350 True (the default value when undefined), then it allows the
1351 job to leave the queue normally. If False, then the job is
1352 placed back into the Idle state. If the user job runs under
1353 the vanilla universe, then the job restarts from the begin‐
1354 ning. If the user job runs under the standard universe, then
1355 it continues from where it left off, using the last check‐
1356 point.
1357 For example, suppose a job occasionally segfaults, but
1359 chances are that the job will finish successfully if the job
1360 is run again with the same data. The on_exit_remove expres‐
1361 sion can cause the job to run again with the following com‐
1362 mand. Assume that the signal identifier for the segmentation
1363 fault is 11 on the platform where the job will be running.
1364 on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)
1366 This expression lets the job leave the queue if the job was
1368 not killed by a signal or if it was killed by a signal other
1369 than 11, representing segmentation fault in this example. So,
1370 if the exited due to signal 11, it will stay in the job
1371 queue. In any other case of the job exiting, the job will
1372 leave the queue as it normally would have done.
1373 As another example, if the job should only leave the queue if
1375 it exited on its own with status 0, this on_exit_remove ex‐
1376 pression works well:
1377 on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
1379 If the job was killed by a signal or exited with a non-zero
1381 exit status, HTCondor would leave the job in the queue to run
1382 again.
1383 periodic_* expressions take precedence over on_exit_* expres‐
1385 sions, and *_hold expressions take precedence over a *_remove
1386 expressions.
1387 Only job ClassAd attributes will be defined for use by this
1389 ClassAd expression.
1390 periodic_hold = <ClassAd Boolean Expression>
1392 This expression is checked periodically when the job is not
1393 in the Held state. If it becomes True, the job will be placed
1394 on hold. If unspecified, the default value is False.
1395 periodic_* expressions take precedence over on_exit_* expres‐
1397 sions, and *_hold expressions take precedence over a *_remove
1398 expressions.
1399 Only job ClassAd attributes will be defined for use by this
1401 ClassAd expression. Note that, by default, this expression is
1402 only checked once every 60 seconds. The period of these eval‐
1403 uations can be adjusted by setting the PERIODIC_EXPR_INTER‐
1404 VAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1405 configuration macros.
1406 periodic_hold_reason = <ClassAd String Expression>
1409 When the job is placed on hold due to the periodic_hold ex‐
1410 pression becoming True, this expression is evaluated to set
1411 the value of HoldReason in the job ClassAd. If this expres‐
1412 sion is UNDEFINED or produces an empty or invalid string, a
1413 default description is used.
1414 periodic_hold_subcode = <ClassAd Integer Expression>
1417 When the job is placed on hold due to the periodic_hold ex‐
1418 pression becoming true, this expression is evaluated to set
1419 the value of HoldReasonSubCode in the job ClassAd. The de‐
1420 fault subcode is 0. The HoldReasonCode will be set to 3,
1421 which indicates that the job went on hold due to a job policy
1422 expression.
1423 periodic_release = <ClassAd Boolean Expression>
1426 This expression is checked periodically when the job is in
1427 the Held state. If the expression becomes True, the job will
1428 be released.
1429 Only job ClassAd attributes will be defined for use by this
1431 ClassAd expression. Note that, by default, this expression is
1432 only checked once every 60 seconds. The period of these eval‐
1433 uations can be adjusted by setting the PERIODIC_EXPR_INTER‐
1434 VAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1435 configuration macros.
1436 periodic_remove = <ClassAd Boolean Expression>
1439 This expression is checked periodically. If it becomes True,
1440 the job is removed from the queue. If unspecified, the de‐
1441 fault value is False.
1442 See the Examples section of this manual page for an example
1444 of a periodic_remove expression.
1445 periodic_* expressions take precedence over on_exit_* expres‐
1447 sions, and *_hold expressions take precedence over a *_remove
1448 expressions. So, the periodic_remove expression takes prece‐
1449 dent over the on_exit_remove expression, if the two describe
1450 conflicting actions.
1451 Only job ClassAd attributes will be defined for use by this
1453 ClassAd expression. Note that, by default, this expression is
1454 only checked once every 60 seconds. The period of these eval‐
1455 uations can be adjusted by setting the PERIODIC_EXPR_INTER‐
1456 VAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1457 configuration macros.
1458 COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
1460 allow_startup_script = <True | False>
1463 If True, a standard universe job will execute a script in‐
1464 stead of submitting the job, and the consistency check to see
1465 if the executable has been linked using condor_compile is
1466 omitted. The executable command within the submit descrip‐
1467 tion file specifies the name of the script. The script is
1468 used to do preprocessing before the job is submitted. The
1469 shell script ends with an exec of the job executable, such
1470 that the process id of the executable is the same as that of
1471 the shell script. Here is an example script that gets a copy
1472 of a machine-specific executable before the exec.
1473 #! /bin/sh
1475 # get the host name of the machine
1477 $host=`uname -n`
1478 # grab a standard universe executable designed specifically
1480 # for this host
1481 scp elsewhere@cs.wisc.edu:${host} executable
1482 # The PID MUST stay the same, so exec the new standard universe process.
1484 exec executable ${1+"$@"}
1485 If this command is not present (defined), then the value de‐
1487 faults to false.
1488 append_files = file1, file2, ...
1490 If your job attempts to access a file mentioned in this list,
1491 HTCondor will force all writes to that file to be appended to
1492 the end. Furthermore, condor_submit will not truncate it.
1493 This list uses the same syntax as compress_files, shown
1494 above.
1495 This option may yield some surprising results. If several
1497 jobs attempt to write to the same file, their output may be
1498 intermixed. If a job is evicted from one or more machines
1499 during the course of its lifetime, such an output file might
1500 contain several copies of the results. This option should be
1501 only be used when you wish a certain file to be treated as a
1502 running log instead of a precise result.
1503 This option only applies to standard-universe jobs.
1505 buffer_files = < name = (size,block-size) ; name2 =
1510 (size,block-size) ... >; buffer_size = <bytes-in-buffer>; buf‐
1511 fer_block_size = <bytes-in-block>
1512 HTCondor keeps a buffer of recently-used data for each file a
1513 job accesses. This buffer is used both to cache commonly-used
1514 data and to consolidate small reads and writes into larger
1515 operations that get better throughput. The default settings
1516 should produce reasonable results for most programs.
1517 These options only apply to standard-universe jobs.
1519 If needed, you may set the buffer controls individually for
1521 each file using the buffer_files option. For example, to set
1522 the buffer size to 1 MiB and the block size to 256 KiB for
1523 the file input.data, use this command:
1524 buffer_files = "input.data=(1000000,256000)"
1526 Alternatively, you may use these two options to set the de‐
1528 fault sizes for all files used by your job:
1529 buffer_size = 1000000
1531 buffer_block_size = 256000
1532 If you do not set these, HTCondor will use the values given
1534 by these two configuration file macros:
1535 DEFAULT_IO_BUFFER_SIZE = 1000000
1537 DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
1538 Finally, if no other settings are present, HTCondor will use
1540 a buffer of 512 KiB and a block size of 32 KiB.
1541 compress_files = file1, file2, ...
1544 If your job attempts to access any of the files mentioned in
1545 this list, HTCondor will automatically compress them (if
1546 writing) or decompress them (if reading). The compress format
1547 is the same as used by GNU gzip.
1548 The files given in this list may be simple file names or com‐
1550 plete paths and may include * as a wild card. For example,
1551 this list causes the file /tmp/data.gz, any file named
1552 event.gz, and any file ending in .gzip to be automatically
1553 compressed or decompressed as needed:
1554 compress_files = /tmp/data.gz, event.gz, *.gzip
1556 Due to the nature of the compression format, compressed files
1558 must only be accessed sequentially. Random access reading is
1559 allowed but is very slow, while random access writing is sim‐
1560 ply not possible. This restriction may be avoided by using
1561 both compress_files and fetch_files at the same time. When
1562 this is done, a file is kept in the decompressed state at the
1563 execution machine, but is compressed for transfer to its
1564 original location.
1565 This option only applies to standard universe jobs.
1567 fetch_files = file1, file2, ...
1570 If your job attempts to access a file mentioned in this list,
1571 HTCondor will automatically copy the whole file to the exe‐
1572 cuting machine, where it can be accessed quickly. When your
1573 job closes the file, it will be copied back to its original
1574 location. This list uses the same syntax as compress_files,
1575 shown above.
1576 This option only applies to standard universe jobs.
1578 file_remaps = < name = newname ; name2 = newname2 ... >
1581 Directs HTCondor to use a new file name in place of an old
1582 one. name describes a file name that your job may attempt to
1583 open, and newname describes the file name it should be re‐
1584 placed with. newname may include an optional leading access
1585 specifier, local: or remote:. If left unspecified, the de‐
1586 fault access specifier is remote:. Multiple remaps can be
1587 specified by separating each with a semicolon.
1588 This option only applies to standard universe jobs.
1590 If you wish to remap file names that contain equals signs or
1592 semicolons, these special characters may be escaped with a
1593 backslash.
1594 Example One:
1596 Suppose that your job reads a file named
1597 dataset.1. To instruct HTCondor to force your job
1598 to read other.dataset instead, add this to the
1599 submit file:
1600 file_remaps = "dataset.1=other.dataset"
1602 Example Two:
1604 Suppose that your run many jobs which all read in
1605 the same large file, called very.big. If this file
1606 can be found in the same place on a local disk in
1607 every machine in the pool, (say /bigdisk/bigfile,)
1608 you can instruct HTCondor of this fact by remap‐
1609 ping very.big to /bigdisk/bigfile and specifying
1610 that the file is to be read locally, which will be
1611 much faster than reading over the network.
1612 file_remaps = "very.big = local:/bigdisk/bigfile"
1614 Example Three:
1616 Several remaps can be applied at once by separat‐
1617 ing each with a semicolon.
1618 file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 = other.dataset"
1620 local_files = file1, file2, ...
1624 If your job attempts to access a file mentioned in this list,
1625 HTCondor will cause it to be read or written at the execution
1626 machine. This is most useful for temporary files not used for
1627 input or output. This list uses the same syntax as com‐
1628 press_files, shown above.
1629 local_files = /tmp/*
1631 This option only applies to standard universe jobs.
1633 want_remote_io = <True | False>
1636 This option controls how a file is opened and manipulated in
1637 a standard universe job. If this option is true, which is the
1638 default, then the condor_shadow makes all decisions about how
1639 each and every file should be opened by the executing job.
1640 This entails a network round trip (or more) from the job to
1641 the condor_shadow and back again for every single open() in
1642 addition to other needed information about the file. If set
1643 to false, then when the job queries the condor_shadow for the
1644 first time about how to open a file, the condor_shadow will
1645 inform the job to automatically perform all of its file ma‐
1646 nipulation on the local file system on the execute machine
1647 and any file remapping will be ignored. This means that there
1648 must be a shared file system (such as NFS or AFS) between the
1649 execute machine and the submit machine and that ALL paths
1650 that the job could open on the execute machine must be valid.
1651 The ability of the standard universe job to checkpoint, pos‐
1652 sibly to a checkpoint server, is not affected by this attri‐
1653 bute. However, when the job resumes it will be expecting the
1654 same file system conditions that were present when the job
1655 checkpointed.
1656 COMMANDS FOR THE GRID
1658 azure_admin_key = <pathname>
1660 For grid type azure jobs, specifies the path and file name of
1661 a file that contains an SSH public key. This key can be used
1662 to log into the administrator account of the instance via
1663 SSH.
1664 azure_admin_username = <account name>
1667 For grid type azure jobs, specifies the name of an adminis‐
1668 trator account to be created in the instance. This account
1669 can be logged into via SSH.
1670 azure_auth_file = <pathname>
1672 For grid type azure jobs, specifies a path and file name of
1673 the authorization file that grants permission for HTCondor to
1674 use the Azure account. If it's not defined, then HTCondor
1675 will attempt to use the default credentials of the Azure CLI
1676 tools.
1677 azure_image = <image id>
1680 For grid type azure jobs, identifies the disk image to be
1681 used for the boot disk of the instance. This image must al‐
1682 ready be registered within Azure.
1683 azure_location = <image id>
1686 For grid type azure jobs, identifies the location within
1687 Azure where the instance should be run. As an example, one
1688 current location is centralus.
1689 azure_size = <machine type>
1692 For grid type azure jobs, the hardware configuration that the
1693 virtual machine instance is to run on.
1694 batch_queue = <queuename>
1697 Used for pbs, lsf, and sge grid universe jobs. Specifies the
1698 name of the PBS/LSF/SGE job queue into which the job should
1699 be submitted. If not specified, the default queue is used.
1700 boinc_authenticator_file = <pathname>
1703 For grid type boinc jobs, specifies a path and file name of
1704 the authorization file that grants permission for HTCondor to
1705 use the BOINC service. There is no default value when not
1706 specified.
1707 cream_attributes = <name=value;...;name=value>
1710 Provides a list of attribute/value pairs to be set in a CREAM
1711 job description of a grid universe job destined for the CREAM
1712 grid system. The pairs are separated by semicolons, and writ‐
1713 ten in New ClassAd syntax.
1714 delegate_job_GSI_credentials_lifetime = <seconds>
1717 Specifies the maximum number of seconds for which delegated
1718 proxies should be valid. The default behavior when this com‐
1719 mand is not specified is determined by the configuration
1720 variable DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME
1721 , which defaults to one day. A value of 0 indicates that the
1722 delegated proxy should be valid for as long as allowed by the
1723 credential used to create the proxy. This setting currently
1724 only applies to proxies delegated for non-grid jobs and for
1725 HTCondor-C jobs. It does not currently apply to globus grid
1726 jobs, which always behave as though this setting were 0. This
1727 variable has no effect if the configuration variable DELE‐
1728 GATE_JOB_GSI_CREDENTIALS
1729 is False, because in that case the job proxy is copied
1730 rather than delegated.
1731 ec2_access_key_id = <pathname>
1734 For grid type ec2 jobs, identifies the file containing the
1735 access key.
1736 ec2_ami_id = <EC2 xMI ID>
1738 For grid type ec2 jobs, identifies the machine image. Ser‐
1739 vices compatible with the EC2 Query API may refer to these
1740 with abbreviations other than AMI, for example EMI is valid
1741 for Eucalyptus.
1742 ec2_availability_zone = <zone name>
1744 For grid type ec2 jobs, specifies the Availability Zone that
1745 the instance should be run in. This command is optional, un‐
1746 less ec2_ebs_volumes is set. As an example, one current zone
1747 is us-east-1b.
1748 ec2_block_device_mapping = <block-device>:<kernel-device>,<block-de‐
1751 vice>:<kernel-device>, ...
1752 For grid type ec2 jobs, specifies the block device to kernel
1753 device mapping. This command is optional.
1754 ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device
1757 name>,...
1758 For grid type ec2 jobs, optionally specifies a list of Elas‐
1759 tic Block Store (EBS) volumes to be made available to the in‐
1760 stance and the device names they should have in the instance.
1761 ec2_elastic_ip = <elastic IP address>
1764 For grid type ec2 jobs, and optional specification of an
1765 Elastic IP address that should be assigned to this instance.
1766 ec2_iam_profile_arn = <IAM profile ARN>
1769 For grid type ec2 jobs, an Amazon Resource Name (ARN) identi‐
1770 fying which Identity and Access Management (IAM) (instance)
1771 profile to associate with the instance.
1772 ec2_iam_profile_name= <IAM profile name>
1775 For grid type ec2 jobs, a name identifying which Identity and
1776 Access Management (IAM) (instance) profile to associate with
1777 the instance.
1778 ec2_instance_type = <instance type>
1780 For grid type ec2 jobs, identifies the instance type. Differ‐
1781 ent services may offer different instance types, so no de‐
1782 fault value is set.
1783 ec2_keypair = <ssh key-pair name>
1785 For grid type ec2 jobs, specifies the name of an SSH key-pair
1786 that is already registered with the EC2 service. The associ‐
1787 ated private key can be used to ssh into the virtual machine
1788 once it is running.
1789 ec2_keypair_file = <pathname>
1791 For grid type ec2 jobs, specifies the complete path and file
1792 name of a file into which HTCondor will write an SSH key for
1793 use with ec2 jobs. The key can be used to ssh into the vir‐
1794 tual machine once it is running. If ec2_keypair is speci‐
1795 fied for a job, ec2_keypair_file is ignored.
1796 ec2_parameter_names = ParameterName1, ParameterName2, ...
1798 For grid type ec2 jobs, a space or comma separated list of
1799 the names of additional parameters to pass when instantiating
1800 an instance.
1801 ec2_parameter_<name> = <value>
1803 For grid type ec2 jobs, specifies the value for the corre‐
1804 spondingly named (instance instantiation) parameter. <name>
1805 is the parameter name specified in the submit command ec2_pa‐
1806 rameter_names , but with any periods replaced by under‐
1807 scores.
1808 ec2_secret_access_key = <pathname>
1811 For grid type ec2 jobs, specifies the path and file name con‐
1812 taining the secret access key.
1813 ec2_security_groups = group1, group2, ...
1816 For grid type ec2 jobs, defines the list of EC2 security
1817 groups which should be associated with the job.
1818 ec2_security_ids = id1, id2, ...
1821 For grid type ec2 jobs, defines the list of EC2 security
1822 group IDs which should be associated with the job.
1823 ec2_spot_price = <bid>
1826 For grid type ec2 jobs, specifies the spot instance bid,
1827 which is the most that the job submitter is willing to pay
1828 per hour to run this job.
1829 ec2_tag_names = <name0,name1,name...>
1831 For grid type ec2 jobs, specifies the case of tag names that
1832 will be associated with the running instance. This is only
1833 necessary if a tag name case matters. By default the list
1834 will be automatically generated.
1835 ec2_tag_<name> = <value>
1838 For grid type ec2 jobs, specifies a tag to be associated with
1839 the running instance. The tag name will be lower-cased, use
1840 ec2_tag_names to change the case.
1841 WantNameTag = <True | False>
1843 For grid type ec2 jobs, a job may request that its 'name' tag
1844 be (not) set by HTCondor. If the job does not otherwise spec‐
1845 ify any tags, not setting its name tag will eliminate a call
1846 by the EC2 GAHP, improving performance.
1847 ec2_user_data = <data>
1850 For grid type ec2 jobs, provides a block of data that can be
1851 accessed by the virtual machine. If both ec2_user_data and
1852 ec2_user_data_file are specified for a job, the two blocks of
1853 data are concatenated, with the data from this ec2_user_data
1854 submit command occurring first.
1855 ec2_user_data_file = <pathname>
1857 For grid type ec2 jobs, specifies a path and file name whose
1858 contents can be accessed by the virtual machine. If both
1859 ec2_user_data and ec2_user_data_file are specified for a job,
1860 the two blocks of data are concatenated, with the data from
1861 that ec2_user_data submit command occurring first.
1862 ec2_vpc_ip = <a.b.c.d>
1864 For grid type ec2 jobs, that are part of a Virtual Private
1865 Cloud (VPC), an optional specification of the IP address that
1866 this instance should have within the VPC.
1867 ec2_vpc_subnet = <subnet specification string>
1870 For grid type ec2 jobs, an optional specification of the Vir‐
1871 tual Private Cloud (VPC) that this instance should be a part
1872 of.
1873 gce_account = <account name>
1876 For grid type gce jobs, specifies the Google cloud services
1877 account to use. If this submit command isn't specified, then
1878 a random account from the authorization file given by
1879 gce_auth_file will be used.
1880 gce_auth_file = <pathname>
1882 For grid type gce jobs, specifies a path and file name of the
1883 authorization file that grants permission for HTCondor to use
1884 the Google account. If this command is not specified, then
1885 the default file of the Google command-line tools will be
1886 used.
1887 gce_image = <image id>
1890 For grid type gce jobs, the identifier of the virtual machine
1891 image representing the HTCondor job to be run. This virtual
1892 machine image must already be register with GCE and reside in
1893 Google's Cloud Storage service.
1894 gce_json_file = <pathname>
1896 For grid type gce jobs, specifies the path and file name of a
1897 file that contains JSON elements that should be added to the
1898 instance description submitted to the GCE service.
1899 gce_machine_type = <machine type>
1902 For grid type gce jobs, the long form of the URL that de‐
1903 scribes the machine configuration that the virtual machine
1904 instance is to run on.
1905 gce_metadata = <name=value,...,name=value>
1907 For grid type gce jobs, a comma separated list of name and
1908 value pairs that define metadata for a virtual machine in‐
1909 stance that is an HTCondor job.
1910 gce_metadata_file = <pathname>
1912 For grid type gce jobs, specifies a path and file name of the
1913 file that contains metadata for a virtual machine instance
1914 that is an HTCondor job. Within the file, each name and value
1915 pair is on its own line; so, the pairs are separated by the
1916 newline character.
1917 gce_preemptible = <True | False>
1920 For grid type gce jobs, specifies whether the virtual machine
1921 instance should be preemptible. The default is for the in‐
1922 stance to not be preemptible.
1923 globus_rematch = <ClassAd Boolean Expression>
1925 This expression is evaluated by the condor_gridmanager when‐
1926 ever:
1927 1. the globus_resubmit expression evaluates to True
1929 2. the condor_gridmanager decides it needs to retry a submis‐
1931 sion (as when a previous submission failed to commit)
1932 If globus_rematch evaluates to True, then before the job is
1934 submitted again to globus, the condor_gridmanager will re‐
1935 quest that the condor_schedd daemon renegotiate with the
1936 matchmaker (the condor_negotiator). The result is this job
1937 will be matched again.
1938 globus_resubmit = <ClassAd Boolean Expression>
1941 The expression is evaluated by the condor_gridmanager each
1942 time the condor_gridmanager gets a job ad to manage. There‐
1943 fore, the expression is evaluated:
1944 1. when a grid universe job is first submitted to HTCondor-G
1946 2. when a grid universe job is released from the hold state
1948 3. when HTCondor-G is restarted (specifically, whenever the
1950 condor_gridmanager is restarted)
1951 If the expression evaluates to True, then any previous sub‐
1953 mission to the grid universe will be forgotten and this job
1954 will be submitted again as a fresh submission to the grid
1955 universe. This may be useful if there is a desire to give up
1956 on a previous submission and try again. Note that this may
1957 result in the same job running more than once. Do not treat
1958 this operation lightly.
1959 globus_rsl = <RSL-string>
1962 Used to provide any additional Globus RSL string attributes
1963 which are not covered by other submit description file com‐
1964 mands or job attributes. Used for grid universe jobs, where
1965 the grid resource has a grid-type-string of gt2.
1966 grid_resource = <grid-type-string> <grid-specific-parameter-list>
1969 For each grid-type-string value, there are further type-spe‐
1970 cific values that must specified. This submit description
1971 file command allows each to be given in a space-separated
1972 list. Allowable grid-type-string values are batch, condor,
1973 cream, ec2, gt2, gt5, lsf, nordugrid, pbs, sge, and unicore.
1974 The HTCondor manual chapter on Grid Computing details the va‐
1975 riety of grid types.
1976 For a grid-type-string of batch, the single parameter is the
1978 name of the local batch system, and will be one of pbs, lsf,
1979 or sge.
1980 For a grid-type-string of condor, the first parameter is the
1982 name of the remote condor_schedd daemon. The second parameter
1983 is the name of the pool to which the remote condor_schedd
1984 daemon belongs.
1985 For a grid-type-string of cream, there are three parameters.
1987 The first parameter is the web services address of the CREAM
1988 server. The second parameter is the name of the batch system
1989 that sits behind the CREAM server. The third parameter iden‐
1990 tifies a site-specific queue within the batch system.
1991 For a grid-type-string of ec2, one additional parameter spec‐
1993 ifies the EC2 URL.
1994 For a grid-type-string of gt2, the single parameter is the
1996 name of the pre-WS GRAM resource to be used.
1997 For a grid-type-string of gt5, the single parameter is the
1999 name of the pre-WS GRAM resource to be used, which is the
2000 same as for the grid-type-string of gt2.
2001 For a grid-type-string of lsf, no additional parameters are
2003 used.
2004 For a grid-type-string of nordugrid, the single parameter is
2006 the name of the NorduGrid resource to be used.
2007 For a grid-type-string of pbs, no additional parameters are
2009 used.
2010 For a grid-type-string of sge, no additional parameters are
2012 used.
2013 For a grid-type-string of unicore, the first parameter is the
2015 name of the Unicore Usite to be used. The second parameter is
2016 the name of the Unicore Vsite to be used.
2017 keystore_alias = <name>
2020 A string to locate the certificate in a Java keystore file,
2021 as used for a unicore job.
2022 keystore_file = <pathname>
2025 The complete path and file name of the Java keystore file
2026 containing the certificate to be used for a unicore job.
2027 keystore_passphrase_file = <pathname>
2030 The complete path and file name to the file containing the
2031 passphrase protecting a Java keystore file containing the
2032 certificate. Relevant for a unicore job.
2033 MyProxyCredentialName = <symbolic name>
2036 The symbolic name that identifies a credential to the MyProxy
2037 server. This symbolic name is set as the credential is ini‐
2038 tially stored on the server (using myproxy-init).
2039 MyProxyHost = <host>:<port>
2042 The Internet address of the host that is the MyProxy server.
2043 The host may be specified by either a host name (as in
2044 head.example.com) or an IP address (of the form 123.456.7.8).
2045 The port number is an integer.
2046 MyProxyNewProxyLifetime = <number-of-minutes>
2049 The new lifetime (in minutes) of the proxy after it is re‐
2050 freshed.
2051 MyProxyPassword = <password>
2054 The password needed to refresh a credential on the MyProxy
2055 server. This password is set when the user initially stores
2056 credentials on the server (using myproxy-init). As an alter‐
2057 native to using MyProxyPassword in the submit description
2058 file, the password may be specified as a command line argu‐
2059 ment to condor_submit with the -password argument.
2060 MyProxyRefreshThreshold = <number-of-seconds>
2062 The time (in seconds) before the expiration of a proxy that
2063 the proxy should be refreshed. For example, if MyProxyRe‐
2064 freshThreshold is set to the value 600, the proxy will be re‐
2065 freshed 10 minutes before it expires.
2066 MyProxyServerDN = <credential subject>
2068 A string that specifies the expected Distinguished Name (cre‐
2069 dential subject, abbreviated DN) of the MyProxy server. It
2070 must be specified when the MyProxy server DN does not follow
2071 the conventional naming scheme of a host credential. This oc‐
2072 curs, for example, when the MyProxy server DN begins with a
2073 user credential.
2074 nordugrid_rsl = <RSL-string>
2077 Used to provide any additional RSL string attributes which
2078 are not covered by regular submit description file parame‐
2079 ters. Used when the universe is grid, and the type of grid
2080 system is nordugrid.
2081 transfer_error = <True | False>
2083 For jobs submitted to the grid universe only. If True, then
2084 the error output (from stderr) from the job is transferred
2085 from the remote machine back to the submit machine. The name
2086 of the file after transfer is given by the error command.
2087 If False, no transfer takes place (from the remote machine to
2088 submit machine), and the name of the file is given by the er‐
2089 ror command. The default value is True.
2090 transfer_input = <True | False>
2093 For jobs submitted to the grid universe only. If True, then
2094 the job input (stdin) is transferred from the machine where
2095 the job was submitted to the remote machine. The name of the
2096 file that is transferred is given by the input command. If
2097 False, then the job's input is taken from a pre-staged file
2098 on the remote machine, and the name of the file is given by
2099 the input command. The default value is True.
2100 For transferring files other than stdin, see transfer_in‐
2102 put_files .
2103 transfer_output = <True | False>
2106 For jobs submitted to the grid universe only. If True, then
2107 the output (from stdout) from the job is transferred from the
2108 remote machine back to the submit machine. The name of the
2109 file after transfer is given by the output command. If
2110 False, no transfer takes place (from the remote machine to
2111 submit machine), and the name of the file is given by the
2112 output command. The default value is True.
2113 For transferring files other than stdout, see transfer_out‐
2115 put_files .
2116 use_x509userproxy = <True | False>
2119 Set this command to True to indicate that the job requires an
2120 X.509 user proxy. If x509userproxy is set, then that file is
2121 used for the proxy. Otherwise, the proxy is looked for in the
2122 standard locations. If x509userproxy is set or if the job is
2123 a grid universe job of grid type gt2, gt5, cream, or nor‐
2124 dugrid, then the value of use_x509userproxy is forced to
2125 True. Defaults to False.
2126 x509userproxy = <full-pathname>
2129 Used to override the default path name for X.509 user cer‐
2130 tificates. The default location for X.509 proxies is the
2131 /tmp directory, which is generally a local file system. Set‐
2132 ting this value would allow HTCondor to access the proxy in a
2133 shared file system (for example, AFS). HTCondor will use the
2134 proxy specified in the submit description file first. If
2135 nothing is specified in the submit description file, it will
2136 use the environment variable X509_USER_PROXY. If that vari‐
2137 able is not present, it will search in the default location.
2138 Note that proxies are only valid for a limited time. Con‐
2139 dor_submit will not submit a job with an expired proxy, it
2140 will return an error. Also, if the configuration parameter
2141 CRED_MIN_TIME_LEFT is set to some number of seconds, and if
2142 the proxy will expire before that many seconds, condor_submit
2143 will also refuse to submit the job. That is, if
2144 CRED_MIN_TIME_LEFT is set to 60, condor_submit will refuse to
2145 submit a job whose proxy will expire 60 seconds from the time
2146 of submission.
2147 x509userproxy is relevant when the universe is vanilla, or
2149 when the universe is grid and the type of grid system is one
2150 of gt2, gt5, condor, cream, or nordugrid. Defining a value
2151 causes the proxy to be delegated to the execute machine.
2152 Further, VOMS attributes defined in the proxy will appear in
2153 the job ClassAd.
2154 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2156 hold_kill_sig = <signal-number>
2159 For the scheduler universe only, signal-number is the sig‐
2160 nal delivered to the job when the job is put on hold with
2161 condor_hold. signal-number may be either the platform-spe‐
2162 cific name or value of the signal. If this command is not
2163 present, the value of kill_sig is used.
2164 jar_files = <file_list>
2167 Specifies a list of additional JAR files to include when us‐
2168 ing the Java universe. JAR files will be transferred along
2169 with the executable and automatically added to the classpath.
2170 java_vm_args = <argument_list>
2173 Specifies a list of additional arguments to the Java VM it‐
2174 self, When HTCondor runs the Java program, these are the ar‐
2175 guments that go before the class name. This can be used to
2176 set VM-specific arguments like stack size, garbage-collector
2177 arguments and initial property values.
2178 machine_count = <max>
2180 For the parallel universe, a single value (max) is required.
2181 It is neither a maximum or minimum, but the number of ma‐
2182 chines to be dedicated toward running the job.
2183 remove_kill_sig = <signal-number>
2186 For the scheduler universe only, signal-number is the sig‐
2187 nal delivered to the job when the job is removed with con‐
2188 dor_rm. signal-number may be either the platform-specific
2189 name or value of the signal. This example shows it both ways
2190 for a Linux signal:
2191 remove_kill_sig = SIGUSR1
2193 remove_kill_sig = 10
2194 If this command is not present, the value of kill_sig is
2196 used.
2197 COMMANDS FOR THE VM UNIVERSE
2199 vm_disk = file1:device1:permission1, file2:device2:permission2:for‐
2201 mat2, ...
2202 A list of comma separated disk files. Each disk file is spec‐
2203 ified by 4 colon separated fields. The first field is the
2204 path and file name of the disk file. The second field speci‐
2205 fies the device. The third field specifies permissions, and
2206 the optional fourth field specifies the image format. If a
2207 disk file will be transferred by HTCondor, then the first
2208 field should just be the simple file name (no path informa‐
2209 tion).
2210 An example that specifies two disk files:
2212 vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2214 vm_checkpoint = <True | False>
2218 A boolean value specifying whether or not to take check‐
2219 points. If not specified, the default value is False. In the
2220 current implementation, setting both vm_checkpoint and
2221 vm_networking to True does not yet work in all cases. Net‐
2222 working cannot be used if a vm universe job uses a checkpoint
2223 in order to continue execution after migration to another ma‐
2224 chine.
2225 vm_macaddr = <MACAddr>
2228 Defines that MAC address that the virtual machine's network
2229 interface should have, in the standard format of six groups
2230 of two hexadecimal digits separated by colons.
2231 vm_memory = <MBytes-of-memory>
2234 The amount of memory in MBytes that a vm universe job re‐
2235 quires.
2236 vm_networking = <True | False>
2239 Specifies whether to use networking or not. In the current
2240 implementation, setting both vm_checkpoint and vm_networking
2241 to True does not yet work in all cases. Networking cannot be
2242 used if a vm universe job uses a checkpoint in order to con‐
2243 tinue execution after migration to another machine.
2244 vm_networking_type = <nat | bridge >
2247 When vm_networking is True, this definition augments the
2248 job's requirements to match only machines with the specified
2249 networking. If not specified, then either networking type
2250 matches.
2251 vm_no_output_vm = <True | False>
2254 When True, prevents HTCondor from transferring output files
2255 back to the machine from which the vm universe job was sub‐
2256 mitted. If not specified, the default value is False.
2257 vm_type = <vmware | xen | kvm>
2260 Specifies the underlying virtual machine software that this
2261 job expects.
2262 vmware_dir = <pathname>
2264 The complete path and name of the directory where VMware-spe‐
2265 cific files and applications such as the VMDK (Virtual Ma‐
2266 chine Disk Format) and VMX (Virtual Machine Configuration)
2267 reside. This command is optional; when not specified, all
2268 relevant VMware image files are to be listed using trans‐
2269 fer_input_files .
2270 vmware_should_transfer_files = <True | False>
2273 Specifies whether HTCondor will transfer VMware-specific
2274 files located as specified by vmware_dir to the execute ma‐
2275 chine (True) or rely on access through a shared file system
2276 (False). Omission of this required command (for VMware vm
2277 universe jobs) results in an error message from condor_sub‐
2278 mit, and the job will not be submitted.
2279 vmware_snapshot_disk = <True | False>
2282 When True, causes HTCondor to utilize a VMware snapshot disk
2283 for new or modified files. If not specified, the default
2284 value is True.
2285 xen_initrd = <image-file>
2287 When xen_kernel gives a file name for the kernel image to
2288 use, this optional command may specify a path to a ramdisk
2289 (initrd) image file. If the image file will be transferred by
2290 HTCondor, then the value should just be the simple file name
2291 (no path information).
2292 xen_kernel = <included | path-to-kernel>
2295 A value of included specifies that the kernel is included in
2296 the disk file. If not one of these values, then the value is
2297 a path and file name of the kernel to be used. If a kernel
2298 file will be transferred by HTCondor, then the value should
2299 just be the simple file name (no path information).
2300 xen_kernel_params = <string>
2302 A string that is appended to the Xen kernel command line.
2303 xen_root = <string>
2306 A string that is appended to the Xen kernel command line to
2307 specify the root device. This string is required when
2308 xen_kernel gives a path to a kernel. Omission for this re‐
2309 quired case results in an error message during submission.
2310 COMMANDS FOR THE DOCKER UNIVERSE
2312 docker_image = < image-name >
2315 Defines the name of the Docker image that is the basis for
2316 the docker container.
2317 ADVANCED COMMANDS
2319 accounting_group = <accounting-group-name>
2321 Causes jobs to negotiate under the given accounting group.
2322 This value is advertised in the job ClassAd as AcctGroup. The
2323 HTCondor Administrator's manual contains more information
2324 about accounting groups.
2325 accounting_group_user = <accounting-group-user-name>
2328 Sets the name associated with this job to be used for re‐
2329 source usage accounting purposes, such as computation of
2330 fair-share priority and reporting via condor_userprio. If
2331 not set, defaults to the value of the job ClassAd attribute
2332 Owner. This value is advertised in the job ClassAd as Acct‐
2333 GroupUser.
2334 concurrency_limits = <string-list>
2337 A list of resources that this job needs. The resources are
2338 presumed to have concurrency limits placed upon them, thereby
2339 limiting the number of concurrent jobs in execution which
2340 need the named resource. Commas and space characters delimit
2341 the items in the list. Each item in the list is a string
2342 that identifies the limit, or it is a ClassAd expression that
2343 evaluates to a string, and it is evaluated in the context of
2344 machine ClassAd being considered as a match. Each item in the
2345 list also may specify a numerical value identifying the inte‐
2346 ger number of resources required for the job. The syntax
2347 follows the resource name by a colon character (:) and the
2348 numerical value. Details on concurrency limits are in the HT‐
2349 Condor Administrator's manual.
2350 concurrency_limits_expr = <ClassAd String Expression>
2353 A ClassAd expression that represents the list of resources
2354 that this job needs after evaluation. The ClassAd expression
2355 may specify machine ClassAd attributes that are evaluated
2356 against a matched machine. After evaluation, the list sets
2357 concurrency_limits.
2358 copy_to_spool = <True | False>
2361 If copy_to_spool is True, then condor_submit copies the exe‐
2362 cutable to the local spool directory before running it on a
2363 remote host. As copying can be quite time consuming and un‐
2364 necessary, the default value is False for all job universes
2365 other than the standard universe. When False, condor_submit
2366 does not copy the executable to a local spool directory. The
2367 default is True in standard universe, because resuming execu‐
2368 tion from a checkpoint can only be guaranteed to work using
2369 precisely the same executable that created the checkpoint.
2370 coresize = <size>
2372 Should the user's program abort and produce a core file,
2373 coresize specifies the maximum size in bytes of the core file
2374 which the user wishes to keep. If coresize is not specified
2375 in the command file, the system's user resource limit core‐
2376 dumpsize is used (note that coredumpsize is not an HTCondor
2377 parameter - it is an operating system parameter that can be
2378 viewed with the limit or ulimit command on Unix and in the
2379 Registry on Windows). A value of -1 results in no limits be‐
2380 ing applied to the core file size. If HTCondor is running as
2381 root, a coresize setting greater than the system coredumpsize
2382 limit will override the system setting; if HTCondor is not
2383 running as root, the system coredumpsize limit will override
2384 coresize.
2385 cron_day_of_month = <Cron-evaluated Day>
2388 The set of days of the month for which a deferral time ap‐
2389 plies. The HTCondor User's manual section on Time Scheduling
2390 for Job Execution has further details.
2391 cron_day_of_week = <Cron-evaluated Day>
2394 The set of days of the week for which a deferral time ap‐
2395 plies. The HTCondor User's manual section on Time Scheduling
2396 for Job Execution has further details.
2397 cron_hour = <Cron-evaluated Hour>
2399 The set of hours of the day for which a deferral time ap‐
2400 plies. The HTCondor User's manual section on Time Scheduling
2401 for Job Execution has further details.
2402 cron_minute = <Cron-evaluated Minute>
2404 The set of minutes within an hour for which a deferral time
2405 applies. The HTCondor User's manual section on Time Schedul‐
2406 ing for Job Execution has further details.
2407 cron_month = <Cron-evaluated Month>
2410 The set of months within a year for which a deferral time ap‐
2411 plies. The HTCondor User's manual section on Time Scheduling
2412 for Job Execution has further details.
2413 cron_prep_time = <ClassAd Integer Expression>
2416 Analogous to deferral_prep_time . The number of seconds
2417 prior to a job's deferral time that the job may be matched
2418 and sent to an execution machine.
2419 cron_window = <ClassAd Integer Expression>
2422 Analogous to the submit command deferral_window . It allows
2423 cron jobs that miss their deferral time to begin execution.
2424 The HTCondor User's manual section on Time Scheduling for Job
2426 Execution has further details.
2427 dagman_log = <pathname>
2430 DAGMan inserts this command to specify an event log that it
2431 watches to maintain the state of the DAG. If the log com‐
2432 mand is not specified in the submit file, DAGMan uses the log
2433 command to specify the event log.
2434 deferral_prep_time = <ClassAd Integer Expression>
2436 The number of seconds prior to a job's deferral time that the
2437 job may be matched and sent to an execution machine.
2438 The HTCondor User's manual section on Time Scheduling for Job
2440 Execution has further details.
2441 deferral_time = <ClassAd Integer Expression>
2444 Allows a job to specify the time at which its execution is to
2445 begin, instead of beginning execution as soon as it arrives
2446 at the execution machine. The deferral time is an expression
2447 that evaluates to a Unix Epoch timestamp (the number of sec‐
2448 onds elapsed since 00:00:00 on January 1, 1970, Coordinated
2449 Universal Time). Deferral time is evaluated with respect to
2450 the execution machine. This option delays the start of execu‐
2451 tion, but not the matching and claiming of a machine for the
2452 job. If the job is not available and ready to begin execution
2453 at the deferral time, it has missed its deferral time. A job
2454 that misses its deferral time will be put on hold in the
2455 queue.
2456 The HTCondor User's manual section on Time Scheduling for Job
2458 Execution has further details.
2459 Due to implementation details, a deferral time may not be
2461 used for scheduler universe jobs.
2462 deferral_window = <ClassAd Integer Expression>
2465 The deferral window is used in conjunction with the defer‐
2466 ral_time command to allow jobs that miss their deferral time
2467 to begin execution.
2468 The HTCondor User's manual section on Time Scheduling for Job
2470 Execution has further details.
2471 description = <string>
2474 A string that sets the value of the job ClassAd attribute
2475 JobDescription. When set, tools which display the executable
2476 such as condor_q will instead use this string.
2477 email_attributes = <list-of-job-ad-attributes>
2480 A comma-separated list of attributes from the job ClassAd.
2481 These attributes and their values will be included in the
2482 e-mail notification of job completion.
2483 image_size = <size>
2486 Advice to HTCondor specifying the maximum virtual image size
2487 to which the job will grow during its execution. HTCondor
2488 will then execute the job only on machines which have enough
2489 resources, (such as virtual memory), to support executing the
2490 job. If not specified, HTCondor will automatically make a
2491 (reasonably accurate) estimate about the job's size and ad‐
2492 just this estimate as the program runs. If specified and un‐
2493 derestimated, the job may crash due to the inability to ac‐
2494 quire more address space; for example, if malloc() fails. If
2495 the image size is overestimated, HTCondor may have difficulty
2496 finding machines which have the required resources. size is
2497 specified in KiB. For example, for an image size of 8 MiB,
2498 size should be 8000.
2499 initialdir = <directory-path>
2501 Used to give jobs a directory with respect to file input and
2502 output. Also provides a directory (on the machine from which
2503 the job is submitted) for the job event log, when a full path
2504 is not specified.
2505 For vanilla universe jobs where there is a shared file sys‐
2507 tem, it is the current working directory on the machine where
2508 the job is executed.
2509 For vanilla or grid universe jobs where file transfer mecha‐
2511 nisms are utilized (there is not a shared file system), it is
2512 the directory on the machine from which the job is submitted
2513 where the input files come from, and where the job's output
2514 files go to.
2515 For standard universe jobs, it is the directory on the ma‐
2517 chine from which the job is submitted where the condor_shadow
2518 daemon runs; the current working directory for file input and
2519 output accomplished through remote system calls.
2520 For scheduler universe jobs, it is the directory on the ma‐
2522 chine from which the job is submitted where the job runs; the
2523 current working directory for file input and output with re‐
2524 spect to relative path names.
2525 Note that the path to the executable is not relative to ini‐
2527 tialdir ; if it is a relative path, it is relative to the
2528 directory in which the condor_submit command is run.
2529 job_ad_information_attrs = <attribute-list>
2532 A comma-separated list of job ClassAd attribute names. The
2533 named attributes and their values are written to the job
2534 event log whenever any event is being written to the log.
2535 This implements the same thing as the configuration variable
2536 EVENT_LOG_INFORMATION_ATTRS (see the Daemon Logging Configu‐
2537 ration File Entries page), but it applies to the job event
2538 log, instead of the system event log.
2539 JobBatchName = <batch_name>
2542 Set the batch name for this submit. The batch name is dis‐
2543 played by condor_q -batch. It is intended for use by users to
2544 give meaningful names to their jobs and to influence how con‐
2545 dor_q groups jobs for display. This value in a submit file
2546 can be overridden by specifying the -batch-name argument on
2547 the condor_submit command line.
2548 job_lease_duration = <number-of-seconds>
2551 For vanilla, parallel, VM, and java universe jobs only, the
2552 duration in seconds of a job lease. The default value is
2553 2,400, or forty minutes. If a job lease is not desired, the
2554 value can be explicitly set to 0 to disable the job lease se‐
2555 mantics. The value can also be a ClassAd expression that
2556 evaluates to an integer. The HTCondor User's manual section
2557 on Special Environment Considerations has further details.
2558 job_machine_attrs = <attr1, attr2, ...>
2561 A comma and/or space separated list of machine attribute
2562 names that should be recorded in the job ClassAd in addition
2563 to the ones specified by the condor_schedd daemon's system
2564 configuration variable SYSTEM_JOB_MACHINE_ATTRS
2565 . When there are multiple run attempts, history of machine
2566 attributes from previous run attempts may be kept. The number
2567 of run attempts to store may be extended beyond the sys‐
2568 tem-specified history length by using the submit file command
2569 job_machine_attrs_history_length . A machine attribute
2570 named X will be inserted into the job ClassAd as an attribute
2571 named MachineAttrX0. The previous value of this attribute
2572 will be named MachineAttrX1, the previous to that will be
2573 named MachineAttrX2, and so on, up to the specified history
2574 length. A history of length 1 means that only MachineAttrX0
2575 will be recorded. The value recorded in the job ClassAd is
2576 the evaluation of the machine attribute in the context of the
2577 job ClassAd when the condor_schedd daemon initiates the start
2578 up of the job. If the evaluation results in an Undefined or
2579 Error result, the value recorded in the job ad will be Unde‐
2580 fined or Error, respectively.
2581 want_graceful_removal = <boolean expression>
2584 If true, this job will be given a chance to shut down cleanly
2585 when removed. The job will be given as much time as the ad‐
2586 ministrator of the execute resource allows, which may be
2587 none. The default is false. For details, see the configura‐
2588 tion setting GRACEFULLY_REMOVE_JOBS.
2589 kill_sig = <signal-number>
2592 When HTCondor needs to kick a job off of a machine, it will
2593 send the job the signal specified by signal-number . sig‐
2594 nal-number needs to be an integer which represents a valid
2595 signal on the execution machine. For jobs submitted to the
2596 standard universe, the default value is the number for SIGT‐
2597 STP which tells the HTCondor libraries to initiate a check‐
2598 point of the process. For jobs submitted to other universes,
2599 the default value, when not defined, is SIGTERM, which is the
2600 standard way to terminate a program in Unix.
2601 kill_sig_timeout = <seconds>
2603 This submit command should no longer be used as of HTCondor
2604 version 7.7.3; use job_max_vacate_time instead. If
2605 job_max_vacate_time is not defined, this defines the number
2606 of seconds that HTCondor should wait following the sending of
2607 the kill signal defined by kill_sig and forcibly killing
2608 the job. The actual amount of time between sending the signal
2609 and forcibly killing the job is the smallest of this value
2610 and the configuration variable KILLING_TIMEOUT
2611 , as defined on the execute machine.
2612 load_profile = <True | False>
2615 When True, loads the account profile of the dedicated run ac‐
2616 count for Windows jobs. May not be used with run_as_owner .
2617 match_list_length = <integer value>
2620 Defaults to the value zero (0). When match_list_length is de‐
2621 fined with an integer value greater than zero (0), attributes
2622 are inserted into the job ClassAd. The maximum number of at‐
2623 tributes defined is given by the integer value. The job Clas‐
2624 sAds introduced are given as
2625 LastMatchName0 = "most-recent-Name"
2627 LastMatchName1 = "next-most-recent-Name"
2628 The value for each introduced ClassAd is given by the value
2630 of the Name attribute from the machine ClassAd of a previous
2631 execution (match). As a job is matched, the definitions for
2632 these attributes will roll, with LastMatchName1 becoming
2633 LastMatchName2, LastMatchName0 becoming LastMatchName1, and
2634 LastMatchName0 being set by the most recent value of the Name
2635 attribute.
2636 An intended use of these job attributes is in the require‐
2638 ments expression. The requirements can allow a job to prefer
2639 a match with either the same or a different resource than a
2640 previous match.
2641 job_max_vacate_time = <integer expression>
2645 An integer-valued expression (in seconds) that may be used to
2646 adjust the time given to an evicted job for gracefully shut‐
2647 ting down. If the job's setting is less than the machine's,
2648 the job's is used. If the job's setting is larger than the
2649 machine's, the result depends on whether the job has any ex‐
2650 cess retirement time. If the job has more retirement time
2651 left than the machine's max vacate time setting, then retire‐
2652 ment time will be converted into vacating time, up to the
2653 amount requested by the job.
2654 Setting this expression does not affect the job's resource
2656 requirements or preferences. For a job to only run on a ma‐
2657 chine with a minimum MachineMaxVacateTime, or to preferen‐
2658 tially run on such machines, explicitly specify this in the
2659 requirements and/or rank expressions.
2660 max_job_retirement_time = <integer expression>
2663 An integer-valued expression (in seconds) that does nothing
2664 unless the machine that runs the job has been configured to
2665 provide retirement time. Retirement time is a grace period
2666 given to a job to finish when a resource claim is about to be
2667 preempted. The default behavior in many cases is to take as
2668 much retirement time as the machine offers, so this command
2669 will rarely appear in a submit description file.
2670 When a resource claim is to be preempted, this expression in
2672 the submit file specifies the maximum run time of the job (in
2673 seconds, since the job started). This expression has no ef‐
2674 fect, if it is greater than the maximum retirement time pro‐
2675 vided by the machine policy. If the resource claim is not
2676 preempted, this expression and the machine retirement policy
2677 are irrelevant. If the resource claim is preempted the job
2678 will be allowed to run until the retirement time expires, at
2679 which point it is hard-killed. The job will be soft-killed
2680 when it is getting close to the end of retirement in order to
2681 give it time to gracefully shut down. The amount of lead-time
2682 for soft-killing is determined by the maximum vacating time
2683 granted to the job.
2684 Standard universe jobs and any jobs running with nice_user
2686 priority have a default max_job_retirement_time of 0, so no
2687 retirement time is utilized by default. In all other cases,
2688 no default value is provided, so the maximum amount of re‐
2689 tirement time is utilized by default.
2690 Setting this expression does not affect the job's resource
2692 requirements or preferences. For a job to only run on a ma‐
2693 chine with a minimum MaxJobRetirementTime, or to preferen‐
2694 tially run on such machines, explicitly specify this in the
2695 requirements and/or rank expressions.
2696 nice_user = <True | False>
2698 Normally, when a machine becomes available to HTCondor, HT‐
2699 Condor decides which job to run based upon user and job pri‐
2700 orities. Setting nice_user equal to True tells HTCondor not
2701 to use your regular user priority, but that this job should
2702 have last priority among all users and all jobs. So jobs sub‐
2703 mitted in this fashion run only on machines which no other
2704 non-nice_user job wants - a true bottom-feeder job! This is
2705 very handy if a user has some jobs they wish to run, but do
2706 not wish to use resources that could instead be used to run
2707 other people's HTCondor jobs. Jobs submitted in this fashion
2708 have "nice-user." prepended to the owner name when viewed
2709 from condor_q or condor_userprio. The default value is False.
2710 noop_job = <ClassAd Boolean Expression>
2712 When this boolean expression is True, the job is immediately
2713 removed from the queue, and HTCondor makes no attempt at run‐
2714 ning the job. The log file for the job will show a job sub‐
2715 mitted event and a job terminated event, along with an exit
2716 code of 0, unless the user specifies a different signal or
2717 exit code.
2718 noop_job_exit_code = <return value>
2721 When noop_job is in the submit description file and evalu‐
2722 ates to True, this command allows the job to specify the re‐
2723 turn value as shown in the job's log file job terminated
2724 event. If not specified, the job will show as having termi‐
2725 nated with status 0. This overrides any value specified with
2726 noop_job_exit_signal .
2727 noop_job_exit_signal = <signal number>
2730 When noop_job is in the submit description file and evalu‐
2731 ates to True, this command allows the job to specify the sig‐
2732 nal number that the job's log event will show the job having
2733 terminated with.
2734 remote_initialdir = <directory-path>
2737 The path specifies the directory in which the job is to be
2738 executed on the remote machine. This is currently supported
2739 in all universes except for the standard universe.
2740 rendezvousdir = <directory-path>
2743 Used to specify the shared file system directory to be used
2744 for file system authentication when submitting to a remote
2745 scheduler. Should be a path to a preexisting directory.
2746 run_as_owner = <True | False>
2749 A boolean value that causes the job to be run under the login
2750 of the submitter, if supported by the joint configuration of
2751 the submit and execute machines. On Unix platforms, this de‐
2752 faults to True, and on Windows platforms, it defaults to
2753 False. May not be used with load_profile . See the HTCondor
2754 manual Platform-Specific Information chapter for administra‐
2755 tive details on configuring Windows to support this option.
2756 stack_size = <size in bytes>
2758 This command applies only to Linux platform jobs that are not
2759 standard universe jobs. An integer number of bytes, repre‐
2760 senting the amount of stack space to be allocated for the
2761 job. This value replaces the default allocation of stack
2762 space, which is unlimited in size.
2763 submit_event_notes = <note>
2765 A string that is appended to the submit event in the job's
2766 log file. For DAGMan jobs, the string DAG Node: and the
2767 node's name is automatically defined for submit_event_notes,
2768 causing the logged submit event to identify the DAG node job
2769 submitted.
2770 +<attribute> = <value>
2772 A line that begins with a '+' (plus) character instructs con‐
2773 dor_submit to insert the given attribute into the job ClassAd
2774 with the given value. Note that setting an attribute should
2775 not be used in place of one of the specific commands listed
2776 above. Often, the command name does not directly correspond
2777 to an attribute name; furthermore, many submit commands re‐
2778 sult in actions more complex than simply setting an attribute
2779 or attributes. See Job ClassAd Attributes for a list of HT‐
2780 Condor job attributes.
2781 MACROS AND COMMENTS
2783 In addition to commands, the submit description file can contain macros
2786 and comments.
2787 Macros Parameterless macros in the form of $(macro_name:default ini‐
2789 tial value) may be used anywhere in HTCondor submit descrip‐
2790 tion files to provide textual substitution at submit time.
2791 Macros can be defined by lines in the form of
2792 <macro_name> = <string>
2794 Two pre-defined macros are supplied by the submit description
2796 file parser. The $(Cluster) or $(ClusterId) macro supplies
2797 the value of the
2798 ClusterId job ClassAd attribute, and the $(Process) or
2800 $(ProcId) macro supplies the value of the ProcId job ClassAd
2801 attribute. These macros are intended to aid in the specifica‐
2802 tion of input/output files, arguments, etc., for clusters
2803 with lots of jobs, and/or could be used to supply an HTCondor
2804 process with its own cluster and process numbers on the com‐
2805 mand line.
2806 The $(Node) macro is defined for parallel universe jobs, and
2808 is especially relevant for MPI applications. It is a unique
2809 value assigned for the duration of the job that essentially
2810 identifies the machine (slot) on which a program is execut‐
2811 ing. Values assigned start at 0 and increase monotonically.
2812 The values are assigned as the parallel job is about to
2813 start.
2814 Recursive definition of macros is permitted. An example of a
2816 construction that works is the following:
2817 foo = bar
2819 foo = snap $(foo)
2820 As a result, foo = snap bar.
2822 Note that both left- and right- recursion works, so
2824 foo = bar
2826 foo = $(foo) snap
2827 has as its result foo = bar snap.
2829 The construction
2831 foo = $(foo) bar
2833 by itself will not work, as it does not have an initial base
2835 case. Mutually recursive constructions such as:
2836 B = bar
2838 C = $(B)
2839 B = $(C) boo
2840 will not work, and will fill memory with expansions.
2842 A default value may be specified, for use if the macro has no
2844 definition. Consider the example
2845 D = $(E:24)
2847 Where E is not defined within the submit description file,
2849 the default value 24 is used, resulting in
2850 D = 24
2852 This is of limited value, as the scope of macro substitution
2854 is the submit description file. Thus, either the macro is or
2855 is not defined within the submit description file. If the
2856 macro is defined, then the default value is useless. If the
2857 macro is not defined, then there is no point in using it in a
2858 submit command.
2859 To use the dollar sign character ($) as a literal, without
2862 macro expansion, use
2863 $(DOLLAR)
2865 In addition to the normal macro, there is also a special kind
2867 of macro called a substitution macro
2868 that allows the substitution of a machine ClassAd attribute
2869 value defined on the resource machine itself (gotten after a
2870 match to the machine has been made) into specific commands
2871 within the submit description file. The substitution macro is
2872 of the form:
2873 $$(attribute)
2875 As this form of the substitution macro is only evaluated
2877 within the context of the machine ClassAd, use of a scope
2878 resolution prefix TARGET. or MY. is not allowed.
2879 A common use of this form of the substitution macro is for
2881 the heterogeneous submission of an executable:
2882 executable = povray.$$(OpSys).$$(Arch)
2884 Values for the OpSys and Arch attributes are substituted at
2886 match time for any given resource. This example allows HTCon‐
2887 dor to automatically choose the correct executable for the
2888 matched machine.
2889 An extension to the syntax of the substitution macro provides
2891 an alternative string to use if the machine attribute within
2892 the substitution macro is undefined. The syntax appears as:
2893 $$(attribute:string_if_attribute_undefined)
2895 An example using this extended syntax provides a path name to
2897 a required input file. Since the file can be placed in dif‐
2898 ferent locations on different machines, the file's path name
2899 is given as an argument to the program.
2900 arguments = $$(input_file_path:/usr/foo)
2902 On the machine, if the attribute input_file_path is not de‐
2904 fined, then the path /usr/foo is used instead.
2905 A further extension to the syntax of the substitution macro
2907 allows the evaluation of a ClassAd expression to define the
2908 value. In this form, the expression may refer to machine at‐
2909 tributes by prefacing them with the TARGET. scope resolution
2910 prefix. To place a ClassAd expression into the substitution
2911 macro, square brackets are added to delimit the expression.
2912 The syntax appears as:
2913 $$([ClassAd expression])
2915 An example of a job that uses this syntax may be one that
2917 wants to know how much memory it can use. The application
2918 cannot detect this itself, as it would potentially use all of
2919 the memory on a multi-slot machine. So the job determines the
2920 memory per slot, reducing it by 10% to account for miscella‐
2921 neous overhead, and passes this as a command line argument to
2922 the application. In the submit description file will be
2923 arguments = --memory $$([TARGET.Memory * 0.9])
2925 To insert two dollar sign characters ($$) as literals into a
2929 ClassAd string, use
2930 $$(DOLLARDOLLAR)
2932 The environment macro, $ENV, allows the evaluation of an en‐
2938 vironment variable to be used in setting a submit description
2939 file command. The syntax used is
2940 $ENV(variable)
2942 An example submit description file command that uses this
2944 functionality evaluates the submitter's home directory in or‐
2945 der to set the path and file name of a log file:
2946 log = $ENV(HOME)/jobs/logfile
2948 The environment variable is evaluated when the submit de‐
2950 scription file is processed.
2951 The $RANDOM_CHOICE macro allows a random choice to be made
2956 from a given list of parameters at submission time. For an
2957 expression, if some randomness needs to be generated, the
2958 macro may appear as
2959 $RANDOM_CHOICE(0,1,2,3,4,5,6)
2961 When evaluated, one of the parameters values will be chosen.
2963 Comments
2965 Blank lines and lines beginning with a pound sign ('#') char‐
2966 acter are ignored by the submit description file parser.
2967
2969 While processing the queue command in a submit file or from the command
2970 line, condor_submit will set the values of several automatic submit
2971 variables so that they can be referred to by statements in the submit
2972 file. With the exception of Cluster and Process, if these variables are
2973 set by the submit file, they will not be modified during queue pro‐
2974 cessing.
2975 ClusterId
2977 Set to the integer value that the ClusterId attribute that
2978 the job ClassAd will have when the job is submitted. All jobs
2979 in a single submit will normally have the same value for the
2980 ClusterId. If the -dry-run argument is specified, The value
2981 will be 1.
2982 Cluster
2984 Alternate name for the ClusterId submit variable. Before HT‐
2985 Condor version 8.4 this was the only name.
2986 ProcId Set to the integer value that the ProcId attribute of the job
2988 ClassAd will have when the job is submitted. The value will
2989 start at 0 and increment by 1 for each job submitted.
2990 Process
2992 Alternate name for the ProcId submit variable. Before HTCon‐
2993 dor version 8.4 this was the only name.
2994 Node For parallel universes, set to the value #pArAlLeLnOdE# or
2996 #MpInOdE# depending on the parallel universe type For other
2997 universes it is set to nothing.
2998 Step Set to the step value as it varies from 0 to N-1 where N is
3000 the number provided on the queue argument. This variable
3001 changes at the same rate as ProcId when it changes at all.
3002 For submit files that don't make use of the queue number op‐
3003 tion, Step will always be 0. For submit files that don't make
3004 use of any of the foreach options, Step and ProcId will al‐
3005 ways be the same.
3006 ItemIndex
3008 Set to the index within the item list being processed by the
3009 various queue foreach options. For submit files that don't
3010 make use of any queue foreach list, ItemIndex will always be
3011 0 For submit files that make use of a slice to select only
3012 some items in a foreach list, ItemIndex will only be set to
3013 selected values.
3014 Row Alternate name for ItemIndex.
3016 Item when a queue foreach option is used and no variable list is
3018 supplied, this variable will be set to the value of the cur‐
3019 rent item.
3020 The automatic variables below are set before parsing the submit file,
3022 and will not vary during processing unless the submit file itself sets
3023 them.
3024 ARCH Set to the CPU architecture of the machine running con‐
3026 dor_submit. The value will be the same as the automatic con‐
3027 figuration variable of the same name.
3028 OPSYS Set to the name of the operating system on the machine run‐
3030 ning condor_submit. The value will be the same as the auto‐
3031 matic configuration variable of the same name.
3032 OPSYSANDVER
3034 Set to the name and major version of the operating system on
3035 the machine running condor_submit. The value will be the same
3036 as the automatic configuration variable of the same name.
3037 OPSYSMAJORVER
3039 Set to the major version of the operating system on the ma‐
3040 chine running condor_submit. The value will be the same as
3041 the automatic configuration variable of the same name.
3042 OPSYSVER
3044 Set to the version of the operating system on the machine
3045 running condor_submit. The value will be the same as the au‐
3046 tomatic configuration variable of the same name.
3047 SPOOL Set to the full path of the HTCondor spool directory. The
3049 value will be the same as the automatic configuration vari‐
3050 able of the same name.
3051 IsLinux
3053 Set to true if the operating system of the machine running
3054 condor_submit is a Linux variant. Set to false otherwise.
3055 IsWindows
3057 Set to true if the operating system of the machine running
3058 condor_submit is a Microsoft Windows variant. Set to false
3059 otherwise.
3060 SUBMIT_FILE
3062 Set to the full pathname of the submit file being processed
3063 by condor_submit. If submit statements are read from standard
3064 input, it is set to nothing.
3065
3067 condor_submit will exit with a status value of 0 (zero) upon success,
3068 and a non-zero value upon failure.
3069
3071 • Submit Description File Example 1: This example queues three jobs for
3072 execution by HTCondor. The first will be given command line arguments
3073 of 15 and 2000, and it will write its standard output to foo.out1.
3074 The second will be given command line arguments of 30 and 2000, and
3075 it will write its standard output to foo.out2. Similarly the third
3076 will have arguments of 45 and 6000, and it will use foo.out3 for its
3077 standard output. Standard error output (if any) from all three pro‐
3078 grams will appear in foo.error.
3079 ####################
3081 #
3082 # submit description file
3083 # Example 1: queuing multiple jobs with differing
3084 # command line arguments and output files.
3085 #
3086 ####################
3087 Executable = foo
3089 Universe = vanilla
3090 Arguments = 15 2000
3092 Output = foo.out0
3093 Error = foo.err0
3094 Queue
3095 Arguments = 30 2000
3097 Output = foo.out1
3098 Error = foo.err1
3099 Queue
3100 Arguments = 45 6000
3102 Output = foo.out2
3103 Error = foo.err2
3104 Queue
3105 Or you can get the same results as the above submit file by using a
3107 list of arguments with the Queue statement
3108 ####################
3110 #
3111 # submit description file
3112 # Example 1b: queuing multiple jobs with differing
3113 # command line arguments and output files, alternate syntax
3114 #
3115 ####################
3116 Executable = foo
3118 Universe = vanilla
3119 # generate different output and error filenames for each process
3121 Output = foo.out$(Process)
3122 Error = foo.err$(Process)
3123 Queue Arguments From (
3125 15 2000
3126 30 2000
3127 45 6000
3128 )
3129 • Submit Description File Example 2: This submit description file exam‐
3131 ple queues 150 runs of program foo which must have been compiled and
3132 linked for an Intel x86 processor running RHEL 3. HTCondor will not
3133 attempt to run the processes on machines which have less than 32
3134 Megabytes of physical memory, and it will run them on machines which
3135 have at least 64 Megabytes, if such machines are available. Stdin,
3136 stdout, and stderr will refer to in.0, out.0, and err.0 for the first
3137 run of this program (process 0). Stdin, stdout, and stderr will refer
3138 to in.1, out.1, and err.1 for process 1, and so forth. A log file
3139 containing entries about where and when HTCondor runs, takes check‐
3140 points, and migrates processes in this cluster will be written into
3141 file foo.log.
3142 ####################
3144 #
3145 # Example 2: Show off some fancy features including
3146 # use of pre-defined macros and logging.
3147 #
3148 ####################
3149 Executable = foo
3151 Universe = standard
3152 Requirements = OpSys == "LINUX" && Arch =="INTEL"
3153 Rank = Memory >= 64
3154 Request_Memory = 32 Mb
3155 Image_Size = 28 Mb
3156 Error = err.$(Process)
3158 Input = in.$(Process)
3159 Output = out.$(Process)
3160 Log = foo.log
3161 Queue 150
3162 • Submit Description File Example 3: This example targets the
3164 /bin/sleep program to run only on a platform running a RHEL 6 operat‐
3165 ing system. The example presumes that the pool contains machines run‐
3166 ning more than one version of Linux, and this job needs the particu‐
3167 lar operating system to run correctly.
3168 ####################
3170 #
3171 # Example 3: Run on a RedHat 6 machine
3172 #
3173 ####################
3174 Universe = vanilla
3175 Executable = /bin/sleep
3176 Arguments = 30
3177 Requirements = (OpSysAndVer == "RedHat6")
3178 Error = err.$(Process)
3180 Input = in.$(Process)
3181 Output = out.$(Process)
3182 Log = sleep.log
3183 Queue
3184 • Command Line example: The following command uses the -append option
3186 to add two commands before the job(s) is queued. A log file and an
3187 error log file are specified. The submit description file is un‐
3188 changed.
3189 condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
3191 Note that each of the added commands is contained within quote marks
3193 because there are space characters within the command.
3194 • periodic_remove example: A job should be removed from the queue, if
3196 the total suspension time of the job is more than half of the run
3197 time of the job.
3198 Including the command
3200 periodic_remove = CumulativeSuspensionTime >
3202 ((RemoteWallClockTime - CumulativeSuspensionTime) / 2.0)
3203 in the submit description file causes this to happen.
3205
3207 • For security reasons, HTCondor will refuse to run any jobs submitted
3208 by user root (UID = 0) or by a user whose default group is group
3209 wheel (GID = 0). Jobs submitted by user root or a user with a default
3210 group of wheel will appear to sit forever in the queue in an idle
3211 state.
3212 • All path names specified in the submit description file must be less
3214 than 256 characters in length, and command line arguments must be
3215 less than 4096 characters in length; otherwise, condor_submit gives a
3216 warning message but the jobs will not execute properly.
3217 • Somewhat understandably, behavior gets bizarre if the user makes the
3219 mistake of requesting multiple HTCondor jobs to write to the same
3220 file, and/or if the user alters any files that need to be accessed by
3221 an HTCondor job which is still in the queue. For example, the com‐
3222 pressing of data or output files before an HTCondor job has completed
3223 is a common mistake.
3224 • To disable checkpointing for Standard Universe jobs, include the
3226 line:
3227 +WantCheckpoint = False
3229 in the submit description file before the queue command(s).
3231
3233 HTCondor User Manual
3234
3236 HTCondor Team
3237
3239 1990-2022, Center for High Throughput Computing, Computer Sciences De‐
3240 partment, University of Wisconsin-Madison, Madison, WI, US. Licensed
3241 under the Apache License, Version 2.0.
32428.8 Jun 13, 2022 CONDOR_SUBMIT(1)