1just-man-pages/condor_submit(G1e)neral Commands Manjuuaslt-man-pages/condor_submit(1)
2
6 condor_submit Queue jobs for execution under HTCondor
7
9 condor_submit [ -terse ] [ -verbose ] [ -unused ] [ -name schedd_name ]
10 [ -remote schedd_name ] [ -addr <ip:port> ] [ -pool pool_name ] [ -dis‐
11 able ] [ -password passphrase ] [ -debug ] [ -append command ... ] [
12 -batch-name batch_name ] [ -spool ] [ -dump filename ] [ -interactive ]
13 [ -dry-run ] [ -maxjobs number-of-jobs ] [ -single-cluster ] [ -stm
14 method ] [ <submit-variable>=<value> ] [ submit description file ] [
15 -queue queue_arguments ]
16
18 condor_submit is the program for submitting jobs for execution under
19 HTCondor. condor_submit requires a submit description file which con‐
20 tains commands to direct the queuing of jobs. One submit description
21 file may contain specifications for the queuing of many HTCondor jobs
22 at once. A single invocation of condor_submit may cause one or more
23 clusters. A cluster is a set of jobs specified in the submit descrip‐
24 tion file between queue commands for which the executable is not
25 changed. It is advantageous to submit multiple jobs as a single cluster
26 because:
27 * Only one copy of the checkpoint file is needed to represent all
29 jobs in a cluster until they begin execution.
30 * There is much less overhead involved for HTCondor to start the
32 next job in a cluster than for HTCondor to start a new cluster. This
33 can make a big difference when submitting lots of short jobs.
34 Multiple clusters may be specified within a single submit description
36 file. Each cluster must specify a single executable.
37 The job ClassAd attribute ClusterId identifies a cluster.
39 The submit description file argument is the path and file name of the
41 submit description file. If this optional argument is missing or is the
42 dash character ( - ), then the commands are taken from standard input.
43 If - is specified for the submit description file , -verbose is
44 implied; this can be overridden by specifying -terse .
45 Note that submission of jobs from a Windows machine requires a stashed
47 password to allow HTCondor to impersonate the user submitting the job.
48 To stash a password, use the condor_store_cred command. See the manual
49 page for details.
50 For lengthy lines within the submit description file, the backslash ()
52 is a line continuation character. Placing the backslash at the end of a
53 line causes the current line's command to be continued with the next
54 line of the file. Submit description files may contain comments. A com‐
55 ment is any line beginning with a pound character ( # ).
56
58 -terse
59 Terse output - display JobId ranges only.
61 -verbose
67 Verbose output - display the created job ClassAd
69 -unused
75 As a default, causes no warnings to be issued about user-defined
77 macros not being used within the submit description file. The mean‐
78 ing reverses (toggles) when the configuration variable
79 WARN_ON_UNUSED_SUBMIT_FILE_MACROS is set to the non default value of
80 False . Printing the warnings can help identify spelling errors of
81 submit description file commands. The warnings are sent to stderr.
82 -name schedd_name
88 Submit to the specified condor_schedd . Use this option to submit to
90 a condor_schedd other than the default local one. schedd_name is
91 the value of the Name ClassAd attribute on the machine where the
92 condor_schedd daemon runs.
93 -remote schedd_name
99 Submit to the specified condor_schedd , spooling all required input
101 files over the network connection. schedd_name is the value of the
102 Name ClassAd attribute on the machine where the condor_schedd daemon
103 runs. This option is equivalent to using both -name and -spool .
104 -addr <ip:port>
110 Submit to the condor_schedd at the IP address and port given by the
112 sinful string argument <ip:port> .
113 -pool pool_name
119 Look in the specified pool for the condor_schedd to submit to. This
121 option is used with -name or -remote .
122 -disable
128 Disable file permission checks when submitting a job for read per‐
130 missions on all input files, such as those defined by commands input
131 and transfer_input_files , as well as write permission to output
132 files, such as a log file defined by log and output files defined
133 with output or transfer_output_files .
134 -password passphrase
140 Specify a password to the MyProxy server.
142 -debug
148 Cause debugging information to be sent to stderr , based on the
150 value of the configuration variable TOOL_DEBUG .
151 -append command
157 Augment the commands in the submit description file with the given
159 command . This command will be considered to immediately precede the
160 queue command within the submit description file, and come after all
161 other previous commands. If the command specifies a queue command,
162 as in the example
163 condor_submit mysubmitfile -append "queue input in A, B, C"
166 then the entire -append command line option and its arguments are
168 converted to
169 condor_submit mysubmitfile -queue input in A, B, C
172 The submit description file is not modified. Multiple commands are
174 specified by using the -append option multiple times. Each new com‐
175 mand is given in a separate -append option. Commands with spaces in
176 them will need to be enclosed in double quote marks.
177 -batch-name batch_name
183 Set the batch name for this submit. The batch name is displayed by
185 condor_q -batch . It is intended for use by users to give meaningful
186 names to their jobs and to influence how condor_q groups jobs for
187 display. Use of this argument takes precedence over a batch name
188 specified in the submit description file itself.
189 -spool
195 Spool all required input files, job event log, and proxy over the
197 connection to the condor_schedd . After submission, modify local
198 copies of the files without affecting your jobs. Any output files
199 for completed jobs need to be retrieved with condor_transfer_data .
200 -dump filename
206 Sends all ClassAds to the specified file, instead of to the con‐
208 dor_schedd .
209 -interactive
215 Indicates that the user wants to run an interactive shell on an exe‐
217 cute machine in the pool. This is equivalent to creating a submit
218 description file of a vanilla universe sleep job, and then running
219 condor_ssh_to_job by hand. Without any additional arguments, con‐
220 dor_submit with the -interactive flag creates a dummy vanilla uni‐
221 verse job that sleeps, submits it to the local scheduler, waits for
222 the job to run, and then launches condor_ssh_to_job to run a shell.
223 If the user would like to run the shell on a machine that matches a
224 particular requirements expression, the submit description file is
225 specified, and it will contain the expression. Note that all policy
226 expressions specified in the submit description file are honored,
227 but any executable or universe commands are overwritten to be sleep
228 and vanilla. The job ClassAd attribute InteractiveJob is set to
229 True to identify interactive jobs for condor_startd policy usage.
230 -dry-run file
236 Parse the submit description file, sending the resulting job ClassAd
238 to the file given by file , but do not submit the job(s). This per‐
239 mits observation of the job specification, and it facilitates debug‐
240 ging the submit description file contents. If file is - , the output
241 is written to stdout .
242 -maxjobs number-of-jobs
248 If the total number of jobs specified by the submit description file
250 is more than the integer value given by number-of-jobs , then no
251 jobs are submitted for execution and an error message is generated.
252 A 0 or negative value for the number-of-jobs causes no limit to be
253 imposed.
254 -single-cluster
260 If the jobs specified by the submit description file causes more
262 than a single cluster value to be assigned, then no jobs are submit‐
263 ted for execution and an error message is generated.
264 -stm method
270 Specify the method use to move a sandbox into HTCondor. method is
272 one of stm_use_schedd_only or stm_use_transferd .
273 <submit-variable>=<value>
279 Defines a submit command or submit variable with a value, and parses
281 it as if it was placed at the beginning of the submit description
282 file. The submit description file is not changed. To correctly parse
283 the condor_submit command line, this option must be specified with‐
284 out white space characters before and after the equals sign ( = ),
285 or the entire option must be surrounded by double quote marks.
286 -queue queue_arguments
292 A command line specification of how many jobs to queue, which is
294 only permitted if the submit description file does not have a queue
295 command. The queue_arguments are the same as may be within a submit
296 description file. The parsing of the queue_arguments finishes at the
297 end of the line or when a dash character ( - ) is encountered.
298 Therefore, its best placement within the command line will be at the
299 end of the command line.
300 On a Unix command line, the shell expands file globs before parsing
302 occurs.
303
309 Note: more information on submitting HTCondor jobs can be found here: .
310 As of version 8.5.6, the condor_submit language supports multi-line
312 values in commands. The syntax is the same as the configuration lan‐
313 guage (see more details here: ).
314 Each submit description file describes one or more clusters of jobs to
316 be placed in the HTCondor execution pool. All jobs in a cluster must
317 share the same executable, but they may have different input and output
318 files, and different program arguments. The submit description file is
319 generally the last command-line argument to condor_submit . If the sub‐
320 mit description file argument is omitted, condor_submit will read the
321 submit description from standard input.
322 The submit description file must contain at least one executable com‐
324 mand and at least one queue command. All of the other commands have
325 default actions.
326 Note that a submit file that contains more than one executable command
328 will produce multiple clusters when submitted. This is not generally
329 recommended, and is not allowed for submit files that are run as DAG
330 node jobs by condor_dagman .
331 The commands which can appear in the submit description file are numer‐
333 ous. They are listed here in alphabetical order by category.
334 BASIC COMMANDS
336 arguments = <argument_list>
344 List of arguments to be supplied to the executable as part of the
346 command line.
347 In the java universe, the first argument must be the name of the
349 class containing main .
350 There are two permissible formats for specifying arguments, identi‐
352 fied as the old syntax and the new syntax. The old syntax supports
353 white space characters within arguments only in special circum‐
354 stances; when used, the command line arguments are represented in
355 the job ClassAd attribute Args . The new syntax supports uniform
356 quoting of white space characters within arguments; when used, the
357 command line arguments are represented in the job ClassAd attribute
358 Arguments .
359 Old Syntax
361 In the old syntax, individual command line arguments are delimited
363 (separated) by space characters. To allow a double quote mark in an
364 argument, it is escaped with a backslash; that is, the two character
365 sequence \" becomes a single double quote mark within an argument.
366 Further interpretation of the argument string differs depending on
368 the operating system. On Windows, the entire argument string is
369 passed verbatim (other than the backslash in front of double quote
370 marks) to the Windows application. Most Windows applications will
371 allow spaces within an argument value by surrounding the argument
372 with double quotes marks. In all other cases, there is no further
373 interpretation of the arguments.
374 Example:
376 arguments = one \"two\" 'three'
380 Produces in Unix vanilla universe:
382 argument 1: one
386 argument 2: "two"
387 argument 3: 'three'
388 New Syntax
390 Here are the rules for using the new syntax:
392 1. The entire string representing the command line arguments is
396 surrounded by double quote marks. This permits the white space
397 characters of spaces and tabs to potentially be embedded within a
398 single argument. Putting the double quote mark within the argu‐
399 ments is accomplished by escaping it with another double quote
400 mark.
401 2. The white space characters of spaces or tabs delimit argu‐
405 ments.
406 3. To embed white space characters of spaces or tabs within a
410 single argument, surround the entire argument with single quote
411 marks.
412 4. To insert a literal single quote mark, escape it within an
416 argument already delimited by single quote marks by adding
417 another single quote mark.
418 Example:
422 arguments = "3 simple arguments"
424 Produces:
426 argument 1: 3
428 argument 2: simple
429 argument 3: arguments
430 Another example:
432 arguments = "one 'two with spaces' 3"
434 Produces:
436 argument 1: one
438 argument 2: two with spaces
439 argument 3: 3
440 And yet another example:
442 arguments = "one ""two"" 'spacey ''quoted'' argument'"
444 Produces:
446 argument 1: one
448 argument 2: "two"
449 argument 3: spacey 'quoted' argument
450 Notice that in the new syntax, the backslash has no special meaning.
452 This is for the convenience of Windows users.
453 environment = <parameter_list>
459 List of environment variables.
461 There are two different formats for specifying the environment vari‐
463 ables: the old format and the new format. The old format is retained
464 for backward-compatibility. It suffers from a platform-dependent
465 syntax and the inability to insert some special characters into the
466 environment.
467 The new syntax for specifying environment values:
469 1. Put double quote marks around the entire argument string. This
473 distinguishes the new syntax from the old. The old syntax does
474 not have double quote marks around it. Any literal double quote
475 marks within the string must be escaped by repeating the double
476 quote mark.
477 2. Each environment entry has the form
481 <name>=<value>
485 3. Use white space (space or tab characters) to separate environ‐
489 ment entries.
490 4. To put any white space in an environment entry, surround the
494 space and as much of the surrounding entry as desired with single
495 quote marks.
496 5. To insert a literal single quote mark, repeat the single quote
500 mark anywhere inside of a section surrounded by single quote
501 marks.
502 Example:
506 environment = "one=1 two=""2"" three='spacey ''quoted'' value'"
510 Produces the following environment entries:
512 one=1
516 two="2"
517 three=spacey 'quoted' value
518 Under the old syntax, there are no double quote marks surrounding
520 the environment specification. Each environment entry remains of the
521 form
522 <name>=<value>
524 Under Unix, list multiple environment entries by separating them
526 with a semicolon (;). Under Windows, separate multiple entries with
527 a vertical bar ( | ). There is no way to insert a literal semicolon
528 under Unix or a literal vertical bar under Windows. Note that spaces
529 are accepted, but rarely desired, characters within parameter names
530 and values, because they are treated as literal characters, not sep‐
531 arators or ignored white space. Place spaces within the parameter
532 list only if required.
533 A Unix example:
535 environment = one=1;two=2;three="quotes have no 'special' meaning"
539 This produces the following:
541 one=1
545 two=2
546 three="quotes have no 'special' meaning"
547 If the environment is set with the environment command and getenv is
549 also set to true, values specified with environment override values
550 in the submitter's environment (regardless of the order of the envi‐
551 ronment and getenv commands).
552 error = <pathname>
558 A path and file name used by HTCondor to capture any error messages
560 the program would normally write to the screen (that is, this file
561 becomes stderr ). A path is given with respect to the file system
562 of the machine on which the job is submitted. The file is written
563 (by the job) in the remote scratch directory of the machine where
564 the job is executed. When the job exits, the resulting file is
565 transferred back to the machine where the job was submitted, and the
566 path is utilized for file placement. If not specified, the default
567 value of /dev/null is used for submission to a Unix machine. If not
568 specified, error messages are ignored for submission to a Windows
569 machine. More than one job should not use the same error file, since
570 this will cause one job to overwrite the errors of another. The
571 error file and the output file should not be the same file as the
572 outputs will overwrite each other or be lost. For grid universe
573 jobs, error may be a URL that the Globus tool globus_url_copy under‐
574 stands.
575 executable = <pathname>
581 An optional path and a required file name of the executable file for
583 this job cluster. Only one executable command within a submit
584 description file is guaranteed to work properly. More than one often
585 works.
586 If no path or a relative path is used, then the executable file is
588 presumed to be relative to the current working directory of the user
589 as the condor_submit command is issued.
590 If submitting into the standard universe, then the named executable
592 must have been re-linked with the HTCondor libraries (such as via
593 the condor_compile command). If submitting into the vanilla universe
594 (the default), then the named executable need not be re-linked and
595 can be any process which can run in the background (shell scripts
596 work fine as well). If submitting into the Java universe, then the
597 argument must be a compiled .class file.
598 getenv = <True | False>
604 If getenv is set to True , then condor_submit will copy all of the
606 user's current shell environment variables at the time of job sub‐
607 mission into the job ClassAd. The job will therefore execute with
608 the same set of environment variables that the user had at submit
609 time. Defaults to False .
610 If the environment is set with the environment command and getenv is
612 also set to true, values specified with environment override values
613 in the submitter's environment (regardless of the order of the envi‐
614 ronment and getenv commands).
615 input = <pathname>
621 HTCondor assumes that its jobs are long-running, and that the user
623 will not wait at the terminal for their completion. Because of this,
624 the standard files which normally access the terminal, ( stdin ,
625 stdout , and stderr ), must refer to files. Thus, the file name
626 specified with input should contain any keyboard input the program
627 requires (that is, this file becomes stdin ). A path is given with
628 respect to the file system of the machine on which the job is sub‐
629 mitted. The file is transferred before execution to the remote
630 scratch directory of the machine where the job is executed. If not
631 specified, the default value of /dev/null is used for submission to
632 a Unix machine. If not specified, input is ignored for submission to
633 a Windows machine. For grid universe jobs, input may be a URL that
634 the Globus tool globus_url_copy understands.
635 Note that this command does not refer to the command-line arguments
637 of the program. The command-line arguments are specified by the
638 arguments command.
639 log = <pathname>
645 Use log to specify a file name where HTCondor will write a log file
647 of what is happening with this job cluster, called a job event log.
648 For example, HTCondor will place a log entry into this file when and
649 where the job begins running, when the job produces a checkpoint, or
650 moves (migrates) to another machine, and when the job completes.
651 Most users find specifying a log file to be handy; its use is recom‐
652 mended. If no log entry is specified, HTCondor does not create a log
653 for this cluster. If a relative path is specified, it is relative to
654 the current working directory as the job is submitted or the direc‐
655 tory specified by submit command initialdir on the submit machine.
656 log_xml = <True | False>
662 If log_xml is True , then the job event log file will be written in
664 ClassAd XML. If not specified, XML is not used. Note that the file
665 is an XML fragment; it is missing the file header and footer. Do not
666 mix XML and non-XML within a single file. If multiple jobs write to
667 a single job event log file, ensure that all of the jobs specify
668 this option in the same way.
669 notification = <Always | Complete | Error | Never>
675 Owners of HTCondor jobs are notified by e-mail when certain events
677 occur. If defined by Always , the owner will be notified whenever
678 the job produces a checkpoint, as well as when the job completes. If
679 defined by Complete , the owner will be notified when the job termi‐
680 nates. If defined by Error , the owner will only be notified if the
681 job terminates abnormally, or if the job is placed on hold because
682 of a failure, and not by user request. If defined by Never (the
683 default), the owner will not receive e-mail, regardless to what hap‐
684 pens to the job. The HTCondor User's manual documents statistics
685 included in the e-mail.
686 notify_user = <email-address>
692 Used to specify the e-mail address to use when HTCondor sends e-mail
694 about a job. If not specified, HTCondor defaults to using the e-mail
695 address defined by
696 job-owner@UID_DOMAIN
698 where the configuration variable UID_DOMAIN is specified by the
700 HTCondor site administrator. If UID_DOMAIN has not been specified,
701 HTCondor sends the e-mail to:
702 job-owner@submit-machine-name
704 output = <pathname>
710 The output file captures any information the program would ordinar‐
712 ily write to the screen (that is, this file becomes stdout ). A
713 path is given with respect to the file system of the machine on
714 which the job is submitted. The file is written (by the job) in the
715 remote scratch directory of the machine where the job is executed.
716 When the job exits, the resulting file is transferred back to the
717 machine where the job was submitted, and the path is utilized for
718 file placement. If not specified, the default value of /dev/null is
719 used for submission to a Unix machine. If not specified, output is
720 ignored for submission to a Windows machine. Multiple jobs should
721 not use the same output file, since this will cause one job to over‐
722 write the output of another. The output file and the error file
723 should not be the same file as the outputs will overwrite each other
724 or be lost. For grid universe jobs, output may be a URL that the
725 Globus tool globus_url_copy understands.
726 Note that if a program explicitly opens and writes to a file, that
728 file should not be specified as the output file.
729 priority = <integer>
735 An HTCondor job priority can be any integer, with 0 being the
737 default. Jobs with higher numerical priority will run before jobs
738 with lower numerical priority. Note that this priority is on a per
739 user basis. One user with many jobs may use this command to order
740 his/her own jobs, and this will have no effect on whether or not
741 these jobs will run ahead of another user's jobs.
742 Note that the priority setting in an HTCondor submit file will be
744 overridden by condor_dagman if the submit file is used for a node in
745 a DAG, and the priority of the node within the DAG is non-zero (see
746 for more details).
747 queue [ <int expr> ]
753 Places zero or more copies of the job into the HTCondor queue.
755 queue
759 [ <int expr> ] [ <varname> ] in [ slice ] <list of items> Places
761 zero or more copies of the job in the queue based on items in a
762 <list of items>
763 queue
767 [ <int expr> ] [ <varname> ] matching [ files | dirs ] [ slice ]
769 <list of items with file globbing> ] Places zero or more copies of
770 the job in the queue based on files that match a <list of items with
771 file globbing>
772 queue
776 [ <int expr> ] [ <list of varnames> ] from [ slice ] <file name> |
778 <list of items> ] Places zero or more copies of the job in the queue
779 based on lines from the submit file or from <file name>
780 The optional argument <int expr> specifies how many times to repeat
782 the job submission for a given set of arguments. It may be an inte‐
783 ger or an expression that evaluates to an integer, and it defaults
784 to 1. All but the first form of this command are various ways of
785 specifying a list of items. When these forms are used <int expr>
786 jobs will be queued for each item in the list. The in , matching and
787 from keyword indicates how the list will be specified.
788 * in The list of items is an explicit comma and/or space sepa‐
790 rated <list of items> . If the <list of items> begins with an
791 open paren, and the close paren is not on the same line as the
792 open, then the list continues until a line that begins with a
793 close paren is read from the submit file.
794 * matching Each item in the <list of items with file globbing>
796 will be matched against the names of files and directories rela‐
797 tive to the current directory, the set of matching names is the
798 resulting list of items.
799 * files Only filenames will matched.
801 * dirs Only directory names will be matched.
803 * from <file name> | <list of items> Each line from <file name>
805 or <list of items> is a single item, this allows for multiple
806 variables to be set for each item. Lines from <file name> or
807 <list of items> will be split on comma and/or space until there
808 are values for each of the variables specified in <list of var‐
809 names> . The last variable will contain the remainder of the
810 line. When the <list of items> form is used, the list continues
811 until the first line that begins with a close paren, and lines
812 beginning with pound sign ('#') will be skipped. When using the
813 <file name> form, if the <file name> ends with |, then it will be
814 executed as a script whatever the script writes to stdout will
815 be the list of items. The optional argument <varname> or <list of
816 varnames> is the name or names of of variables that will be set
817 to the value of the current item when queuing the job. If no
818 <varname> is specified the variable ITEM will be used. Leading
819 and trailing whitespace be trimmed. The optional argument <slice>
820 is a python style slice selecting only some of the items in the
821 list of items. Negative step values are not supported.
822 A submit file may contain more than one queue statement, and if
824 desired, any commands may be placed between subsequent queue com‐
825 mands, such as new input , output , error , initialdir , or argu‐
826 ments commands. This is handy when submitting multiple runs into one
827 cluster with one submit description file.
828 universe = <vanilla | standard | scheduler | local | grid | java | vm |
834 parallel | docker>
835 Specifies which HTCondor universe to use when running this job. The
837 HTCondor universe specifies an HTCondor execution environment.
838 The vanilla universe is the default (except where the configuration
840 variable DEFAULT_UNIVERSE defines it otherwise), and is an execu‐
841 tion environment for jobs which do not use HTCondor's mechanisms for
842 taking checkpoints; these are ones that have not been linked with
843 the HTCondor libraries. Use the vanilla universe to submit shell
844 scripts to HTCondor.
845 The standard universe tells HTCondor that this job has been re-
847 linked via condor_compile with the HTCondor libraries and therefore
848 supports taking checkpoints and remote system calls.
849 The scheduler universe is for a job that is to run on the machine
851 where the job is submitted. This universe is intended for a job that
852 acts as a metascheduler and will not be preempted.
853 The local universe is for a job that is to run on the machine where
855 the job is submitted. This universe runs the job immediately and
856 will not preempt the job.
857 The grid universe forwards the job to an external job management
859 system. Further specification of the grid universe is done with the
860 grid_resource command.
861 The java universe is for programs written to the Java Virtual
863 Machine.
864 The vm universe facilitates the execution of a virtual machine.
866 The parallel universe is for parallel jobs (e.g. MPI) that require
868 multiple machines in order to run.
869 The docker universe runs a docker container as an HTCondor job.
871 COMMANDS FOR MATCHMAKING
877 rank = <ClassAd Float Expression>
885 A ClassAd Floating-Point expression that states how to rank machines
887 which have already met the requirements expression. Essentially,
888 rank expresses preference. A higher numeric value equals better
889 rank. HTCondor will give the job the machine with the highest rank.
890 For example,
891 request_memory = max({60, Target.TotalSlotMemory})
893 rank = Memory
894 asks HTCondor to find all available machines with more than 60
896 megabytes of memory and give to the job the machine with the most
897 amount of memory. The HTCondor User's Manual contains complete
898 information on the syntax and available attributes that can be used
899 in the ClassAd expression.
900 request_cpus = <num-cpus>
906 A requested number of CPUs (cores). If not specified, the number
908 requested will be 1. If specified, the expression
909 && (RequestCpus <= Target.Cpus)
911 is appended to the requirements expression for the job.
913 For pools that enable dynamic condor_startd provisioning, specifies
915 the minimum number of CPUs requested for this job, resulting in a
916 dynamic slot being created with this many cores.
917 request_disk = <quantity>
923 The requested amount of disk space in KiB requested for this job. If
925 not specified, it will be set to the job ClassAd attribute
926 DiskUsage . The expression
927 && (RequestDisk <= Target.Disk)
929 is appended to the requirements expression for the job.
931 For pools that enable dynamic condor_startd provisioning, a dynamic
933 slot will be created with at least this much disk space.
934 Characters may be appended to a numerical value to indicate units.
936 K or KB indicates KiB, numbers of bytes. M or MB indicates MiB,
937 numbers of bytes. G or GB indicates GiB, numbers of bytes. T or
938 TB indicates TiB, numbers of bytes.
939 request_memory = <quantity>
945 The required amount of memory in MiB that this job needs to avoid
947 excessive swapping. If not specified and the submit command vm_mem‐
948 ory is specified, then the value specified for vm_memory defines
949 request_memory . If neither request_memory nor vm_memory is speci‐
950 fied, the value is set by the configuration variable
951 JOB_DEFAULT_REQUESTMEMORY . The actual amount of memory used by a
952 job is represented by the job ClassAd attribute MemoryUsage .
953 For pools that enable dynamic condor_startd provisioning, a dynamic
955 slot will be created with at least this much RAM.
956 The expression
958 && (RequestMemory <= Target.Memory)
960 is appended to the requirements expression for the job.
962 Characters may be appended to a numerical value to indicate units.
964 K or KB indicates KiB, numbers of bytes. M or MB indicates MiB,
965 numbers of bytes. G or GB indicates GiB, numbers of bytes. T or
966 TB indicates TiB, numbers of bytes.
967 request_<name> = <quantity>
973 The required amount of the custom machine resource identified by
975 <name> that this job needs. The custom machine resource is defined
976 in the machine's configuration. Machines that have available GPUs
977 will define <name> to be GPUs .
978 requirements = <ClassAd Boolean Expression>
984 The requirements command is a boolean ClassAd expression which uses
986 C-like operators. In order for any job in this cluster to run on a
987 given machine, this requirements expression must evaluate to true on
988 the given machine.
989 For scheduler and local universe jobs, the requirements expression
991 is evaluated against the Scheduler ClassAd which represents the the
992 condor_schedd daemon running on the submit machine, rather than a
993 remote machine. Like all commands in the submit description file, if
994 multiple requirements commands are present, all but the last one are
995 ignored. By default, condor_submit appends the following clauses to
996 the requirements expression:
997 1. Arch and OpSys are set equal to the Arch and OpSys of the sub‐
999 mit machine. In other words: unless you request otherwise, HTCon‐
1000 dor will give your job machines with the same architecture and
1001 operating system version as the machine running condor_submit .
1002 2. Cpus >=RequestCpus, if the job ClassAd attribute RequestCpus
1004 is defined.
1005 3. Disk >=RequestDisk, if the job ClassAd attribute RequestDisk
1007 is defined. Otherwise, Disk >=DiskUsage is appended to the
1008 requirements. The DiskUsage attribute is initialized to the size
1009 of the executable plus the size of any files specified in a
1010 transfer_input_files command. It exists to ensure there is enough
1011 disk space on the target machine for HTCondor to copy over both
1012 the executable and needed input files. The DiskUsage attribute
1013 represents the maximum amount of total disk space required by the
1014 job in kilobytes. HTCondor automatically updates the DiskUsage
1015 attribute approximately every 20 minutes while the job runs with
1016 the amount of space being used by the job on the execute machine.
1017 4. Memory >=RequestMemory, if the job ClassAd attribute Request‐
1019 Memory is defined.
1020 5. If Universe is set to Vanilla, FileSystemDomain is set equal
1022 to the submit machine's FileSystemDomain. View the requirements
1023 of a job which has already been submitted (along with everything
1024 else about the job ClassAd) with the command condor_q -l ; see
1025 the command reference for condor_q on page . Also, see the HTCon‐
1026 dor Users Manual for complete information on the syntax and
1027 available attributes that can be used in the ClassAd expression.
1028 FILE TRANSFER COMMANDS
1034 dont_encrypt_input_files = < file1,file2,file... >
1042 A comma and/or space separated list of input files that are not to
1044 be network encrypted when transferred with the file transfer mecha‐
1045 nism. Specification of files in this manner overrides configuration
1046 that would use encryption. Each input file must also be in the list
1047 given by transfer_input_files . When a path to an input file or
1048 directory is specified, this specifies the path to the file on the
1049 submit side. A single wild card character ( * ) may be used in each
1050 file name.
1051 dont_encrypt_output_files = < file1,file2,file... >
1057 A comma and/or space separated list of output files that are not to
1059 be network encrypted when transferred back with the file transfer
1060 mechanism. Specification of files in this manner overrides configu‐
1061 ration that would use encryption. The output file(s) must also
1062 either be in the list given by transfer_output_files or be discov‐
1063 ered and to be transferred back with the file transfer mechanism.
1064 When a path to an output file or directory is specified, this speci‐
1065 fies the path to the file on the execute side. A single wild card
1066 character ( * ) may be used in each file name.
1067 encrypt_execute_directory = <True | False>
1073 Defaults to False . If set to True , HTCondor will encrypt the
1075 contents of the remote scratch directory of the machine where the
1076 job is executed. This encryption is transparent to the job itself,
1077 but ensures that files left behind on the local disk of the execute
1078 machine, perhaps due to a system crash, will remain private. In
1079 addition, condor_submit will append to the job's requirements
1080 expression
1081 && (TARGET.HasEncryptExecuteDirectory)
1083 to ensure the job is matched to a machine that is capable of
1085 encrypting the contents of the execute directory. This support is
1086 limited to Windows platforms that use the NTFS file system and Linux
1087 platforms with the ecryptfs-utils package installed.
1088 encrypt_input_files = < file1,file2,file... >
1094 A comma and/or space separated list of input files that are to be
1096 network encrypted when transferred with the file transfer mechanism.
1097 Specification of files in this manner overrides configuration that
1098 would not use encryption. Each input file must also be in the list
1099 given by transfer_input_files . When a path to an input file or
1100 directory is specified, this specifies the path to the file on the
1101 submit side. A single wild card character ( * ) may be used in each
1102 file name. The method of encryption utilized will be as agreed upon
1103 in security negotiation; if that negotiation failed, then the file
1104 transfer mechanism must also fail for files to be network encrypted.
1105 encrypt_output_files = < file1,file2,file... >
1111 A comma and/or space separated list of output files that are to be
1113 network encrypted when transferred back with the file transfer mech‐
1114 anism. Specification of files in this manner overrides configuration
1115 that would not use encryption. The output file(s) must also either
1116 be in the list given by transfer_output_files or be discovered and
1117 to be transferred back with the file transfer mechanism. When a path
1118 to an output file or directory is specified, this specifies the path
1119 to the file on the execute side. A single wild card character ( * )
1120 may be used in each file name. The method of encryption utilized
1121 will be as agreed upon in security negotiation; if that negotiation
1122 failed, then the file transfer mechanism must also fail for files to
1123 be network encrypted.
1124 max_transfer_input_mb = <ClassAd Integer Expression>
1130 This integer expression specifies the maximum allowed total size in
1132 MiB of the input files that are transferred for a job. This expres‐
1133 sion does not apply to grid universe, standard universe, or files
1134 transferred via file transfer plug-ins. The expression may refer to
1135 attributes of the job. The special value -1 indicates no limit. If
1136 not defined, the value set by configuration variable MAX_TRANS‐
1137 FER_INPUT_MB is used. If the observed size of all input files at
1138 submit time is larger than the limit, the job will be immediately
1139 placed on hold with a HoldReasonCode value of 32. If the job passes
1140 this initial test, but the size of the input files increases or the
1141 limit decreases so that the limit is violated, the job will be
1142 placed on hold at the time when the file transfer is attempted.
1143 max_transfer_output_mb = <ClassAd Integer Expression>
1149 This integer expression specifies the maximum allowed total size in
1151 MiB of the output files that are transferred for a job. This expres‐
1152 sion does not apply to grid universe, standard universe, or files
1153 transferred via file transfer plug-ins. The expression may refer to
1154 attributes of the job. The special value -1 indicates no limit. If
1155 not set, the value set by configuration variable MAX_TRANSFER_OUT‐
1156 PUT_MB is used. If the total size of the job's output files to be
1157 transferred is larger than the limit, the job will be placed on hold
1158 with a HoldReasonCode value of 33. The output will be transferred
1159 up to the point when the limit is hit, so some files may be fully
1160 transferred, some partially, and some not at all.
1161 output_destination = <destination-URL>
1167 When present, defines a URL that specifies both a plug-in and a des‐
1169 tination for the transfer of the entire output sandbox or a subset
1170 of output files as specified by the submit command transfer_out‐
1171 put_files . The plug-in does the transfer of files, and no files are
1172 sent back to the submit machine. The HTCondor Administrator's manual
1173 has full details.
1174 should_transfer_files = <YES | NO | IF_NEEDED >
1180 The should_transfer_files setting is used to define if HTCondor
1182 should transfer files to and from the remote machine where the job
1183 runs. The file transfer mechanism is used to run jobs which are not
1184 in the standard universe (and can therefore use remote system calls
1185 for file access) on machines which do not have a shared file system
1186 with the submit machine. should_transfer_files equal to YES will
1187 cause HTCondor to always transfer files for the job. NO disables
1188 HTCondor's file transfer mechanism. IF_NEEDED will not transfer
1189 files for the job if it is matched with a resource in the same
1190 FileSystemDomain as the submit machine (and therefore, on a machine
1191 with the same shared file system). If the job is matched with a
1192 remote resource in a different FileSystemDomain , HTCondor will
1193 transfer the necessary files.
1194 For more information about this and other settings related to trans‐
1196 ferring files, see the HTCondor User's manual section on the file
1197 transfer mechanism.
1198 Note that should_transfer_files is not supported for jobs submitted
1200 to the grid universe.
1201 skip_filechecks = <True | False>
1207 When True , file permission checks for the submitted job are dis‐
1209 abled. When False , file permissions are checked; this is the
1210 behavior when this command is not present in the submit description
1211 file. File permissions are checked for read permissions on all input
1212 files, such as those defined by commands input and trans‐
1213 fer_input_files , and for write permission to output files, such as
1214 a log file defined by log and output files defined with output or
1215 transfer_output_files .
1216 stream_error = <True | False>
1222 If True , then stderr is streamed back to the machine from which
1224 the job was submitted. If False , stderr is stored locally and
1225 transferred back when the job completes. This command is ignored if
1226 the job ClassAd attribute TransferErr is False . The default value
1227 is False . This command must be used in conjunction with error ,
1228 otherwise stderr will sent to /dev/null on Unix machines and
1229 ignored on Windows machines.
1230 stream_input = <True | False>
1236 If True , then stdin is streamed from the machine on which the job
1238 was submitted. The default value is False . The command is only
1239 relevant for jobs submitted to the vanilla or java universes, and it
1240 is ignored by the grid universe. This command must be used in con‐
1241 junction with input , otherwise stdin will be /dev/null on Unix
1242 machines and ignored on Windows machines.
1243 stream_output = <True | False>
1249 If True , then stdout is streamed back to the machine from which
1251 the job was submitted. If False , stdout is stored locally and
1252 transferred back when the job completes. This command is ignored if
1253 the job ClassAd attribute TransferOut is False . The default value
1254 is False . This command must be used in conjunction with output ,
1255 otherwise stdout will sent to /dev/null on Unix machines and
1256 ignored on Windows machines.
1257 transfer_executable = <True | False>
1263 This command is applicable to jobs submitted to the grid and vanilla
1265 universes. If transfer_executable is set to False , then HTCondor
1266 looks for the executable on the remote machine, and does not trans‐
1267 fer the executable over. This is useful for an already pre-staged
1268 executable; HTCondor behaves more like rsh. The default value is
1269 True .
1270 transfer_input_files = < file1,file2,file... >
1276 A comma-delimited list of all the files and directories to be trans‐
1278 ferred into the working directory for the job, before the job is
1279 started. By default, the file specified in the executable command
1280 and any file specified in the input command (for example, stdin )
1281 are transferred.
1282 When a path to an input file or directory is specified, this speci‐
1284 fies the path to the file on the submit side. The file is placed in
1285 the job's temporary scratch directory on the execute side, and it is
1286 named using the base name of the original path. For example,
1287 /path/to/input_file becomes input_file in the job's scratch direc‐
1288 tory.
1289 A directory may be specified by appending the forward slash charac‐
1291 ter (/) as a trailing path separator. This syntax is used for both
1292 Windows and Linux submit hosts. A directory example using a trailing
1293 path separator is input_data/ . When a directory is specified with
1294 the trailing path separator, the contents of the directory are
1295 transferred, but the directory itself is not transferred. It is as
1296 if each of the items within the directory were listed in the trans‐
1297 fer list. When there is no trailing path separator, the directory is
1298 transferred, its contents are transferred, and these contents are
1299 placed inside the transferred directory.
1300 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1302 tories is not currently supported.
1303 Symbolic links to files are transferred as the files they point to.
1305 Transfer of symbolic links to directories is not currently sup‐
1306 ported.
1307 For vanilla and vm universe jobs only, a file may be specified by
1309 giving a URL, instead of a file name. The implementation for URL
1310 transfers requires both configuration and available plug-in.
1311 transfer_output_files = < file1,file2,file... >
1317 This command forms an explicit list of output files and directories
1319 to be transferred back from the temporary working directory on the
1320 execute machine to the submit machine. If there are multiple files,
1321 they must be delimited with commas. Setting transfer_output_files to
1322 the empty string ( "" ) means that no files are to be transferred.
1323 For HTCondor-C jobs and all other non-grid universe jobs, if trans‐
1325 fer_output_files is not specified, HTCondor will automatically
1326 transfer back all files in the job's temporary working directory
1327 which have been modified or created by the job. Subdirectories are
1328 not scanned for output, so if output from subdirectories is desired,
1329 the output list must be explicitly specified. For grid universe jobs
1330 other than HTCondor-C, desired output files must also be explicitly
1331 listed. Another reason to explicitly list output files is for a job
1332 that creates many files, and the user wants only a subset trans‐
1333 ferred back.
1334 For grid universe jobs other than with grid type condor , to have
1336 files other than standard output and standard error transferred from
1337 the execute machine back to the submit machine, do use transfer_out‐
1338 put_files , listing all files to be transferred. These files are
1339 found on the execute machine in the working directory of the job.
1340 When a path to an output file or directory is specified, it speci‐
1342 fies the path to the file on the execute side. As a destination on
1343 the submit side, the file is placed in the job's initial working
1344 directory, and it is named using the base name of the original path.
1345 For example, path/to/output_file becomes output_file in the job's
1346 initial working directory. The name and path of the file that is
1347 written on the submit side may be modified by using transfer_out‐
1348 put_remaps . Note that this remap function only works with files but
1349 not with directories.
1350 A directory may be specified using a trailing path separator. An
1352 example of a trailing path separator is the slash character on Unix
1353 platforms; a directory example using a trailing path separator is
1354 input_data/ . When a directory is specified with a trailing path
1355 separator, the contents of the directory are transferred, but the
1356 directory itself is not transferred. It is as if each of the items
1357 within the directory were listed in the transfer list. When there is
1358 no trailing path separator, the directory is transferred, its con‐
1359 tents are transferred, and these contents are placed inside the
1360 transferred directory.
1361 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1363 tories is not currently supported.
1364 Symbolic links to files are transferred as the files they point to.
1366 Transfer of symbolic links to directories is not currently sup‐
1367 ported.
1368 transfer_output_remaps < `` name newname ; name2 newname2 ... ''>
1374 This specifies the name (and optionally path) to use when download‐
1376 ing output files from the completed job. Normally, output files are
1377 transferred back to the initial working directory with the same name
1378 they had in the execution directory. This gives you the option to
1379 save them with a different path or name. If you specify a relative
1380 path, the final path will be relative to the job's initial working
1381 directory.
1382 name describes an output file name produced by your job, and newname
1384 describes the file name it should be downloaded to. Multiple remaps
1385 can be specified by separating each with a semicolon. If you wish to
1386 remap file names that contain equals signs or semicolons, these spe‐
1387 cial characters may be escaped with a backslash. You cannot specify
1388 directories to be remapped.
1389 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT >
1395 Setting when_to_transfer_output equal to ON_EXIT will cause HTCondor
1399 to transfer the job's output files back to the submitting machine
1400 only when the job completes (exits on its own).
1401 The ON_EXIT_OR_EVICT option is intended for fault tolerant jobs
1403 which periodically save their own state and can restart where they
1404 left off. In this case, files are spooled to the submit machine any
1405 time the job leaves a remote site, either because it exited on its
1406 own, or was evicted by the HTCondor system for any reason prior to
1407 job completion. The files spooled back are placed in a directory
1408 defined by the value of the SPOOL configuration variable. Any out‐
1409 put files transferred back to the submit machine are automatically
1410 sent back out again as input files if the job restarts.
1411 POLICY COMMANDS
1417 max_retries = <integer>
1425 The maximum number of retries allowed for this job (must be non-neg‐
1427 ative). If the job fails (does not exit with the success_exit_code
1428 exit code) it will be retried up to max_retries times (unless
1429 retries are ceased because of the retry_until command). If
1430 max_retries is not defined, and either retry_until or suc‐
1431 cess_exit_code is, the value of DEFAULT_JOB_MAX_RETRIES will be
1432 used for the maximum number of retries.
1433 The combination of the max_retries , retry_until , and suc‐
1435 cess_exit_code commands causes an appropriate OnExitRemove expres‐
1436 sion to be automatically generated. If retry command(s) and
1437 on_exit_remove are both defined, the OnExitRemove expression will
1438 be generated by OR'ing the expression specified in OnExitRemove and
1439 the expression generated by the retry commands.
1440 retry_until <Integer | ClassAd Boolean Expression>
1446 An integer value or boolean expression that prevents further retries
1448 from taking place, even if max_retries have not been exhausted. If
1449 retry_until is an integer, the job exiting with that exit code will
1450 cause retries to cease. If retry_until is a ClassAd expression, the
1451 expression evaluating to True will cause retries to cease.
1452 success_exit_code = <integer>
1458 The exit code that is considered successful for this job. Defaults
1460 to 0 if not defined.
1461 Note: non-zero values of success_exit_code should generally not be
1463 used for DAG node jobs. At the present time, condor_dagman does not
1464 take into account the value of success_exit_code . This means that,
1465 if success_exit_code is set to a non-zero value, condor_dagman will
1466 consider the job failed when it actually succeeds. For single-proc
1467 DAG node jobs, this can be overcome by using a POST script that
1468 takes into account the value of success_exit_code (although this is
1469 not recommended). For multi-proc DAG node jobs, there is currently
1470 no way to overcome this limitation.
1471 hold = <True | False>
1477 If hold is set to True , then the submitted job will be placed into
1479 the Hold state. Jobs in the Hold state will not run until released
1480 by condor_release . Defaults to False .
1481 keep_claim_idle = <integer>
1487 An integer number of seconds that a job requests the condor_schedd
1489 to wait before releasing its claim after the job exits or after the
1490 job is removed.
1491 The process by which the condor_schedd claims a condor_startd is
1493 somewhat time-consuming. To amortize this cost, the condor_schedd
1494 tries to reuse claims to run subsequent jobs, after a job using a
1495 claim is done. However, it can only do this if there is an idle job
1496 in the queue at the moment the previous job completes. Sometimes,
1497 and especially for the node jobs when using DAGMan, there is a sub‐
1498 sequent job about to be submitted, but it has not yet arrived in the
1499 queue when the previous job completes. As a result, the con‐
1500 dor_schedd releases the claim, and the next job must wait an entire
1501 negotiation cycle to start. When this submit command is defined with
1502 a non-negative integer, when the job exits, the condor_schedd tries
1503 as usual to reuse the claim. If it cannot, instead of releasing the
1504 claim, the condor_schedd keeps the claim until either the number of
1505 seconds given as a parameter, or a new job which matches that claim
1506 arrives, whichever comes first. The condor_startd in question will
1507 remain in the Claimed/Idle state, and the original job will be
1508 "charged" (in terms of priority) for the time in this state.
1509 leave_in_queue = <ClassAd Boolean Expression>
1515 When the ClassAd Expression evaluates to True , the job is not
1517 removed from the queue upon completion. This allows the user of a
1518 remotely spooled job to retrieve output files in cases where HTCon‐
1519 dor would have removed them as part of the cleanup associated with
1520 completion. The job will only exit the queue once it has been marked
1521 for removal (via condor_rm , for example) and the leave_in_queue
1522 expression has become False . leave_in_queue defaults to False .
1523 As an example, if the job is to be removed once the output is
1525 retrieved with condor_transfer_data , then use
1526 leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED)
1528 ||\
1529 (StageOutFinish == 0))
1530 next_job_start_delay = <ClassAd Boolean Expression>
1536 This expression specifies the number of seconds to delay after
1538 starting up this job before the next job is started. The maximum
1539 allowed delay is specified by the HTCondor configuration variable
1540 MAX_NEXT_JOB_START_DELAY , which defaults to 10 minutes. This com‐
1541 mand does not apply to scheduler or local universe jobs.
1542 This command has been historically used to implement a form of job
1544 start throttling from the job submitter's perspective. It was effec‐
1545 tive for the case of multiple job submission where the transfer of
1546 extremely large input data sets to the execute machine caused
1547 machine performance to suffer. This command is no longer useful, as
1548 throttling should be accomplished through configuration of the con‐
1549 dor_schedd daemon.
1550 on_exit_hold = <ClassAd Boolean Expression>
1556 The ClassAd expression is checked when the job exits, and if True ,
1558 places the job into the Hold state. If False (the default value
1559 when not defined), then nothing happens and the on_exit_remove
1560 expression is checked to determine if that needs to be applied.
1561 For example: Suppose a job is known to run for a minimum of an hour.
1563 If the job exits after less than an hour, the job should be placed
1564 on hold and an e-mail notification sent, instead of being allowed to
1565 leave the queue.
1566 on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1570 This expression places the job on hold if it exits for any reason
1572 before running for an hour. An e-mail will be sent to the user
1573 explaining that the job was placed on hold because this expression
1574 became True .
1575 periodic_* expressions take precedence over on_exit_* expressions,
1578 and *_hold expressions take precedence over a *_remove expres‐
1579 sions.
1580 Only job ClassAd attributes will be defined for use by this ClassAd
1582 expression. This expression is available for the vanilla, java, par‐
1583 allel, grid, local and scheduler universes. It is additionally
1584 available, when submitted from a Unix machine, for the standard uni‐
1585 verse.
1586 on_exit_hold_reason = <ClassAd String Expression>
1592 When the job is placed on hold due to the on_exit_hold expression
1594 becoming True , this expression is evaluated to set the value of
1595 HoldReason in the job ClassAd. If this expression is UNDEFINED or
1596 produces an empty or invalid string, a default description is used.
1597 on_exit_hold_subcode = <ClassAd Integer Expression>
1603 When the job is placed on hold due to the on_exit_hold expression
1605 becoming True , this expression is evaluated to set the value of
1606 HoldReasonSubCode in the job ClassAd. The default subcode is 0. The
1607 HoldReasonCode will be set to 3, which indicates that the job went
1608 on hold due to a job policy expression.
1609 on_exit_remove = <ClassAd Boolean Expression>
1615 The ClassAd expression is checked when the job exits, and if True
1617 (the default value when undefined), then it allows the job to leave
1618 the queue normally. If False , then the job is placed back into the
1619 Idle state. If the user job runs under the vanilla universe, then
1620 the job restarts from the beginning. If the user job runs under the
1621 standard universe, then it continues from where it left off, using
1622 the last checkpoint.
1623 For example, suppose a job occasionally segfaults, but chances are
1625 that the job will finish successfully if the job is run again with
1626 the same data. The on_exit_remove expression can cause the job to
1627 run again with the following command. Assume that the signal identi‐
1628 fier for the segmentation fault is 11 on the platform where the job
1629 will be running.
1630 on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)
1632 This expression lets the job leave the queue if the job was not
1634 killed by a signal or if it was killed by a signal other than 11,
1635 representing segmentation fault in this example. So, if the exited
1636 due to signal 11, it will stay in the job queue. In any other case
1637 of the job exiting, the job will leave the queue as it normally
1638 would have done.
1639 As another example, if the job should only leave the queue if it
1641 exited on its own with status 0, this on_exit_remove expression
1642 works well:
1643 on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
1647 If the job was killed by a signal or exited with a non-zero exit
1649 status, HTCondor would leave the job in the queue to run again.
1650 periodic_* expressions take precedence over on_exit_* expressions,
1653 and *_hold expressions take precedence over a *_remove expres‐
1654 sions.
1655 Only job ClassAd attributes will be defined for use by this ClassAd
1657 expression.
1658 periodic_hold = <ClassAd Boolean Expression>
1664 This expression is checked periodically when the job is not in the
1666 Held state. If it becomes True , the job will be placed on hold. If
1667 unspecified, the default value is False .
1668 periodic_* expressions take precedence over on_exit_* expressions,
1671 and *_hold expressions take precedence over a *_remove expres‐
1672 sions.
1673 Only job ClassAd attributes will be defined for use by this ClassAd
1675 expression. Note that, by default, this expression is only checked
1676 once every 60 seconds. The period of these evaluations can be
1677 adjusted by setting the PERIODIC_EXPR_INTERVAL , MAX_PERI‐
1678 ODIC_EXPR_INTERVAL , and PERIODIC_EXPR_TIMESLICE configuration
1679 macros.
1680 periodic_hold_reason = <ClassAd String Expression>
1686 When the job is placed on hold due to the periodic_hold expression
1688 becoming True , this expression is evaluated to set the value of
1689 HoldReason in the job ClassAd. If this expression is UNDEFINED or
1690 produces an empty or invalid string, a default description is used.
1691 periodic_hold_subcode = <ClassAd Integer Expression>
1697 When the job is placed on hold due to the periodic_hold expression
1699 becoming true, this expression is evaluated to set the value of
1700 HoldReasonSubCode in the job ClassAd. The default subcode is 0. The
1701 HoldReasonCode will be set to 3, which indicates that the job went
1702 on hold due to a job policy expression.
1703 periodic_release = <ClassAd Boolean Expression>
1709 This expression is checked periodically when the job is in the Held
1711 state. If the expression becomes True , the job will be released.
1712 Only job ClassAd attributes will be defined for use by this ClassAd
1714 expression. Note that, by default, this expression is only checked
1715 once every 60 seconds. The period of these evaluations can be
1716 adjusted by setting the PERIODIC_EXPR_INTERVAL , MAX_PERI‐
1717 ODIC_EXPR_INTERVAL , and PERIODIC_EXPR_TIMESLICE configuration
1718 macros.
1719 periodic_remove = <ClassAd Boolean Expression>
1725 This expression is checked periodically. If it becomes True , the
1727 job is removed from the queue. If unspecified, the default value is
1728 False .
1729 See the Examples section of this manual page for an example of a
1731 periodic_remove expression.
1732 periodic_* expressions take precedence over on_exit_* expressions,
1735 and *_hold expressions take precedence over a *_remove expres‐
1736 sions. So, the periodic_remove expression takes precedent over the
1737 on_exit_remove expression, if the two describe conflicting actions.
1738 Only job ClassAd attributes will be defined for use by this ClassAd
1740 expression. Note that, by default, this expression is only checked
1741 once every 60 seconds. The period of these evaluations can be
1742 adjusted by setting the PERIODIC_EXPR_INTERVAL , MAX_PERI‐
1743 ODIC_EXPR_INTERVAL , and PERIODIC_EXPR_TIMESLICE configuration
1744 macros.
1745 COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
1751 allow_startup_script = <True | False>
1759 If True, a standard universe job will execute a script instead of
1761 submitting the job, and the consistency check to see if the exe‐
1762 cutable has been linked using condor_compile is omitted. The exe‐
1763 cutable command within the submit description file specifies the
1764 name of the script. The script is used to do preprocessing before
1765 the job is submitted. The shell script ends with an exec of the job
1766 executable, such that the process id of the executable is the same
1767 as that of the shell script. Here is an example script that gets a
1768 copy of a machine-specific executable before the exec .
1769 #! /bin/sh
1771 # get the host name of the machine
1773 $host=`uname -n`
1774 # grab a standard universe executable designed specifically
1776 # for this host
1777 scp elsewhere@cs.wisc.edu:${host} executable
1778 # The PID MUST stay the same, so exec the new standard universe
1780 process.
1781 exec executable ${1+"$@"} If this command is not present (defined),
1782 then the value defaults to false.
1783 append_files = file1, file2, ...
1789 If your job attempts to access a file mentioned in this list, HTCon‐
1793 dor will force all writes to that file to be appended to the end.
1794 Furthermore, condor_submit will not truncate it. This list uses the
1795 same syntax as compress_files, shown above.
1796 This option may yield some surprising results. If several jobs
1798 attempt to write to the same file, their output may be intermixed.
1799 If a job is evicted from one or more machines during the course of
1800 its lifetime, such an output file might contain several copies of
1801 the results. This option should be only be used when you wish a cer‐
1802 tain file to be treated as a running log instead of a precise
1803 result.
1804 This option only applies to standard-universe jobs.
1806 buffer_files < `` name (size,block-size) ; name2 (size,block-size) ...
1812 '' >
1813 buffer_size <bytes-in-buffer>
1819 buffer_block_size <bytes-in-block>
1825 HTCondor keeps a buffer of recently-used data for each file a job
1827 accesses. This buffer is used both to cache commonly-used data and
1828 to consolidate small reads and writes into larger operations that
1829 get better throughput. The default settings should produce reason‐
1830 able results for most programs.
1831 These options only apply to standard-universe jobs.
1833 If needed, you may set the buffer controls individually for each
1835 file using the buffer_files option. For example, to set the buffer
1836 size to 1 MiB and the block size to 256 KiB for the file input.data
1837 , use this command:
1838 buffer_files = "input.data=(1000000,256000)"
1842 Alternatively, you may use these two options to set the default
1844 sizes for all files used by your job:
1845 buffer_size = 1000000
1849 buffer_block_size = 256000
1850 If you do not set these, HTCondor will use the values given by these
1852 two configuration file macros:
1853 DEFAULT_IO_BUFFER_SIZE = 1000000
1857 DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
1858 Finally, if no other settings are present, HTCondor will use a buf‐
1860 fer of 512 KiB and a block size of 32 KiB.
1861 compress_files = file1, file2, ...
1867 If your job attempts to access any of the files mentioned in this
1871 list, HTCondor will automatically compress them (if writing) or
1872 decompress them (if reading). The compress format is the same as
1873 used by GNU gzip.
1874 The files given in this list may be simple file names or complete
1876 paths and may include as a wild card. For example, this list causes
1877 the file /tmp/data.gz, any file named event.gz, and any file ending
1878 in .gzip to be automatically compressed or decompressed as needed:
1879 compress_files = /tmp/data.gz, event.gz, *.gzip
1883 Due to the nature of the compression format, compressed files must
1885 only be accessed sequentially. Random access reading is allowed but
1886 is very slow, while random access writing is simply not possible.
1887 This restriction may be avoided by using both compress_files and
1888 fetch_files at the same time. When this is done, a file is kept in
1889 the decompressed state at the execution machine, but is compressed
1890 for transfer to its original location.
1891 This option only applies to standard universe jobs.
1893 fetch_files = file1, file2, ...
1899 If your job attempts to access a file mentioned in this list, HTCon‐
1901 dor will automatically copy the whole file to the executing machine,
1902 where it can be accessed quickly. When your job closes the file, it
1903 will be copied back to its original location. This list uses the
1904 same syntax as compress_files, shown above.
1905 This option only applies to standard universe jobs.
1907 file_remaps < `` name newname ; name2 newname2 ... ''>
1913 Directs HTCondor to use a new file name in place of an old one.
1917 name describes a file name that your job may attempt to open, and
1918 newname describes the file name it should be replaced with. newname
1919 may include an optional leading access specifier, local: or
1920 remote: . If left unspecified, the default access specifier is
1921 remote: . Multiple remaps can be specified by separating each with a
1922 semicolon.
1923 This option only applies to standard universe jobs.
1925 If you wish to remap file names that contain equals signs or semi‐
1927 colons, these special characters may be escaped with a backslash.
1928 Example One:
1932 Suppose that your job reads a file named dataset.1 . To instruct
1934 HTCondor to force your job to read other.dataset instead, add
1935 this to the submit file:
1936 file_remaps = "dataset.1=other.dataset"
1938 Example Two:
1942 Suppose that your run many jobs which all read in the same large
1944 file, called very.big . If this file can be found in the same
1945 place on a local disk in every machine in the pool, (say
1946 /bigdisk/bigfile ,) you can instruct HTCondor of this fact by
1947 remapping very.big to /bigdisk/bigfile and specifying that the
1948 file is to be read locally, which will be much faster than read‐
1949 ing over the network.
1950 file_remaps = "very.big = local:/bigdisk/bigfile"
1952 Example Three:
1956 Several remaps can be applied at once by separating each with a
1958 semicolon.
1959 file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 =
1961 other.dataset"
1962 local_files = file1, file2, ...
1970 If your job attempts to access a file mentioned in this list, HTCon‐
1974 dor will cause it to be read or written at the execution machine.
1975 This is most useful for temporary files not used for input or out‐
1976 put. This list uses the same syntax as compress_files, shown above.
1977 local_files = /tmp/*
1981 This option only applies to standard universe jobs.
1983 want_remote_io = <True | False>
1989 This option controls how a file is opened and manipulated in a stan‐
1991 dard universe job. If this option is true, which is the default,
1992 then the condor_shadow makes all decisions about how each and every
1993 file should be opened by the executing job. This entails a network
1994 round trip (or more) from the job to the condor_shadow and back
1995 again for every single open() in addition to other needed informa‐
1996 tion about the file. If set to false, then when the job queries the
1997 condor_shadow for the first time about how to open a file, the con‐
1998 dor_shadow will inform the job to automatically perform all of its
1999 file manipulation on the local file system on the execute machine
2000 and any file remapping will be ignored. This means that there must
2001 be a shared file system (such as NFS or AFS) between the execute
2002 machine and the submit machine and that ALL paths that the job could
2003 open on the execute machine must be valid. The ability of the stan‐
2004 dard universe job to checkpoint, possibly to a checkpoint server, is
2005 not affected by this attribute. However, when the job resumes it
2006 will be expecting the same file system conditions that were present
2007 when the job checkpointed.
2008 COMMANDS FOR THE GRID
2014 batch_queue = <queuename>
2022 Used for pbs , lsf , and sge grid universe jobs. Specifies the name
2024 of the PBS/LSF/SGE job queue into which the job should be submitted.
2025 If not specified, the default queue is used.
2026 boinc_authenticator_file = <pathname>
2032 For grid type boinc jobs, specifies a path and file name of the
2034 authorization file that grants permission for HTCondor to use the
2035 BOINC service. There is no default value when not specified.
2036 cream_attributes = <name=value;...;name=value>
2042 Provides a list of attribute/value pairs to be set in a CREAM job
2044 description of a grid universe job destined for the CREAM grid sys‐
2045 tem. The pairs are separated by semicolons, and written in New Clas‐
2046 sAd syntax.
2047 delegate_job_GSI_credentials_lifetime = <seconds>
2053 Specifies the maximum number of seconds for which delegated proxies
2055 should be valid. The default behavior when this command is not spec‐
2056 ified is determined by the configuration variable DELE‐
2057 GATE_JOB_GSI_CREDENTIALS_LIFETIME , which defaults to one day. A
2058 value of 0 indicates that the delegated proxy should be valid for as
2059 long as allowed by the credential used to create the proxy. This
2060 setting currently only applies to proxies delegated for non-grid
2061 jobs and for HTCondor-C jobs. It does not currently apply to globus
2062 grid jobs, which always behave as though this setting were 0. This
2063 variable has no effect if the configuration variable DELE‐
2064 GATE_JOB_GSI_CREDENTIALS is False , because in that case the job
2065 proxy is copied rather than delegated.
2066 ec2_access_key_id = <pathname>
2072 For grid type ec2 jobs, identifies the file containing the access
2074 key.
2075 ec2_ami_id = <EC2 xMI ID>
2081 For grid type ec2 jobs, identifies the machine image. Services com‐
2083 patible with the EC2 Query API may refer to these with abbreviations
2084 other than AMI , for example EMI is valid for Eucalyptus.
2085 ec2_availability_zone = <zone name>
2091 For grid type ec2 jobs, specifies the Availability Zone that the
2093 instance should be run in. This command is optional, unless
2094 ec2_ebs_volumes is set. As an example, one current zone is us-
2095 east-1b .
2096 ec2_block_device_mapping = <block-device>:<kernel-device>,<block-
2102 device>:<kernel-device>, ...
2103 For grid type ec2 jobs, specifies the block device to kernel device
2105 mapping. This command is optional.
2106 ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device name>,...
2112 For grid type ec2 jobs, optionally specifies a list of Elastic Block
2114 Store (EBS) volumes to be made available to the instance and the
2115 device names they should have in the instance.
2116 ec2_elastic_ip = <elastic IP address>
2122 For grid type ec2 jobs, and optional specification of an Elastic IP
2124 address that should be assigned to this instance.
2125 ec2_iam_profile_arn = <IAM profile ARN>
2131 For grid type ec2 jobs, an Amazon Resource Name (ARN) identifying
2133 which Identity and Access Management (IAM) (instance) profile to as‐
2134 sociate with the instance.
2135 ec2_iam_profile_name= <IAM profile name>
2141 For grid type ec2 jobs, a name identifying which Identity and Access
2143 Management (IAM) (instance) profile to associate with the instance.
2144 ec2_instance_type = <instance type>
2150 For grid type ec2 jobs, identifies the instance type. Different ser‐
2152 vices may offer different instance types, so no default value is
2153 set.
2154 ec2_keypair = <ssh key-pair name>
2160 For grid type ec2 jobs, specifies the name of an SSH key-pair that
2162 is already registered with the EC2 service. The associated private
2163 key can be used to ssh into the virtual machine once it is running.
2164 ec2_keypair_file = <pathname>
2170 For grid type ec2 jobs, specifies the complete path and file name of
2172 a file into which HTCondor will write an SSH key for use with ec2
2173 jobs. The key can be used to ssh into the virtual machine once it is
2174 running. If ec2_keypair is specified for a job, ec2_keypair_file is
2175 ignored.
2176 ec2_parameter_names = ParameterName1, ParameterName2, ...
2182 For grid type ec2 jobs, a space or comma separated list of the names
2184 of additional parameters to pass when instantiating an instance.
2185 ec2_parameter_<name> = <value>
2191 For grid type ec2 jobs, specifies the value for the correspondingly
2193 named (instance instantiation) parameter. <name> is the parameter
2194 name specified in the submit command ec2_parameter_names , but with
2195 any periods replaced by underscores.
2196 ec2_secret_access_key = <pathname>
2202 For grid type ec2 jobs, specifies the path and file name containing
2204 the secret access key.
2205 ec2_security_groups = group1, group2, ...
2211 For grid type ec2 jobs, defines the list of EC2 security groups
2213 which should be associated with the job.
2214 ec2_security_ids = id1, id2, ...
2220 For grid type ec2 jobs, defines the list of EC2 security group IDs
2222 which should be associated with the job.
2223 ec2_spot_price = <bid>
2229 For grid type ec2 jobs, specifies the spot instance bid, which is
2231 the most that the job submitter is willing to pay per hour to run
2232 this job.
2233 ec2_tag_names = <name0,name1,name...>
2239 For grid type ec2 jobs, specifies the case of tag names that will be
2241 associated with the running instance. This is only necessary if a
2242 tag name case matters. By default the list will be automatically
2243 generated.
2244 ec2_tag_<name> = <value>
2250 For grid type ec2 jobs, specifies a tag to be associated with the
2252 running instance. The tag name will be lower-cased, use
2253 ec2_tag_names to change the case.
2254 WantNameTag = <True | False>
2260 For grid type ec2 jobs, a job may request that its 'name' tag be
2262 (not) set by HTCondor. If the job does not otherwise specify any
2263 tags, not setting its name tag will eliminate a call by the EC2
2264 GAHP, improving performance.
2265 ec2_user_data = <data>
2271 For grid type ec2 jobs, provides a block of data that can be
2273 accessed by the virtual machine. If both ec2_user_data and
2274 ec2_user_data_file are specified for a job, the two blocks of data
2275 are concatenated, with the data from this ec2_user_data submit com‐
2276 mand occurring first.
2277 ec2_user_data_file = <pathname>
2283 For grid type ec2 jobs, specifies a path and file name whose con‐
2285 tents can be accessed by the virtual machine. If both ec2_user_data
2286 and ec2_user_data_file are specified for a job, the two blocks of
2287 data are concatenated, with the data from that ec2_user_data submit
2288 command occurring first.
2289 ec2_vpc_ip = <a.b.c.d>
2295 For grid type ec2 jobs, that are part of a Virtual Private Cloud
2297 (VPC), an optional specification of the IP address that this
2298 instance should have within the VPC.
2299 ec2_vpc_subnet = <subnet specification string>
2305 For grid type ec2 jobs, an optional specification of the Virtual
2307 Private Cloud (VPC) that this instance should be a part of.
2308 gce_auth_file = <pathname>
2314 For grid type gce jobs, specifies a path and file name of the autho‐
2316 rization file that grants permission for HTCondor to use the Google
2317 account. There is no default value when not specified.
2318 gce_image = <image id>
2324 For grid type gce jobs, the identifier of the virtual machine image
2326 representing the HTCondor job to be run. This virtual machine image
2327 must already be register with GCE and reside in Google's Cloud Stor‐
2328 age service.
2329 gce_json_file = <pathname>
2335 For grid type gce jobs, specifies the path and file name of a file
2337 that contains JSON elements that should be added to the instance
2338 description submitted to the GCE service.
2339 gce_machine_type = <machine type>
2345 For grid type gce jobs, the long form of the URL that describes the
2347 machine configuration that the virtual machine instance is to run
2348 on.
2349 gce_metadata = <name=value,...,name=value>
2355 For grid type gce jobs, a comma separated list of name and value
2357 pairs that define metadata for a virtual machine instance that is an
2358 HTCondor job.
2359 gce_metadata_file = <pathname>
2365 For grid type gce jobs, specifies a path and file name of the file
2367 that contains metadata for a virtual machine instance that is an
2368 HTCondor job. Within the file, each name and value pair is on its
2369 own line; so, the pairs are separated by the newline character.
2370 gce_preemptible = <True | False>
2376 For grid type gce jobs, specifies whether the virtual machine
2378 instance should be preemptible. The default is for the instance to
2379 not be preemptible.
2380 globus_rematch = <ClassAd Boolean Expression>
2386 This expression is evaluated by the condor_gridmanager whenever:
2388 1. the globus_resubmit expression evaluates to True
2390 2. the condor_gridmanager decides it needs to retry a submission
2392 (as when a previous submission failed to commit) If
2393 globus_rematch evaluates to True , then before the job is sub‐
2394 mitted again to globus, the condor_gridmanager will request that
2395 the condor_schedd daemon renegotiate with the matchmaker (the
2396 condor_negotiator ). The result is this job will be matched
2397 again.
2398 globus_resubmit = <ClassAd Boolean Expression>
2404 The expression is evaluated by the condor_gridmanager each time the
2406 condor_gridmanager gets a job ad to manage. Therefore, the expres‐
2407 sion is evaluated:
2408 1. when a grid universe job is first submitted to HTCondor-G
2410 2. when a grid universe job is released from the hold state
2412 3. when HTCondor-G is restarted (specifically, whenever the con‐
2414 dor_gridmanager is restarted) If the expression evaluates to
2415 True , then any previous submission to the grid universe will be
2416 forgotten and this job will be submitted again as a fresh submis‐
2417 sion to the grid universe. This may be useful if there is a
2418 desire to give up on a previous submission and try again. Note
2419 that this may result in the same job running more than once. Do
2420 not treat this operation lightly.
2421 globus_rsl = <RSL-string>
2427 Used to provide any additional Globus RSL string attributes which
2429 are not covered by other submit description file commands or job
2430 attributes. Used for grid universe jobs, where the grid resource has
2431 a grid-type-string of gt2 .
2432 grid_resource = <grid-type-string> <grid-specific-parameter-list>
2438 For each grid-type-string value, there are further type-specific
2440 values that must specified. This submit description file command
2441 allows each to be given in a space-separated list. Allowable grid-
2442 type-string values are batch , condor , cream , ec2 , gt2 , gt5 ,
2443 lsf , nordugrid , pbs , sge , and unicore . The HTCondor manual
2444 chapter on Grid Computing details the variety of grid types.
2445 For a grid-type-string of batch , the single parameter is the name
2447 of the local batch system, and will be one of pbs , lsf , or sge
2448 .
2449 For a grid-type-string of condor , the first parameter is the name
2451 of the remote condor_schedd daemon. The second parameter is the name
2452 of the pool to which the remote condor_schedd daemon belongs.
2453 For a grid-type-string of cream , there are three parameters. The
2455 first parameter is the web services address of the CREAM server. The
2456 second parameter is the name of the batch system that sits behind
2457 the CREAM server. The third parameter identifies a site-specific
2458 queue within the batch system.
2459 For a grid-type-string of ec2 , one additional parameter specifies
2461 the EC2 URL.
2462 For a grid-type-string of gt2 , the single parameter is the name of
2464 the pre-WS GRAM resource to be used.
2465 For a grid-type-string of gt5 , the single parameter is the name of
2467 the pre-WS GRAM resource to be used, which is the same as for the
2468 grid-type-string of gt2 .
2469 For a grid-type-string of lsf , no additional parameters are used.
2471 For a grid-type-string of nordugrid , the single parameter is the
2473 name of the NorduGrid resource to be used.
2474 For a grid-type-string of pbs , no additional parameters are used.
2476 For a grid-type-string of sge , no additional parameters are used.
2478 For a grid-type-string of unicore , the first parameter is the name
2480 of the Unicore Usite to be used. The second parameter is the name of
2481 the Unicore Vsite to be used.
2482 keystore_alias = <name>
2488 A string to locate the certificate in a Java keystore file, as used
2490 for a unicore job.
2491 keystore_file = <pathname>
2497 The complete path and file name of the Java keystore file containing
2499 the certificate to be used for a unicore job.
2500 keystore_passphrase_file = <pathname>
2506 The complete path and file name to the file containing the
2508 passphrase protecting a Java keystore file containing the certifi‐
2509 cate. Relevant for a unicore job.
2510 MyProxyCredentialName = <symbolic name>
2516 The symbolic name that identifies a credential to the MyProxy
2518 server. This symbolic name is set as the credential is initially
2519 stored on the server (using myproxy-init ).
2520 MyProxyHost = <host>:<port>
2526 The Internet address of the host that is the MyProxy server. The
2528 host may be specified by either a host name (as in head.example.com
2529 ) or an IP address (of the form 123.456.7.8). The port number is an
2530 integer.
2531 MyProxyNewProxyLifetime = <number-of-minutes>
2537 The new lifetime (in minutes) of the proxy after it is refreshed.
2539 MyProxyPassword = <password>
2545 The password needed to refresh a credential on the MyProxy server.
2547 This password is set when the user initially stores credentials on
2548 the server (using myproxy-init ). As an alternative to using MyProx‐
2549 yPassword in the submit description file, the password may be speci‐
2550 fied as a command line argument to condor_submit with the -password
2551 argument.
2552 MyProxyRefreshThreshold = <number-of-seconds>
2558 The time (in seconds) before the expiration of a proxy that the
2560 proxy should be refreshed. For example, if MyProxyRefreshThreshold
2561 is set to the value 600, the proxy will be refreshed 10 minutes
2562 before it expires.
2563 MyProxyServerDN = <credential subject>
2569 A string that specifies the expected Distinguished Name (credential
2571 subject, abbreviated DN) of the MyProxy server. It must be specified
2572 when the MyProxy server DN does not follow the conventional naming
2573 scheme of a host credential. This occurs, for example, when the
2574 MyProxy server DN begins with a user credential.
2575 nordugrid_rsl = <RSL-string>
2581 Used to provide any additional RSL string attributes which are not
2583 covered by regular submit description file parameters. Used when the
2584 universe is grid , and the type of grid system is nordugrid .
2585 transfer_error = <True | False>
2591 For jobs submitted to the grid universe only. If True , then the
2593 error output (from stderr ) from the job is transferred from the
2594 remote machine back to the submit machine. The name of the file
2595 after transfer is given by the error command. If False , no trans‐
2596 fer takes place (from the remote machine to submit machine), and the
2597 name of the file is given by the error command. The default value is
2598 True .
2599 transfer_input = <True | False>
2605 For jobs submitted to the grid universe only. If True , then the
2607 job input ( stdin ) is transferred from the machine where the job
2608 was submitted to the remote machine. The name of the file that is
2609 transferred is given by the input command. If False , then the
2610 job's input is taken from a pre-staged file on the remote machine,
2611 and the name of the file is given by the input command. The default
2612 value is True .
2613 For transferring files other than stdin , see transfer_input_files
2615 .
2616 transfer_output = <True | False>
2622 For jobs submitted to the grid universe only. If True , then the
2624 output (from stdout ) from the job is transferred from the remote
2625 machine back to the submit machine. The name of the file after
2626 transfer is given by the output command. If False , no transfer
2627 takes place (from the remote machine to submit machine), and the
2628 name of the file is given by the output command. The default value
2629 is True .
2630 For transferring files other than stdout , see transfer_out‐
2632 put_files .
2633 use_x509userproxy = <True | False>
2639 Set this command to True to indicate that the job requires an X.509
2641 user proxy. If x509userproxy is set, then that file is used for the
2642 proxy. Otherwise, the proxy is looked for in the standard locations.
2643 If x509userproxy is set or if the job is a grid universe job of grid
2644 type gt2 , gt5 , cream , or nordugrid , then the value of
2645 use_x509userproxy is forced to True . Defaults to False .
2646 x509userproxy = <full-pathname>
2652 Used to override the default path name for X.509 user certificates.
2654 The default location for X.509 proxies is the /tmp directory, which
2655 is generally a local file system. Setting this value would allow
2656 HTCondor to access the proxy in a shared file system (for example,
2657 AFS). HTCondor will use the proxy specified in the submit descrip‐
2658 tion file first. If nothing is specified in the submit description
2659 file, it will use the environment variable X509_USER_PROXY. If that
2660 variable is not present, it will search in the default location.
2661 Note that proxies are only valid for a limited time. Condor_submit
2662 will not submit a job with an expired proxy, it will return an
2663 error. Also, if the configuration parameter CRED_MIN_TIME_LEFT is
2664 set to some number of seconds, and if the proxy will expire before
2665 that many seconds, condor_submit will also refuse to submit the job.
2666 That is, if CRED_MIN_TIME_LEFT is set to 60, condor_submit will
2667 refuse to submit a job whose proxy will expire 60 seconds from the
2668 time of submission.
2669 x509userproxy is relevant when the universe is vanilla , or when the
2671 universe is grid and the type of grid system is one of gt2 , gt5 ,
2672 condor , cream , or nordugrid . Defining a value causes the proxy to
2673 be delegated to the execute machine. Further, VOMS attributes
2674 defined in the proxy will appear in the job ClassAd.
2675 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2681 hold_kill_sig = <signal-number>
2689 For the scheduler universe only, signal-number is the signal deliv‐
2691 ered to the job when the job is put on hold with condor_hold . sig‐
2692 nal-number may be either the platform-specific name or value of the
2693 signal. If this command is not present, the value of kill_sig is
2694 used.
2695 jar_files = <file_list>
2701 Specifies a list of additional JAR files to include when using the
2703 Java universe. JAR files will be transferred along with the exe‐
2704 cutable and automatically added to the classpath.
2705 java_vm_args = <argument_list>
2711 Specifies a list of additional arguments to the Java VM itself, When
2713 HTCondor runs the Java program, these are the arguments that go
2714 before the class name. This can be used to set VM-specific arguments
2715 like stack size, garbage-collector arguments and initial property
2716 values.
2717 machine_count = <max>
2723 For the parallel universe, a single value ( max ) is required. It is
2725 neither a maximum or minimum, but the number of machines to be dedi‐
2726 cated toward running the job.
2727 remove_kill_sig = <signal-number>
2733 For the scheduler universe only, signal-number is the signal deliv‐
2735 ered to the job when the job is removed with condor_rm . signal-
2736 number may be either the platform-specific name or value of the sig‐
2737 nal. This example shows it both ways for a Linux signal:
2738 remove_kill_sig = SIGUSR1
2740 remove_kill_sig = 10
2741 If this command is not present, the value of kill_sig is used.
2743 COMMANDS FOR THE VM UNIVERSE
2749 vm_disk = file1:device1:permission1, file2:device2:permission2:format2,
2757 ...
2758 A list of comma separated disk files. Each disk file is specified by
2760 4 colon separated fields. The first field is the path and file name
2761 of the disk file. The second field specifies the device. The third
2762 field specifies permissions, and the optional fourth field specifies
2763 the image format. If a disk file will be transferred by HTCondor,
2764 then the first field should just be the simple file name (no path
2765 information).
2766 An example that specifies two disk files:
2768 vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2770 vm_checkpoint = <True | False>
2776 A boolean value specifying whether or not to take checkpoints. If
2778 not specified, the default value is False . In the current imple‐
2779 mentation, setting both vm_checkpoint and vm_networking to True
2780 does not yet work in all cases. Networking cannot be used if a vm
2781 universe job uses a checkpoint in order to continue execution after
2782 migration to another machine.
2783 vm_macaddr = <MACAddr>
2789 Defines that MAC address that the virtual machine's network inter‐
2791 face should have, in the standard format of six groups of two hexa‐
2792 decimal digits separated by colons.
2793 vm_memory = <MBytes-of-memory>
2799 The amount of memory in MBytes that a vm universe job requires.
2801 vm_networking = <True | False>
2807 Specifies whether to use networking or not. In the current implemen‐
2809 tation, setting both vm_checkpoint and vm_networking to True does
2810 not yet work in all cases. Networking cannot be used if a vm uni‐
2811 verse job uses a checkpoint in order to continue execution after
2812 migration to another machine.
2813 vm_networking_type = <nat | bridge >
2819 When vm_networking is True , this definition augments the job's
2821 requirements to match only machines with the specified networking.
2822 If not specified, then either networking type matches.
2823 vm_no_output_vm = <True | False>
2829 When True , prevents HTCondor from transferring output files back
2831 to the machine from which the vm universe job was submitted. If not
2832 specified, the default value is False .
2833 vm_type = <vmware | xen | kvm>
2839 Specifies the underlying virtual machine software that this job
2841 expects.
2842 vmware_dir = <pathname>
2848 The complete path and name of the directory where VMware-specific
2850 files and applications such as the VMDK (Virtual Machine Disk For‐
2851 mat) and VMX (Virtual Machine Configuration) reside. This command is
2852 optional; when not specified, all relevant VMware image files are to
2853 be listed using transfer_input_files .
2854 vmware_should_transfer_files = <True | False>
2860 Specifies whether HTCondor will transfer VMware-specific files
2862 located as specified by vmware_dir to the execute machine ( True )
2863 or rely on access through a shared file system ( False ). Omission
2864 of this required command (for VMware vm universe jobs) results in an
2865 error message from condor_submit , and the job will not be submit‐
2866 ted.
2867 vmware_snapshot_disk = <True | False>
2873 When True , causes HTCondor to utilize a VMware snapshot disk for
2875 new or modified files. If not specified, the default value is True
2876 .
2877 xen_initrd = <image-file>
2883 When xen_kernel gives a file name for the kernel image to use, this
2885 optional command may specify a path to a ramdisk ( initrd ) image
2886 file. If the image file will be transferred by HTCondor, then the
2887 value should just be the simple file name (no path information).
2888 xen_kernel = <included | path-to-kernel>
2894 A value of included specifies that the kernel is included in the
2896 disk file. If not one of these values, then the value is a path and
2897 file name of the kernel to be used. If a kernel file will be trans‐
2898 ferred by HTCondor, then the value should just be the simple file
2899 name (no path information).
2900 xen_kernel_params = <string>
2906 A string that is appended to the Xen kernel command line.
2908 xen_root = <string>
2914 A string that is appended to the Xen kernel command line to specify
2916 the root device. This string is required when xen_kernel gives a
2917 path to a kernel. Omission for this required case results in an
2918 error message during submission.
2919 COMMANDS FOR THE DOCKER UNIVERSE
2925 docker_image = < image-name >
2933 Defines the name of the Docker image that is the basis for the
2935 docker container.
2936 ADVANCED COMMANDS
2942 accounting_group = <accounting-group-name>
2950 Causes jobs to negotiate under the given accounting group. This
2952 value is advertised in the job ClassAd as AcctGroup . The HTCondor
2953 Administrator's manual contains more information about accounting
2954 groups.
2955 accounting_group_user = <accounting-group-user-name>
2961 Sets the user name associated with the accounting group name for
2963 resource usage accounting purposes. If not set, defaults to the
2964 value of the job ClassAd attribute Owner . This value is advertised
2965 in the job ClassAd as AcctGroupUser . If an accounting group has
2966 not been set with the accounting_group command, this command is
2967 ignored.
2968 concurrency_limits = <string-list>
2974 A list of resources that this job needs. The resources are presumed
2976 to have concurrency limits placed upon them, thereby limiting the
2977 number of concurrent jobs in execution which need the named
2978 resource. Commas and space characters delimit the items in the list.
2979 Each item in the list is a string that identifies the limit, or it
2980 is a ClassAd expression that evaluates to a string, and it is evalu‐
2981 ated in the context of machine ClassAd being considered as a match.
2982 Each item in the list also may specify a numerical value identifying
2983 the integer number of resources required for the job. The syntax
2984 follows the resource name by a colon character ( : ) and the numeri‐
2985 cal value. Details on concurrency limits are in the HTCondor Admin‐
2986 istrator's manual.
2987 concurrency_limits_expr = <ClassAd String Expression>
2993 A ClassAd expression that represents the list of resources that this
2995 job needs after evaluation. The ClassAd expression may specify
2996 machine ClassAd attributes that are evaluated against a matched
2997 machine. After evaluation, the list sets concurrency_limits .
2998 copy_to_spool = <True | False>
3004 If copy_to_spool is True , then condor_submit copies the executable
3006 to the local spool directory before running it on a remote host. As
3007 copying can be quite time consuming and unnecessary, the default
3008 value is False for all job universes other than the standard uni‐
3009 verse. When False , condor_submit does not copy the executable to a
3010 local spool directory. The default is True in standard universe,
3011 because resuming execution from a checkpoint can only be guaranteed
3012 to work using precisely the same executable that created the check‐
3013 point.
3014 coresize = <size>
3020 Should the user's program abort and produce a core file, coresize
3022 specifies the maximum size in bytes of the core file which the user
3023 wishes to keep. If coresize is not specified in the command file,
3024 the system's user resource limit coredumpsize is used (note that
3025 coredumpsize is not an HTCondor parameter - it is an operating sys‐
3026 tem parameter that can be viewed with the limit or ulimit command on
3027 Unix and in the Registry on Windows). A value of -1 results in no
3028 limits being applied to the core file size. If HTCondor is running
3029 as root, a coresize setting greater than the system coredumpsize
3030 limit will override the system setting; if HTCondor is not running
3031 as root, the system coredumpsize limit will override coresize .
3032 cron_day_of_month = <Cron-evaluated Day>
3038 The set of days of the month for which a deferral time applies. The
3040 HTCondor User's manual section on Time Scheduling for Job Execution
3041 has further details.
3042 cron_day_of_week = <Cron-evaluated Day>
3048 The set of days of the week for which a deferral time applies. The
3050 HTCondor User's manual section on Time Scheduling for Job Execution
3051 has further details.
3052 cron_hour = <Cron-evaluated Hour>
3058 The set of hours of the day for which a deferral time applies. The
3060 HTCondor User's manual section on Time Scheduling for Job Execution
3061 has further details.
3062 cron_minute = <Cron-evaluated Minute>
3068 The set of minutes within an hour for which a deferral time applies.
3070 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3071 tion has further details.
3072 cron_month = <Cron-evaluated Month>
3078 The set of months within a year for which a deferral time applies.
3080 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3081 tion has further details.
3082 cron_prep_time = <ClassAd Integer Expression>
3088 Analogous to deferral_prep_time . The number of seconds prior to a
3090 job's deferral time that the job may be matched and sent to an exe‐
3091 cution machine.
3092 cron_window = <ClassAd Integer Expression>
3098 Analogous to the submit command deferral_window . It allows cron
3100 jobs that miss their deferral time to begin execution.
3101 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3103 tion has further details.
3104 dagman_log = <pathname>
3110 DAGMan inserts this command to specify an event log that it watches
3112 to maintain the state of the DAG. If the log command is not speci‐
3113 fied in the submit file, DAGMan uses the log command to specify the
3114 event log.
3115 deferral_prep_time = <ClassAd Integer Expression>
3121 The number of seconds prior to a job's deferral time that the job
3123 may be matched and sent to an execution machine.
3124 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3126 tion has further details.
3127 deferral_time = <ClassAd Integer Expression>
3133 Allows a job to specify the time at which its execution is to begin,
3135 instead of beginning execution as soon as it arrives at the execu‐
3136 tion machine. The deferral time is an expression that evaluates to a
3137 Unix Epoch timestamp (the number of seconds elapsed since 00:00:00
3138 on January 1, 1970, Coordinated Universal Time). Deferral time is
3139 evaluated with respect to the execution machine. This option delays
3140 the start of execution, but not the matching and claiming of a
3141 machine for the job. If the job is not available and ready to begin
3142 execution at the deferral time, it has missed its deferral time. A
3143 job that misses its deferral time will be put on hold in the queue.
3144 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3146 tion has further details.
3147 Due to implementation details, a deferral time may not be used for
3149 scheduler universe jobs.
3150 deferral_window = <ClassAd Integer Expression>
3156 The deferral window is used in conjunction with the deferral_time
3158 command to allow jobs that miss their deferral time to begin execu‐
3159 tion.
3160 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3162 tion has further details.
3163 description = <string>
3169 A string that sets the value of the job ClassAd attribute JobDe‐
3171 scription . When set, tools which display the executable such as
3172 condor_q will instead use this string.
3173 email_attributes = <list-of-job-ad-attributes>
3179 A comma-separated list of attributes from the job ClassAd. These
3181 attributes and their values will be included in the e-mail notifica‐
3182 tion of job completion.
3183 image_size = <size>
3189 Advice to HTCondor specifying the maximum virtual image size to
3191 which the job will grow during its execution. HTCondor will then
3192 execute the job only on machines which have enough resources, (such
3193 as virtual memory), to support executing the job. If not specified,
3194 HTCondor will automatically make a (reasonably accurate) estimate
3195 about the job's size and adjust this estimate as the program runs.
3196 If specified and underestimated, the job may crash due to the
3197 inability to acquire more address space; for example, if malloc()
3198 fails. If the image size is overestimated, HTCondor may have diffi‐
3199 culty finding machines which have the required resources. size is
3200 specified in KiB. For example, for an image size of 8 MiB, size
3201 should be 8000.
3202 initialdir = <directory-path>
3208 Used to give jobs a directory with respect to file input and output.
3210 Also provides a directory (on the machine from which the job is sub‐
3211 mitted) for the job event log, when a full path is not specified.
3212 For vanilla universe jobs where there is a shared file system, it is
3214 the current working directory on the machine where the job is exe‐
3215 cuted.
3216 For vanilla or grid universe jobs where file transfer mechanisms are
3218 utilized (there is not a shared file system), it is the directory on
3219 the machine from which the job is submitted where the input files
3220 come from, and where the job's output files go to.
3221 For standard universe jobs, it is the directory on the machine from
3223 which the job is submitted where the condor_shadow daemon runs; the
3224 current working directory for file input and output accomplished
3225 through remote system calls.
3226 For scheduler universe jobs, it is the directory on the machine from
3228 which the job is submitted where the job runs; the current working
3229 directory for file input and output with respect to relative path
3230 names.
3231 Note that the path to the executable is not relative to initialdir ;
3233 if it is a relative path, it is relative to the directory in which
3234 the condor_submit command is run.
3235 job_ad_information_attrs = <attribute-list>
3241 A comma-separated list of job ClassAd attribute names. The named
3243 attributes and their values are written to the job event log when‐
3244 ever any event is being written to the log. This implements the same
3245 thing as the configuration variable EVENT_LOG_INFORMATION_ATTRS
3246 (see page ), but it applies to the job event log, instead of the
3247 system event log.
3248 JobBatchName = <batch_name>
3254 Set the batch name for this submit. The batch name is displayed by
3256 condor_q -batch . It is intended for use by users to give meaningful
3257 names to their jobs and to influence how condor_q groups jobs for
3258 display. This value in a submit file can be overridden by specifying
3259 the -batch-name argument on the condor_submit command line.
3260 job_lease_duration = <number-of-seconds>
3266 For vanilla, parallel, VM, and java universe jobs only, the duration
3268 in seconds of a job lease. The default value is 2,400, or forty min‐
3269 utes. If a job lease is not desired, the value can be explicitly set
3270 to 0 to disable the job lease semantics. The value can also be a
3271 ClassAd expression that evaluates to an integer. The HTCondor User's
3272 manual section on Special Environment Considerations has further
3273 details.
3274 job_machine_attrs = <attr1, attr2, ...>
3280 A comma and/or space separated list of machine attribute names that
3282 should be recorded in the job ClassAd in addition to the ones speci‐
3283 fied by the condor_schedd daemon's system configuration variable
3284 SYSTEM_JOB_MACHINE_ATTRS . When there are multiple run attempts,
3285 history of machine attributes from previous run attempts may be
3286 kept. The number of run attempts to store may be extended beyond the
3287 system-specified history length by using the submit file command
3288 job_machine_attrs_history_length . A machine attribute named X will
3289 be inserted into the job ClassAd as an attribute named MachineAt‐
3290 trX0 . The previous value of this attribute will be named
3291 MachineAttrX1 , the previous to that will be named MachineAttrX2 ,
3292 and so on, up to the specified history length. A history of length 1
3293 means that only MachineAttrX0 will be recorded. The value recorded
3294 in the job ClassAd is the evaluation of the machine attribute in the
3295 context of the job ClassAd when the condor_schedd daemon initiates
3296 the start up of the job. If the evaluation results in an Undefined
3297 or Error result, the value recorded in the job ad will be Unde‐
3298 fined or Error , respectively.
3299 want_graceful_removal = <boolean expression>
3305 When True , this causes a graceful shutdown of the job when the job
3307 is removed or put on hold, giving it time to clean up or save state.
3308 Otherwise, the job is abruptly killed. The default is false .
3309 kill_sig = <signal-number>
3315 When HTCondor needs to kick a job off of a machine, it will send the
3317 job the signal specified by signal-number . signal-number needs to
3318 be an integer which represents a valid signal on the execution
3319 machine. For jobs submitted to the standard universe, the default
3320 value is the number for SIGTSTP which tells the HTCondor libraries
3321 to initiate a checkpoint of the process. For jobs submitted to other
3322 universes, the default value, when not defined, is SIGTERM , which
3323 is the standard way to terminate a program in Unix.
3324 kill_sig_timeout = <seconds>
3330 This submit command should no longer be used as of HTCondor version
3332 7.7.3; use job_max_vacate_time instead. If job_max_vacate_time is
3333 not defined, this defines the number of seconds that HTCondor should
3334 wait following the sending of the kill signal defined by kill_sig
3335 and forcibly killing the job. The actual amount of time between
3336 sending the signal and forcibly killing the job is the smallest of
3337 this value and the configuration variable KILLING_TIMEOUT , as
3338 defined on the execute machine.
3339 load_profile = <True | False>
3345 When True , loads the account profile of the dedicated run account
3347 for Windows jobs. May not be used with run_as_owner .
3348 match_list_length = <integer value>
3354 Defaults to the value zero (0). When match_list_length is defined
3356 with an integer value greater than zero (0), attributes are inserted
3357 into the job ClassAd. The maximum number of attributes defined is
3358 given by the integer value. The job ClassAds introduced are given as
3359 LastMatchName0 = "most-recent-Name"
3361 LastMatchName1 = "next-most-recent-Name"
3362 The value for each introduced ClassAd is given by the value of the
3364 Name attribute from the machine ClassAd of a previous execution
3365 (match). As a job is matched, the definitions for these attributes
3366 will roll, with LastMatchName1 becoming LastMatchName2 , Last‐
3367 MatchName0 becoming LastMatchName1 , and LastMatchName0 being set
3368 by the most recent value of the Name attribute.
3369 An intended use of these job attributes is in the requirements
3371 expression. The requirements can allow a job to prefer a match with
3372 either the same or a different resource than a previous match.
3373 job_max_vacate_time = <integer expression>
3379 An integer-valued expression (in seconds) that may be used to adjust
3381 the time given to an evicted job for gracefully shutting down. If
3382 the job's setting is less than the machine's, the job's is used. If
3383 the job's setting is larger than the machine's, the result depends
3384 on whether the job has any excess retirement time. If the job has
3385 more retirement time left than the machine's max vacate time set‐
3386 ting, then retirement time will be converted into vacating time, up
3387 to the amount requested by the job.
3388 Setting this expression does not affect the job's resource require‐
3390 ments or preferences. For a job to only run on a machine with a min‐
3391 imum MachineMaxVacateTime , or to preferentially run on such
3392 machines, explicitly specify this in the requirements and/or rank
3393 expressions.
3394 max_job_retirement_time = <integer expression>
3400 An integer-valued expression (in seconds) that does nothing unless
3402 the machine that runs the job has been configured to provide retire‐
3403 ment time. Retirement time is a grace period given to a job to fin‐
3404 ish when a resource claim is about to be preempted. The default
3405 behavior in many cases is to take as much retirement time as the
3406 machine offers, so this command will rarely appear in a submit
3407 description file.
3408 When a resource claim is to be preempted, this expression in the
3410 submit file specifies the maximum run time of the job (in seconds,
3411 since the job started). This expression has no effect, if it is
3412 greater than the maximum retirement time provided by the machine
3413 policy. If the resource claim is not preempted, this expression and
3414 the machine retirement policy are irrelevant. If the resource claim
3415 is preempted the job will be allowed to run until the retirement
3416 time expires, at which point it is hard-killed. The job will be
3417 soft-killed when it is getting close to the end of retirement in
3418 order to give it time to gracefully shut down. The amount of lead-
3419 time for soft-killing is determined by the maximum vacating time
3420 granted to the job.
3421 Standard universe jobs and any jobs running with nice_user priority
3423 have a default max_job_retirement_time of 0, so no retirement time
3424 is utilized by default. In all other cases, no default value is pro‐
3425 vided, so the maximum amount of retirement time is utilized by
3426 default.
3427 Setting this expression does not affect the job's resource require‐
3429 ments or preferences. For a job to only run on a machine with a min‐
3430 imum MaxJobRetirementTime , or to preferentially run on such
3431 machines, explicitly specify this in the requirements and/or rank
3432 expressions.
3433 nice_user = <True | False>
3439 Normally, when a machine becomes available to HTCondor, HTCondor
3441 decides which job to run based upon user and job priorities. Setting
3442 nice_user equal to True tells HTCondor not to use your regular user
3443 priority, but that this job should have last priority among all
3444 users and all jobs. So jobs submitted in this fashion run only on
3445 machines which no other non-nice_user job wants -- a true bottom-
3446 feeder job! This is very handy if a user has some jobs they wish to
3447 run, but do not wish to use resources that could instead be used to
3448 run other people's HTCondor jobs. Jobs submitted in this fashion
3449 have "nice-user." prepended to the owner name when viewed from con‐
3450 dor_q or condor_userprio . The default value is False .
3451 noop_job = <ClassAd Boolean Expression>
3457 When this boolean expression is True , the job is immediately
3459 removed from the queue, and HTCondor makes no attempt at running the
3460 job. The log file for the job will show a job submitted event and a
3461 job terminated event, along with an exit code of 0, unless the user
3462 specifies a different signal or exit code.
3463 noop_job_exit_code = <return value>
3469 When noop_job is in the submit description file and evaluates to
3471 True , this command allows the job to specify the return value as
3472 shown in the job's log file job terminated event. If not specified,
3473 the job will show as having terminated with status 0. This overrides
3474 any value specified with noop_job_exit_signal .
3475 noop_job_exit_signal = <signal number>
3481 When noop_job is in the submit description file and evaluates to
3483 True , this command allows the job to specify the signal number that
3484 the job's log event will show the job having terminated with.
3485 remote_initialdir = <directory-path>
3491 The path specifies the directory in which the job is to be executed
3493 on the remote machine. This is currently supported in all universes
3494 except for the standard universe.
3495 rendezvousdir = <directory-path>
3501 Used to specify the shared file system directory to be used for file
3503 system authentication when submitting to a remote scheduler. Should
3504 be a path to a preexisting directory.
3505 run_as_owner = <True | False>
3511 A boolean value that causes the job to be run under the login of the
3513 submitter, if supported by the joint configuration of the submit and
3514 execute machines. On Unix platforms, this defaults to True , and on
3515 Windows platforms, it defaults to False . May not be used with
3516 load_profile . See the HTCondor manual Platform-Specific Information
3517 chapter for administrative details on configuring Windows to support
3518 this option.
3519 stack_size = <size in bytes>
3525 This command applies only to Linux platform jobs that are not stan‐
3527 dard universe jobs. An integer number of bytes, representing the
3528 amount of stack space to be allocated for the job. This value
3529 replaces the default allocation of stack space, which is unlimited
3530 in size.
3531 submit_event_notes = <note>
3537 A string that is appended to the submit event in the job's log file.
3539 For DAGMan jobs, the string DAG Node: and the node's name is auto‐
3540 matically defined for submit_event_notes , causing the logged submit
3541 event to identify the DAG node job submitted.
3542 +<attribute> = <value>
3548 A line that begins with a '+' (plus) character instructs condor_sub‐
3550 mit to insert the given attribute into the job ClassAd with the
3551 given value . Note that setting an attribute should not be used in
3552 place of one of the specific commands listed above. Often, the com‐
3553 mand name does not directly correspond to an attribute name; fur‐
3554 thermore, many submit commands result in actions more complex than
3555 simply setting an attribute or attributes. See for a list of HTCon‐
3556 dor job attributes.
3557 PRE AND POST SCRIPTS IMPLEMENTED WITH SPECIALLY-NAMED ATTRIBUTES
3563 +PreCmd = <executable>
3571 A vanilla universe job may specify that a script is to be run on the
3573 execute machine before the job, and this is called a prescript. Def‐
3574 inition of this specifically-named attribute causes the script,
3575 identified by path and file name, to be executed. The prescript
3576 could prepare or initialize the job. Note that this definition of a
3577 prescript is different from the PRE script described in DAGMan. The
3578 prescript is not automatically transferred with the job, as the main
3579 executable is, so it must be entered into the transfer_input_files
3580 list, when file transfer is enabled.
3581 +PreArgs = <argument_list>
3587 Defines command line arguments for the prescript, presuming the Old
3589 argument syntax.
3590 +PreArguments = <argument_list>
3596 Defines command line arguments for the prescript, presuming the New
3598 argument syntax. An exception to the syntax is that double quotes
3599 must be escaped with a backslash instead of another double quote.
3600 Note that if both +PreArgs and +PreArguments are specified, the
3602 +PreArguments value is used and the +PreArgs value is ignored.
3603 +PreEnv = <environment_vars>
3609 Defines the environment for the prescript, presuming the Old envi‐
3611 ronment syntax.
3612 +PreEnvironment = <environment_vars>
3618 Defines the environment for the prescript, presuming the New envi‐
3620 ronment syntax.
3621 Note that if both +PreEnv and +PreEnvironment are specified, the
3623 +PreEnvironment value is used and the +PreEnv value is ignored.
3624 +PostCmd = <executable>
3630 A vanilla universe job may specify that a script is to be run on the
3632 execute machine after the job exits, and this is called a post‐
3633 script. Definition of this specifically-named attribute causes the
3634 script, identified by path and file name, to be executed. The post‐
3635 script is run if the job exits, but not if the job is evicted. Note
3636 that this definition of a postscript is different from the POST
3637 script described in DAGMan. The postscript is not automatically
3638 transferred with the job, as the main executable is, so it must be
3639 entered into the transfer_input_files list, when file transfer is
3640 enabled.
3641 +PostArgs = <argument_list>
3647 Defines command line arguments for the postscript, presuming the Old
3649 argument syntax.
3650 +PostArguments = <argument_list>
3656 Defines command line arguments for the postscript, presuming the New
3658 argument syntax. An exception to the syntax is that double quotes
3659 must be escaped with a backslash instead of another double quote
3660 mark.
3661 Note that if both +PostArgs and +PostArguments are specified, the
3663 +PostArguments value is used and the +PostArgs value is ignored.
3664 +PostEnv = <environment_vars>
3670 Defines the environment for the postscript, presuming the Old envi‐
3672 ronment syntax.
3673 +PostEnvironment = <environment_vars>
3679 Defines the environment for the postscript, presuming the New envi‐
3681 ronment syntax.
3682 Note that if both +PostEnv and +PostEnvironment are specified, the
3684 +PostEnvironment value is used and the +PostEnv value is ignored.
3685 If any of the prescript or postscript values are not enclosed in double
3691 quotes, they are silently ignored.
3692 Below is an example of the use of starter pre and post scripts:
3694 +PreCmd = "my_pre"
3696 +PreArgs = "pre\"arg1 prea'rg2"
3697 +PreEnv = "one=1;two=\"2\""
3698 +PostCmd = "my_post"
3699 +PostArguments = "post\"arg1 'post''ar g2'"
3700 +PostEnvironment = "one=1 two=\"2\""
3701 For this example PreArgs generates a first argument of pre"a1" and a
3703 second argument of pre'a2 . PostArguments generates a first argument
3704 of post"a1 and a second argument of post'a 2 .
3705 MACROS AND COMMENTS
3707 In addition to commands, the submit description file can contain macros
3709 and comments.
3710 Macros
3712 Parameterless macros in the form of $(macro_name:default initial
3714 value) may be used anywhere in HTCondor submit description files to
3715 provide textual substitution at submit time. Macros can be defined
3716 by lines in the form of
3717 <macro_name> = <string>
3719 Two pre-defined macros are supplied by the submit description file
3721 parser. The $(Cluster) or $(ClusterId) macro supplies the value of
3722 the ClusterId job ClassAd attribute, and the $(Process) or $(Pro‐
3723 cId) macro supplies the value of the ProcId job ClassAd attribute.
3724 These macros are intended to aid in the specification of input/out‐
3725 put files, arguments, etc., for clusters with lots of jobs, and/or
3726 could be used to supply an HTCondor process with its own cluster and
3727 process numbers on the command line.
3728 The $(Node) macro is defined for parallel universe jobs, and is
3730 especially relevant for MPI applications. It is a unique value
3731 assigned for the duration of the job that essentially identifies the
3732 machine (slot) on which a program is executing. Values assigned
3733 start at 0 and increase monotonically. The values are assigned as
3734 the parallel job is about to start.
3735 Recursive definition of macros is permitted. An example of a con‐
3737 struction that works is the following:
3738 foo = bar
3740 foo = snap $(foo)
3741 As a result, foo = snap bar .
3743 Note that both left- and right- recursion works, so
3745 foo = bar
3747 foo = $(foo) snap
3748 has as its result foo = bar snap .
3750 The construction
3752 foo = $(foo) bar
3754 by itself will not work, as it does not have an initial base case.
3756 Mutually recursive constructions such as:
3757 B = bar
3759 C = $(B)
3760 B = $(C) boo
3761 will not work, and will fill memory with expansions.
3763 A default value may be specified, for use if the macro has no defi‐
3765 nition. Consider the example
3766 D = $(E:24)
3768 Where E is not defined within the submit description file, the
3770 default value 24 is used, resulting in
3771 D = 24
3773 This is of limited value, as the scope of macro substitution is the
3775 submit description file. Thus, either the macro is or is not defined
3776 within the submit description file. If the macro is defined, then
3777 the default value is useless. If the macro is not defined, then
3778 there is no point in using it in a submit command.
3779 To use the dollar sign character ( $ ) as a literal, without macro
3781 expansion, use
3782 $(DOLLAR)
3784 In addition to the normal macro, there is also a special kind of
3786 macro called a substitution macro that allows the substitution of a
3787 machine ClassAd attribute value defined on the resource machine
3788 itself (gotten after a match to the machine has been made) into spe‐
3789 cific commands within the submit description file. The substitution
3790 macro is of the form:
3791 $$(attribute)
3793 As this form of the substitution macro is only evaluated within the
3795 context of the machine ClassAd, use of a scope resolution prefix
3796 TARGET. or MY. is not allowed.
3797 A common use of this form of the substitution macro is for the het‐
3799 erogeneous submission of an executable:
3800 executable = povray.$$(OpSys).$$(Arch)
3802 Values for the OpSys and Arch attributes are substituted at match
3804 time for any given resource. This example allows HTCondor to auto‐
3805 matically choose the correct executable for the matched machine.
3806 An extension to the syntax of the substitution macro provides an
3808 alternative string to use if the machine attribute within the sub‐
3809 stitution macro is undefined. The syntax appears as:
3810 $$(attribute:string_if_attribute_undefined)
3812 An example using this extended syntax provides a path name to a
3814 required input file. Since the file can be placed in different loca‐
3815 tions on different machines, the file's path name is given as an
3816 argument to the program.
3817 arguments = $$(input_file_path:/usr/foo)
3819 On the machine, if the attribute input_file_path is not defined,
3821 then the path /usr/foo is used instead.
3822 A further extension to the syntax of the substitution macro allows
3824 the evaluation of a ClassAd expression to define the value. In this
3825 form, the expression may refer to machine attributes by prefacing
3826 them with the TARGET. scope resolution prefix. To place a ClassAd
3827 expression into the substitution macro, square brackets are added to
3828 delimit the expression. The syntax appears as:
3829 $$([ClassAd expression])
3831 An example of a job that uses this syntax may be one that wants to
3833 know how much memory it can use. The application cannot detect this
3834 itself, as it would potentially use all of the memory on a multi-
3835 slot machine. So the job determines the memory per slot, reducing it
3836 by 10% to account for miscellaneous overhead, and passes this as a
3837 command line argument to the application. In the submit description
3838 file will be
3839 arguments = --memory $$([TARGET.Memory * 0.9])
3841 To insert two dollar sign characters ( $$ ) as literals into a Clas‐
3843 sAd string, use
3844 $$(DOLLARDOLLAR)
3846 The environment macro, $ENV, allows the evaluation of an environment
3848 variable to be used in setting a submit description file command.
3849 The syntax used is
3850 $ENV(variable)
3852 An example submit description file command that uses this function‐
3854 ality evaluates the submitter's home directory in order to set the
3855 path and file name of a log file:
3856 log = $ENV(HOME)/jobs/logfile
3858 The environment variable is evaluated when the submit description
3860 file is processed.
3861 The $RANDOM_CHOICE macro allows a random choice to be made from a
3863 given list of parameters at submission time. For an expression, if
3864 some randomness needs to be generated, the macro may appear as
3865 $RANDOM_CHOICE(0,1,2,3,4,5,6)
3867 When evaluated, one of the parameters values will be chosen.
3869 Comments
3875 Blank lines and lines beginning with a pound sign ('#') character
3877 are ignored by the submit description file parser.
3878
3884 While processing the queue command in a submit file or from the command
3885 line, condor_submit will set the values of several automatic submit
3886 variables so that they can be referred to by statements in the submit
3887 file. With the exception of Cluster and Process, if these variables are
3888 set by the submit file, they will not be modified during queue process‐
3889 ing.
3890 ClusterId
3892 Set to the integer value that the ClusterId attribute that the job
3894 ClassAd will have when the job is submitted. All jobs in a single
3895 submit will normally have the same value for the ClusterId . If the
3896 -dry-run argument is specified, The value will be 1.
3897 Cluster
3903 Alternate name for the ClusterId submit variable. Before HTCondor
3905 version 8.4 this was the only name.
3906 ProcId
3912 Set to the integer value that the ProcId attribute of the job Clas‐
3914 sAd will have when the job is submitted. The value will start at 0
3915 and increment by 1 for each job submitted.
3916 Process
3922 Alternate name for the ProcId submit variable. Before HTCondor ver‐
3924 sion 8.4 this was the only name.
3925 Node
3931 For parallel universes, set to the value #pArAlLeLnOdE# or #MpInOdE#
3933 depending on the parallel universe type For other universes it is
3934 set to nothing.
3935 Step
3941 Set to the step value as it varies from 0 to N-1 where N is the num‐
3943 ber provided on the queue argument. This variable changes at the
3944 same rate as ProcId when it changes at all. For submit files that
3945 don't make use of the queue number option, Step will always be 0.
3946 For submit files that don't make use of any of the foreach options,
3947 Step and ProcId will always be the same.
3948 ItemIndex
3954 Set to the index within the item list being processed by the various
3956 queue foreach options. For submit files that don't make use of any
3957 queue foreach list, ItemIndex will always be 0 For submit files that
3958 make use of a slice to select only some items in a foreach list,
3959 ItemIndex will only be set to selected values.
3960 Row
3966 Alternate name for ItemIndex.
3968 Item
3974 when a queue foreach option is used and no variable list is sup‐
3976 plied, this variable will be set to the value of the current item.
3977 The automatic variables below are set before parsing the submit file,
3981 and will not vary during processing unless the submit file itself sets
3982 them.
3983 ARCH
3985 Set to the CPU architecture of the machine running condor_submit .
3987 The value will be the same as the automatic configuration variable
3988 of the same name.
3989 OPSYS
3995 Set to the name of the operating system on the machine running con‐
3997 dor_submit . The value will be the same as the automatic configura‐
3998 tion variable of the same name.
3999 OPSYSANDVER
4005 Set to the name and major version of the operating system on the
4007 machine running condor_submit . The value will be the same as the
4008 automatic configuration variable of the same name.
4009 OPSYSMAJORVER
4015 Set to the major version of the operating system on the machine run‐
4017 ning condor_submit . The value will be the same as the automatic
4018 configuration variable of the same name.
4019 OPSYSVER
4025 Set to the version of the operating system on the machine running
4027 condor_submit . The value will be the same as the automatic configu‐
4028 ration variable of the same name.
4029 SPOOL
4035 Set to the full path of the HTCondor spool directory. The value will
4037 be the same as the automatic configuration variable of the same
4038 name.
4039 IsLinux
4045 Set to true if the operating system of the machine running con‐
4047 dor_submit is a Linux variant. Set to false otherwise.
4048 IsWindows
4054 Set to true if the operating system of the machine running con‐
4056 dor_submit is a Microsoft Windows variant. Set to false otherwise.
4057 SUBMIT_FILE
4063 Set to the full pathname of the submit file being processed by con‐
4065 dor_submit . If submit statements are read from standard input, it
4066 is set to nothing.
4067
4071 condor_submit will exit with a status value of 0 (zero) upon success,
4072 and a non-zero value upon failure.
4073
4075 * Submit Description File Example 1: This example queues three jobs
4076 for execution by HTCondor. The first will be given command line
4077 arguments of 15 and 2000 , and it will write its standard output to
4078 foo.out1 . The second will be given command line arguments of 30 and
4079 2000 , and it will write its standard output to foo.out2 . Simi‐
4080 larly the third will have arguments of 45 and 6000 , and it will use
4081 foo.out3 for its standard output. Standard error output (if any)
4082 from all three programs will appear in foo.error .
4083 ####################
4087 #
4088 # submit description file
4089 # Example 1: queuing multiple jobs with differing
4090 # command line arguments and output files.
4091 #
4092 ####################
4093 Executable = foo
4095 Universe = vanilla
4096 Arguments = 15 2000
4098 Output = foo.out0
4099 Error = foo.err0
4100 Queue
4101 Arguments = 30 2000
4103 Output = foo.out1
4104 Error = foo.err1
4105 Queue
4106 Arguments = 45 6000
4108 Output = foo.out2
4109 Error = foo.err2
4110 Queue
4111 Or you can get the same results as the above submit file by using a
4113 list of arguments with the Queue statement
4114 ####################
4118 #
4119 # submit description file
4120 # Example 1b: queuing multiple jobs with differing
4121 # command line arguments and output files, alternate syntax
4122 #
4123 ####################
4124 Executable = foo
4126 Universe = vanilla
4127 # generate different output and error filenames for each process
4129 Output = foo.out$(Process)
4130 Error = foo.err$(Process)
4131 Queue Arguments From (
4133 15 2000
4134 30 2000
4135 45 6000
4136 )
4137 * Submit Description File Example 2: This submit description file
4141 example queues 150 runs of program foo which must have been compiled
4142 and linked for an Intel x86 processor running RHEL 3. HTCondor will
4143 not attempt to run the processes on machines which have less than 32
4144 Megabytes of physical memory, and it will run them on machines which
4145 have at least 64 Megabytes, if such machines are available. Stdin,
4146 stdout, and stderr will refer to in.0 , out.0 , and err.0 for the
4147 first run of this program (process 0). Stdin, stdout, and stderr
4148 will refer to in.1 , out.1 , and err.1 for process 1, and so
4149 forth. A log file containing entries about where and when HTCondor
4150 runs, takes checkpoints, and migrates processes in this cluster will
4151 be written into file foo.log .
4152 ####################
4156 #
4157 # Example 2: Show off some fancy features including
4158 # use of pre-defined macros and logging.
4159 #
4160 ####################
4161 Executable = foo
4163 Universe = standard
4164 Requirements = OpSys == "LINUX" && Arch =="INTEL"
4165 Rank = Memory >= 64
4166 Request_Memory = 32 Mb
4167 Image_Size = 28 Mb
4168 Error = err.$(Process)
4170 Input = in.$(Process)
4171 Output = out.$(Process)
4172 Log = foo.log
4173 Queue 150
4174 * Submit Description File Example 3: This example targets the
4178 /bin/sleep program to run only on a platform running a RHEL 6 oper‐
4179 ating system. The example presumes that the pool contains machines
4180 running more than one version of Linux, and this job needs the par‐
4181 ticular operating system to run correctly.
4182 ####################
4186 #
4187 # Example 3: Run on a RedHat 6 machine
4188 #
4189 ####################
4190 Universe = vanilla
4191 Executable = /bin/sleep
4192 Arguments = 30
4193 Requirements = (OpSysAndVer == "RedHat6")
4194 Error = err.$(Process)
4196 Input = in.$(Process)
4197 Output = out.$(Process)
4198 Log = sleep.log
4199 Queue
4200 * Command Line example: The following command uses the -append
4204 option to add two commands before the job(s) is queued. A log file
4205 and an error log file are specified. The submit description file is
4206 unchanged.
4207 condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
4209 Note that each of the added commands is contained within quote marks
4211 because there are space characters within the command.
4212 * periodic_remove example: A job should be removed from the queue,
4216 if the total suspension time of the job is more than half of the run
4217 time of the job.
4218 Including the command
4220 periodic_remove = CumulativeSuspensionTime >
4222 ((RemoteWallClockTime - CumulativeSuspensionTime)
4223 / 2.0)
4224 in the submit description file causes this to happen.
4226
4230 * For security reasons, HTCondor will refuse to run any jobs submit‐
4231 ted by user root (UID = 0) or by a user whose default group is group
4232 wheel (GID = 0). Jobs submitted by user root or a user with a
4233 default group of wheel will appear to sit forever in the queue in an
4234 idle state.
4235 * All path names specified in the submit description file must be
4239 less than 256 characters in length, and command line arguments must
4240 be less than 4096 characters in length; otherwise, condor_submit
4241 gives a warning message but the jobs will not execute properly.
4242 * Somewhat understandably, behavior gets bizarre if the user makes
4246 the mistake of requesting multiple HTCondor jobs to write to the
4247 same file, and/or if the user alters any files that need to be
4248 accessed by an HTCondor job which is still in the queue. For exam‐
4249 ple, the compressing of data or output files before an HTCondor job
4250 has completed is a common mistake.
4251 * To disable checkpointing for Standard Universe jobs, include the
4255 line:
4256 +WantCheckpoint = False
4258 in the submit description file before the queue command(s).
4260
4262 HTCondor User Manual
4263
4265 Center for High Throughput Computing, University of Wisconsin-Madison
4266
4268 Copyright (C) 1990-2018 Center for High Throughput Computing, Computer
4269 Sciences Department, University of Wisconsin-Madison, Madison, WI. All
4270 Rights Reserved. Licensed under the Apache License, Version 2.0.
4271 date just-man-pages/condor_submit(1)