1condor_submit(1) General Commands Manual condor_submit(1)
2
6 condor_submitQueue jobs for execution under HTCondor
7
9 condor_submit[-terse] [-verbose] [-unused] [-file submit_file] [-name
10 schedd_name] [-remote schedd_name] [-addr <ip:port>] [-pool pool_name]
11 [-disable] [-password passphrase] [-debug] [-appendcommand...] [-batch-
12 name batch_name] [-spool] [-dump filename] [-interactive] [-allow-crlf-
13 script] [-dry-run] [-maxjobs number-of-jobs] [-single-cluster] [-stm
14 method] [<submit-variable>=<value>] [submit description file] [-queue
15 queue_arguments]
16
18 condor_submitis the program for submitting jobs for execution under
19 HTCondor. condor_submitrequires one or more submit description commands
20 to direct the queuing of jobs. These commands may come from a file,
21 standard input, the command line, or from some combination of these.
22 One submit description may contain specifications for the queuing of
23 many HTCondor jobs at once. A single invocation of condor_submitmay
24 cause one or more clusters. A cluster is a set of jobs specified in the
25 submit description between queuecommands for which the executable is
26 not changed. It is advantageous to submit multiple jobs as a single
27 cluster because:
28 * Much less memory is used by the scheduler to hold the same number
30 of jobs.
31 * Only one copy of the checkpoint file is needed to represent all
33 jobs in a cluster until they begin execution.
34 * There is much less overhead involved for HTCondor to start the
36 next job in a cluster than for HTCondor to start a new cluster. This
37 can make a big difference when submitting lots of short jobs.
38 Multiple clusters may be specified within a single submit description.
40 Each cluster must specify a single executable.
41 The job ClassAd attribute ClusterIdidentifies a cluster.
43 The submit description fileargument is the path and file name of the
45 submit description file. If this optional argument is the dash charac‐
46 ter (-), then the commands are taken from standard input. If -is speci‐
47 fied for the submit description file, -verboseis implied; this can be
48 overridden by specifying -terse.
49 If no submit discription fileargument is given, and no -queueargument
51 is given, commands are taken automatically from standard input.
52 Note that submission of jobs from a Windows machine requires a stashed
54 password to allow HTCondor to impersonate the user submitting the job.
55 To stash a password, use the condor_store_credcommand. See the manual
56 page for details.
57 For lengthy lines within the submit description file, the backslash ( \
59 ) is a line continuation character. Placing the backslash at the end of
60 a line causes the current line's command to be continued with the next
61 line of the file. Submit description files may contain comments. A com‐
62 ment is any line beginning with a pound character ( # ).
63
65 -terse
66 Terse output - display JobId ranges only.
68 -verbose
74 Verbose output - display the created job ClassAd
76 -unused
82 As a default, causes no warnings to be issued about user-defined
84 macros not being used within the submit description file. The mean‐
85 ing reverses (toggles) when the configuration variable
86 WARN_ON_UNUSED_SUBMIT_FILE_MACROSis set to the non default value of
87 False. Printing the warnings can help identify spelling errors of
88 submit description file commands. The warnings are sent to stderr.
89 -file submit_file
95 Use submit_fileas the submit discription file. This is equivalent to
97 providing submit_fileas an argument without the preceeding -file.
98 -name schedd_name
104 Submit to the specified condor_schedd. Use this option to submit to
106 a condor_scheddother than the default local one. schedd_nameis the
107 value of the NameClassAd attribute on the machine where the con‐
108 dor_schedddaemon runs.
109 -remote schedd_name
115 Submit to the specified condor_schedd, spooling all required input
117 files over the network connection. schedd_nameis the value of the
118 NameClassAd attribute on the machine where the condor_schedddaemon
119 runs. This option is equivalent to using both -nameand -spool.
120 -addr <ip:port>
126 Submit to the condor_scheddat the IP address and port given by the
128 sinful stringargument <ip:port>.
129 -pool pool_name
135 Look in the specified pool for the condor_scheddto submit to. This
137 option is used with -nameor -remote.
138 -disable
144 Disable file permission checks when submitting a job for read per‐
146 missions on all input files, such as those defined by commands
147 inputand transfer_input_files, as well as write permission to output
148 files, such as a log file defined by logand output files defined
149 with outputor transfer_output_files.
150 -password passphrase
156 Specify a password to the MyProxyserver.
158 -debug
164 Cause debugging information to be sent to stderr, based on the value
166 of the configuration variable TOOL_DEBUG.
167 -append command
173 Augment the commands in the submit description file with the given
175 command. This command will be considered to immediately precede the
176 queuecommand within the submit description file, and come after all
177 other previous commands. If the commandspecifies a queuecommand, as
178 in the example
179 condor_submit mysubmitfile -append "queue input in A, B, C"
181 then the entire -appendcommand line option and its arguments are
183 converted to
184 condor_submit mysubmitfile -queue input in A, B, C
186 The submit description file is not modified. Multiple commands are
188 specified by using the -appendoption multiple times. Each new com‐
189 mand is given in a separate -appendoption. Commands with spaces in
190 them will need to be enclosed in double quote marks.
191 -batch-name batch_name
197 Set the batch name for this submit. The batch name is displayed by
199 condor_q-batch. It is intended for use by users to give meaningful
200 names to their jobs and to influence how condor_qgroups jobs for
201 display. Use of this argument takes precedence over a batch name
202 specified in the submit description file itself.
203 -spool
209 Spool all required input files, job event log, and proxy over the
211 connection to the condor_schedd. After submission, modify local
212 copies of the files without affecting your jobs. Any output files
213 for completed jobs need to be retrieved with condor_transfer_data.
214 -dump filename
220 Sends all ClassAds to the specified file, instead of to the con‐
222 dor_schedd.
223 -interactive
229 Indicates that the user wants to run an interactive shell on an exe‐
231 cute machine in the pool. This is equivalent to creating a submit
232 description file of a vanilla universe sleep job, and then running
233 condor_ssh_to_jobby hand. Without any additional arguments, con‐
234 dor_submitwith the -interactive flag creates a dummy vanilla uni‐
235 verse job that sleeps, submits it to the local scheduler, waits for
236 the job to run, and then launches condor_ssh_to_jobto run a shell.
237 If the user would like to run the shell on a machine that matches a
238 particular requirementsexpression, the submit description file is
239 specified, and it will contain the expression. Note that all policy
240 expressions specified in the submit description file are honored,
241 but any executableor universecommands are overwritten to be sleep
242 and vanilla. The job ClassAd attribute InteractiveJobis set to
243 Trueto identify interactive jobs for condor_startdpolicy usage.
244 -allow-crlf-script
250 Changes the check for an invalid line ending on the executable
252 script's #!line from an ERROR to a WARNING. The #!line will be
253 ignored by Windows, so it won't matter if it is invalid; but Unix
254 and Linux will not run a script that has a Windows/DOS line ending
255 on the first line of the script. So condor_submitwill not allow such
256 a script to be submitted as the job's executable unless this option
257 is supplied.
258 -dry-run file
264 Parse the submit description file, sending the resulting job ClassAd
266 to the file given by file, but do not submit the job(s). This per‐
267 mits observation of the job specification, and it facilitates debug‐
268 ging the submit description file contents. If fileis -, the output
269 is written to stdout.
270 -maxjobs number-of-jobs
276 If the total number of jobs specified by the submit description file
278 is more than the integer value given by number-of-jobs, then no jobs
279 are submitted for execution and an error message is generated. A 0
280 or negative value for the number-of-jobscauses no limit to be
281 imposed.
282 -single-cluster
288 If the jobs specified by the submit description file causes more
290 than a single cluster value to be assigned, then no jobs are submit‐
291 ted for execution and an error message is generated.
292 -stm method
298 Specify the method use to move a sandbox into HTCondor. methodis one
300 of stm_use_schedd_onlyor stm_use_transferd.
301 <submit-variable>=<value>
307 Defines a submit command or submit variable with a value, and parses
309 it as if it was placed at the beginning of the submit description
310 file. The submit description file is not changed. To correctly parse
311 the condor_submitcommand line, this option must be specified without
312 white space characters before and after the equals sign (=), or the
313 entire option must be surrounded by double quote marks.
314 -queue queue_arguments
320 A command line specification of how many jobs to queue, which is
322 only permitted if the submit description file does not have a
323 queuecommand. The queue_argumentsare the same as may be within a
324 submit description file. The parsing of the queue_argumentsfinishes
325 at the end of the line or when a dash character (-) is encountered.
326 Therefore, its best placement within the command line will be at the
327 end of the command line.
328 On a Unix command line, the shell expands file globs before parsing
330 occurs.
331
337 Note: more information on submitting HTCondor jobs can be found here: .
338 As of version 8.5.6, the condor_submitlanguage supports multi-line val‐
340 ues in commands. The syntax is the same as the configuration language
341 (see more details here: ).
342 Each submit description file describes one or more clusters of jobs to
344 be placed in the HTCondor execution pool. All jobs in a cluster must
345 share the same executable, but they may have different input and output
346 files, and different program arguments. The submit description file is
347 generally the last command-line argument to condor_submit. If the sub‐
348 mit description file argument is omitted, condor_submitwill read the
349 submit description from standard input.
350 The submit description file must contain at least one executablecommand
352 and at least one queuecommand. All of the other commands have default
353 actions.
354 Note that a submit file that contains more than one executable command
356 will produce multiple clusters when submitted. This is not generally
357 recommended, and is not allowed for submit files that are run as DAG
358 node jobs by condor_dagman.
359 The commands which can appear in the submit description file are numer‐
361 ous. They are listed here in alphabetical order by category.
362 BASIC COMMANDS
364 arguments = <argument_list>
372 List of arguments to be supplied to the executable as part of the
374 command line.
375 In the javauniverse, the first argument must be the name of the
377 class containing main.
378 There are two permissible formats for specifying arguments, identi‐
380 fied as the old syntax and the new syntax. The old syntax supports
381 white space characters within arguments only in special circum‐
382 stances; when used, the command line arguments are represented in
383 the job ClassAd attribute Args. The new syntax supports uniform
384 quoting of white space characters within arguments; when used, the
385 command line arguments are represented in the job ClassAd attribute
386 Arguments.
387 Old Syntax
389 In the old syntax, individual command line arguments are delimited
391 (separated) by space characters. To allow a double quote mark in an
392 argument, it is escaped with a backslash; that is, the two character
393 sequence \" becomes a single double quote mark within an argument.
394 Further interpretation of the argument string differs depending on
396 the operating system. On Windows, the entire argument string is
397 passed verbatim (other than the backslash in front of double quote
398 marks) to the Windows application. Most Windows applications will
399 allow spaces within an argument value by surrounding the argument
400 with double quotes marks. In all other cases, there is no further
401 interpretation of the arguments.
402 Example:
404 arguments = one \"two\" 'three'
408 Produces in Unix vanilla universe:
410 argument 1: one
414 argument 2: "two"
415 argument 3: 'three'
416 New Syntax
418 Here are the rules for using the new syntax:
420 1. The entire string representing the command line arguments is
424 surrounded by double quote marks. This permits the white space
425 characters of spaces and tabs to potentially be embedded within a
426 single argument. Putting the double quote mark within the argu‐
427 ments is accomplished by escaping it with another double quote
428 mark.
429 2. The white space characters of spaces or tabs delimit argu‐
433 ments.
434 3. To embed white space characters of spaces or tabs within a
438 single argument, surround the entire argument with single quote
439 marks.
440 4. To insert a literal single quote mark, escape it within an
444 argument already delimited by single quote marks by adding
445 another single quote mark.
446 Example:
450 arguments = "3 simple arguments"
452 Produces:
454 argument 1: 3
456 argument 2: simple
457 argument 3: arguments
458 Another example:
460 arguments = "one 'two with spaces' 3"
462 Produces:
464 argument 1: one
466 argument 2: two with spaces
467 argument 3: 3
468 And yet another example:
470 arguments = "one ""two"" 'spacey ”quoted” argument'"
472 Produces:
474 argument 1: one
476 argument 2: "two"
477 argument 3: spacey 'quoted' argument
478 Notice that in the new syntax, the backslash has no special meaning.
480 This is for the convenience of Windows users.
481 environment = <parameter_list>
487 List of environment variables.
489 There are two different formats for specifying the environment vari‐
491 ables: the old format and the new format. The old format is retained
492 for backward-compatibility. It suffers from a platform-dependent
493 syntax and the inability to insert some special characters into the
494 environment.
495 The new syntax for specifying environment values:
497 1. Put double quote marks around the entire argument string. This
501 distinguishes the new syntax from the old. The old syntax does
502 not have double quote marks around it. Any literal double quote
503 marks within the string must be escaped by repeating the double
504 quote mark.
505 2. Each environment entry has the form
509 <name>=<value>
513 3. Use white space (space or tab characters) to separate environ‐
517 ment entries.
518 4. To put any white space in an environment entry, surround the
522 space and as much of the surrounding entry as desired with single
523 quote marks.
524 5. To insert a literal single quote mark, repeat the single quote
528 mark anywhere inside of a section surrounded by single quote
529 marks.
530 Example:
534 environment = "one=1 two=""2"" three='spacey ”quoted”
538 value'"
539 Produces the following environment entries:
541 one=1
545 two="2"
546 three=spacey 'quoted' value
547 Under the old syntax, there are no double quote marks surrounding
549 the environment specification. Each environment entry remains of the
550 form
551 <name>=<value>
553 Under Unix, list multiple environment entries by separating them
555 with a semicolon (;). Under Windows, separate multiple entries with
556 a vertical bar (|). There is no way to insert a literal semicolon
557 under Unix or a literal vertical bar under Windows. Note that spaces
558 are accepted, but rarely desired, characters within parameter names
559 and values, because they are treated as literal characters, not sep‐
560 arators or ignored white space. Place spaces within the parameter
561 list only if required.
562 A Unix example:
564 environment = one=1;two=2;three="quotes have no 'special' meaning"
568 This produces the following:
570 one=1
574 two=2
575 three="quotes have no 'special' meaning"
576 If the environment is set with the environmentcommand andgetenvis
578 also set to true, values specified with environmentoverride values
579 in the submitter's environment (regardless of the order of the envi‐
580 ronmentand getenvcommands).
581 error = <pathname>
587 A path and file name used by HTCondor to capture any error messages
589 the program would normally write to the screen (that is, this file
590 becomes stderr). A path is given with respect to the file system of
591 the machine on which the job is submitted. The file is written (by
592 the job) in the remote scratch directory of the machine where the
593 job is executed. When the job exits, the resulting file is trans‐
594 ferred back to the machine where the job was submitted, and the path
595 is utilized for file placement. If not specified, the default value
596 of /dev/nullis used for submission to a Unix machine. If not speci‐
597 fied, error messages are ignored for submission to a Windows
598 machine. More than one job should not use the same error file, since
599 this will cause one job to overwrite the errors of another. If
600 HTCondor detects that the error and output files for a job are the
601 same, it will run the job such that the output and error data is
602 merged.
603 executable = <pathname>
609 An optional path and a required file name of the executable file for
611 this job cluster. Only one executablecommand within a submit
612 description file is guaranteed to work properly. More than one often
613 works.
614 If no path or a relative path is used, then the executable file is
616 presumed to be relative to the current working directory of the user
617 as the condor_submitcommand is issued.
618 If submitting into the standard universe, then the named executable
620 must have been re-linked with the HTCondor libraries (such as via
621 the condor_compilecommand). If submitting into the vanilla universe
622 (the default), then the named executable need not be re-linked and
623 can be any process which can run in the background (shell scripts
624 work fine as well). If submitting into the Java universe, then the
625 argument must be a compiled .classfile.
626 getenv = <True | False>
632 If getenvis set to True, then condor_submitwill copy all of the
634 user's current shell environment variables at the time of job sub‐
635 mission into the job ClassAd. The job will therefore execute with
636 the same set of environment variables that the user had at submit
637 time. Defaults to False.
638 If the environment is set with the environmentcommand andgetenvis
640 also set to true, values specified with environmentoverride values
641 in the submitter's environment (regardless of the order of the envi‐
642 ronmentand getenvcommands).
643 input = <pathname>
649 HTCondor assumes that its jobs are long-running, and that the user
651 will not wait at the terminal for their completion. Because of this,
652 the standard files which normally access the terminal, (stdin, std‐
653 out, and stderr), must refer to files. Thus, the file name specified
654 with inputshould contain any keyboard input the program requires
655 (that is, this file becomes stdin). A path is given with respect to
656 the file system of the machine on which the job is submitted. The
657 file is transferred before execution to the remote scratch directory
658 of the machine where the job is executed. If not specified, the
659 default value of /dev/nullis used for submission to a Unix machine.
660 If not specified, input is ignored for submission to a Windows
661 machine. For grid universe jobs, inputmay be a URL that the Globus
662 tool globus_url_copyunderstands.
663 Note that this command does notrefer to the command-line arguments
665 of the program. The command-line arguments are specified by the
666 argumentscommand.
667 log = <pathname>
673 Use logto specify a file name where HTCondor will write a log file
675 of what is happening with this job cluster, called a job event log.
676 For example, HTCondor will place a log entry into this file when and
677 where the job begins running, when the job produces a checkpoint, or
678 moves (migrates) to another machine, and when the job completes.
679 Most users find specifying a logfile to be handy; its use is recom‐
680 mended. If no logentry is specified, HTCondor does not create a log
681 for this cluster. If a relative path is specified, it is relative to
682 the current working directory as the job is submitted or the direc‐
683 tory specified by submit command initialdiron the submit machine.
684 log_xml = <True | False>
690 If log_xmlis True, then the job event log file will be written in
692 ClassAd XML. If not specified, XML is not used. Note that the file
693 is an XML fragment; it is missing the file header and footer. Do not
694 mix XML and non-XML within a single file. If multiple jobs write to
695 a single job event log file, ensure that all of the jobs specify
696 this option in the same way.
697 notification = <Always | Complete | Error | Never>
703 Owners of HTCondor jobs are notified by e-mail when certain events
705 occur. If defined by Always, the owner will be notified whenever the
706 job produces a checkpoint, as well as when the job completes. If
707 defined by Complete, the owner will be notified when the job termi‐
708 nates. If defined by Error, the owner will only be notified if the
709 job terminates abnormally, (as defined by JobSuccessExitCode, if
710 defined) or if the job is placed on hold because of a failure, and
711 not by user request. If defined by Never(the default), the owner
712 will not receive e-mail, regardless to what happens to the job. The
713 HTCondor User's manual documents statistics included in the e-mail.
714 notify_user = <email-address>
720 Used to specify the e-mail address to use when HTCondor sends e-mail
722 about a job. If not specified, HTCondor defaults to using the e-mail
723 address defined by
724 job-owner@UID_DOMAIN
726 where the configuration variable UID_DOMAINis specified by the
728 HTCondor site administrator. If UID_DOMAINhas not been specified,
729 HTCondor sends the e-mail to:
730 job-owner@submit-machine-name
732 output = <pathname>
738 The outputfile captures any information the program would ordinarily
740 write to the screen (that is, this file becomes stdout). A path is
741 given with respect to the file system of the machine on which the
742 job is submitted. The file is written (by the job) in the remote
743 scratch directory of the machine where the job is executed. When the
744 job exits, the resulting file is transferred back to the machine
745 where the job was submitted, and the path is utilized for file
746 placement. If not specified, the default value of /dev/nullis used
747 for submission to a Unix machine. If not specified, output is
748 ignored for submission to a Windows machine. Multiple jobs should
749 not use the same output file, since this will cause one job to over‐
750 write the output of another. If HTCondor detects that the error and
751 output files for a job are the same, it will run the job such that
752 the output and error data is merged.
753 Note that if a program explicitly opens and writes to a file, that
755 file should notbe specified as the outputfile.
756 priority = <integer>
762 An HTCondor job priority can be any integer, with 0 being the
764 default. Jobs with higher numerical priority will run before jobs
765 with lower numerical priority. Note that this priority is on a per
766 user basis. One user with many jobs may use this command to order
767 his/her own jobs, and this will have no effect on whether or not
768 these jobs will run ahead of another user's jobs.
769 Note that the priority setting in an HTCondor submit file will be
771 overridden by condor_dagmanif the submit file is used for a node in
772 a DAG, and the priority of the node within the DAG is non-zero (see
773 for more details).
774 queue [<int expr>]
780 Places zero or more copies of the job into the HTCondor queue.
782 queue
786 [<int expr>] [<varname>] in[slice] <list of items>Places zero or
788 more copies of the job in the queue based on items in a <list of
789 items>
790 queue
794 [<int expr>] [<varname>] matching[files | dirs] [slice] <list of
796 items with file globbing>] Places zero or more copies of the job in
797 the queue based on files that match a <list of items with file glob‐
798 bing>
799 queue
803 [<int expr>] [<list of varnames>] from[slice] <file name> | <list of
805 items>] Places zero or more copies of the job in the queue based on
806 lines from the submit file or from <file name>
807 The optional argument <int expr>specifies how many times to repeat
809 the job submission for a given set of arguments. It may be an inte‐
810 ger or an expression that evaluates to an integer, and it defaults
811 to 1. All but the first form of this command are various ways of
812 specifying a list of items. When these forms are used <int expr>jobs
813 will be queued for each item in the list. The in, matchingand
814 fromkeyword indicates how the list will be specified.
815 * inThe list of items is an explicit comma and/or space separated
817 <list of items>. If the <list of items>begins with an open paren,
818 and the close paren is not on the same line as the open, then the
819 list continues until a line that begins with a close paren is
820 read from the submit file.
821 * matchingEach item in the <list of items with file globbing>will
823 be matched against the names of files and directories relative to
824 the current directory, the set of matching names is the resulting
825 list of items.
826 * filesOnly filenames will matched.
828 * dirsOnly directory names will be matched.
830 * from<file name> | <list of items>Each line from <file name>or
832 <list of items>is a single item, this allows for multiple vari‐
833 ables to be set for each item. Lines from <file name>or <list of
834 items>will be split on comma and/or space until there are values
835 for each of the variables specified in <list of varnames>. The
836 last variable will contain the remainder of the line. When the
837 <list of items>form is used, the list continues until the first
838 line that begins with a close paren, and lines beginning with
839 pound sign ('#') will be skipped. When using the <file name>form,
840 if the <file name>ends with |, then it will be executed as a
841 script whatever the script writes to stdoutwill be the list of
842 items. The optional argument <varname>or <list of varnames>is the
843 name or names of of variables that will be set to the value of
844 the current item when queuing the job. If no <varname>is speci‐
845 fied the variable ITEM will be used. Leading and trailing white‐
846 space be trimmed. The optional argument <slice>is a python style
847 slice selecting only some of the items in the list of items. Neg‐
848 ative step values are not supported.
849 A submit file may contain more than one queuestatement, and if
851 desired, any commands may be placed between subsequent queuecom‐
852 mands, such as new input, output, error, initialdir, or argu‐
853 mentscommands. This is handy when submitting multiple runs into one
854 cluster with one submit description file.
855 universe = <vanilla | standard | scheduler | local | grid | java | vm |
861 parallel | docker>
862 Specifies which HTCondor universe to use when running this job. The
864 HTCondor universe specifies an HTCondor execution environment.
865 The vanillauniverse is the default (except where the configuration
867 variable DEFAULT_UNIVERSEdefines it otherwise), and is an execution
868 environment for jobs which do not use HTCondor's mechanisms for tak‐
869 ing checkpoints; these are ones that have not been linked with the
870 HTCondor libraries. Use the vanillauniverse to submit shell scripts
871 to HTCondor.
872 The standarduniverse tells HTCondor that this job has been re-linked
874 via condor_compilewith the HTCondor libraries and therefore supports
875 taking checkpoints and remote system calls.
876 The scheduleruniverse is for a job that is to run on the machine
878 where the job is submitted. This universe is intended for a job that
879 acts as a metascheduler and will not be preempted.
880 The localuniverse is for a job that is to run on the machine where
882 the job is submitted. This universe runs the job immediately and
883 will not preempt the job.
884 The griduniverse forwards the job to an external job management sys‐
886 tem. Further specification of the griduniverse is done with the
887 grid_resourcecommand.
888 The javauniverse is for programs written to the Java Virtual
890 Machine.
891 The vmuniverse facilitates the execution of a virtual machine.
893 The paralleluniverse is for parallel jobs (e.g. MPI) that require
895 multiple machines in order to run.
896 The dockeruniverse runs a docker container as an HTCondor job.
898 COMMANDS FOR MATCHMAKING
904 rank = <ClassAd Float Expression>
912 A ClassAd Floating-Point expression that states how to rank machines
914 which have already met the requirements expression. Essentially,
915 rank expresses preference. A higher numeric value equals better
916 rank. HTCondor will give the job the machine with the highest rank.
917 For example,
918 request_memory = max({60, Target.TotalSlotMemory})
920 rank = Memory
921 asks HTCondor to find all available machines with more than 60
923 megabytes of memory and give to the job the machine with the most
924 amount of memory. The HTCondor User's Manual contains complete
925 information on the syntax and available attributes that can be used
926 in the ClassAd expression.
927 request_cpus = <num-cpus>
933 A requested number of CPUs (cores). If not specified, the number
935 requested will be 1. If specified, the expression
936 && (RequestCpus <= Target.Cpus)
938 is appended to the requirementsexpression for the job.
940 For pools that enable dynamic condor_startdprovisioning, specifies
942 the minimum number of CPUs requested for this job, resulting in a
943 dynamic slot being created with this many cores.
944 request_disk = <quantity>
950 The requested amount of disk space in KiB requested for this job. If
952 not specified, it will be set to the job ClassAd attribute
953 DiskUsage. The expression
954 && (RequestDisk <= Target.Disk)
956 is appended to the requirementsexpression for the job.
958 For pools that enable dynamic condor_startdprovisioning, a dynamic
960 slot will be created with at least this much disk space.
961 Characters may be appended to a numerical value to indicate units.
963 Kor KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers
964 of bytes. Gor GBindicates GiB, numbers of bytes. Tor TBindicates
965 TiB, numbers of bytes.
966 request_memory = <quantity>
972 The required amount of memory in MiB that this job needs to avoid
974 excessive swapping. If not specified and the submit command vm_memo‐
975 ryis specified, then the value specified for vm_memorydefines
976 request_memory. If neither request_memorynor vm_memoryis specified,
977 the value is set by the configuration variable JOB_DEFAULT_REQUEST‐
978 MEMORY. The actual amount of memory used by a job is represented by
979 the job ClassAd attribute MemoryUsage.
980 For pools that enable dynamic condor_startdprovisioning, a dynamic
982 slot will be created with at least this much RAM.
983 The expression
985 && (RequestMemory <= Target.Memory)
987 is appended to the requirementsexpression for the job.
989 Characters may be appended to a numerical value to indicate units.
991 Kor KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers
992 of bytes. Gor GBindicates GiB, numbers of bytes. Tor TBindicates
993 TiB, numbers of bytes.
994 request_<name> = <quantity>
1000 The required amount of the custom machine resource identified by
1002 <name>that this job needs. The custom machine resource is defined in
1003 the machine's configuration. Machines that have available GPUs will
1004 define <name>to be GPUs.
1005 requirements = <ClassAd Boolean Expression>
1011 The requirements command is a boolean ClassAd expression which uses
1013 C-like operators. In order for any job in this cluster to run on a
1014 given machine, this requirements expression must evaluate to true on
1015 the given machine.
1016 For scheduler and local universe jobs, the requirements expression
1018 is evaluated against the SchedulerClassAd which represents the the
1019 condor_schedddaemon running on the submit machine, rather than a
1020 remote machine. Like all commands in the submit description file, if
1021 multiple requirements commands are present, all but the last one are
1022 ignored. By default, condor_submitappends the following clauses to
1023 the requirements expression:
1024 1. Arch and OpSys are set equal to the Arch and OpSys of the sub‐
1026 mit machine. In other words: unless you request otherwise, HTCon‐
1027 dor will give your job machines with the same architecture and
1028 operating system version as the machine running condor_submit.
1029 2. Cpus >= RequestCpus, if the job ClassAd attribute RequestCpu‐
1031 sis defined.
1032 3. Disk >= RequestDisk, if the job ClassAd attribute Request‐
1034 Diskis defined. Otherwise, Disk >= DiskUsage is appended to the
1035 requirements. The DiskUsageattribute is initialized to the size
1036 of the executable plus the size of any files specified in a
1037 transfer_input_filescommand. It exists to ensure there is enough
1038 disk space on the target machine for HTCondor to copy over both
1039 the executable and needed input files. The DiskUsageattribute
1040 represents the maximum amount of total disk space required by the
1041 job in kilobytes. HTCondor automatically updates the DiskUsageat‐
1042 tribute approximately every 20 minutes while the job runs with
1043 the amount of space being used by the job on the execute machine.
1044 4. Memory >= RequestMemory, if the job ClassAd attribute Request‐
1046 Memoryis defined.
1047 5. If Universe is set to Vanilla, FileSystemDomain is set equal
1049 to the submit machine's FileSystemDomain. View the requirements
1050 of a job which has already been submitted (along with everything
1051 else about the job ClassAd) with the command condor_q -l; see the
1052 command reference for condor_qon page . Also, see the HTCondor
1053 Users Manual for complete information on the syntax and available
1054 attributes that can be used in the ClassAd expression.
1055 FILE TRANSFER COMMANDS
1061 dont_encrypt_input_files = < file1,file2,file... >
1069 A comma and/or space separated list of input files that are notto be
1071 network encrypted when transferred with the file transfer mechanism.
1072 Specification of files in this manner overrides configuration that
1073 would use encryption. Each input file must also be in the list given
1074 by transfer_input_files. When a path to an input file or directory
1075 is specified, this specifies the path to the file on the submit
1076 side. A single wild card character (*) may be used in each file
1077 name.
1078 dont_encrypt_output_files = < file1,file2,file... >
1084 A comma and/or space separated list of output files that are notto
1086 be network encrypted when transferred back with the file transfer
1087 mechanism. Specification of files in this manner overrides configu‐
1088 ration that would use encryption. The output file(s) must also
1089 either be in the list given by transfer_output_filesor be discovered
1090 and to be transferred back with the file transfer mechanism. When a
1091 path to an output file or directory is specified, this specifies the
1092 path to the file on the execute side. A single wild card character
1093 (*) may be used in each file name.
1094 encrypt_execute_directory = <True | False>
1100 Defaults to False. If set to True, HTCondor will encrypt the con‐
1102 tents of the remote scratch directory of the machine where the job
1103 is executed. This encryption is transparent to the job itself, but
1104 ensures that files left behind on the local disk of the execute
1105 machine, perhaps due to a system crash, will remain private. In
1106 addition, condor_submitwill append to the job's requirementsexpres‐
1107 sion
1108 && (TARGET.HasEncryptExecuteDirectory)
1110 to ensure the job is matched to a machine that is capable of
1112 encrypting the contents of the execute directory. This support is
1113 limited to Windows platforms that use the NTFS file system and Linux
1114 platforms with the ecryptfs-utilspackage installed.
1115 encrypt_input_files = < file1,file2,file... >
1121 A comma and/or space separated list of input files that are to be
1123 network encrypted when transferred with the file transfer mechanism.
1124 Specification of files in this manner overrides configuration that
1125 would notuse encryption. Each input file must also be in the list
1126 given by transfer_input_files. When a path to an input file or
1127 directory is specified, this specifies the path to the file on the
1128 submit side. A single wild card character (*) may be used in each
1129 file name. The method of encryption utilized will be as agreed upon
1130 in security negotiation; if that negotiation failed, then the file
1131 transfer mechanism must also fail for files to be network encrypted.
1132 encrypt_output_files = < file1,file2,file... >
1138 A comma and/or space separated list of output files that are to be
1140 network encrypted when transferred back with the file transfer mech‐
1141 anism. Specification of files in this manner overrides configuration
1142 that would notuse encryption. The output file(s) must also either be
1143 in the list given by transfer_output_filesor be discovered and to be
1144 transferred back with the file transfer mechanism. When a path to an
1145 output file or directory is specified, this specifies the path to
1146 the file on the execute side. A single wild card character (*) may
1147 be used in each file name. The method of encryption utilized will be
1148 as agreed upon in security negotiation; if that negotiation failed,
1149 then the file transfer mechanism must also fail for files to be net‐
1150 work encrypted.
1151 max_transfer_input_mb = <ClassAd Integer Expression>
1157 This integer expression specifies the maximum allowed total size in
1159 MiB of the input files that are transferred for a job. This expres‐
1160 sion does notapply to grid universe, standard universe, or files
1161 transferred via file transfer plug-ins. The expression may refer to
1162 attributes of the job. The special value -1 indicates no limit. If
1163 not defined, the value set by configuration variable MAX_TRANS‐
1164 FER_INPUT_MBis used. If the observed size of all input files at sub‐
1165 mit time is larger than the limit, the job will be immediately
1166 placed on hold with a HoldReasonCodevalue of 32. If the job passes
1167 this initial test, but the size of the input files increases or the
1168 limit decreases so that the limit is violated, the job will be
1169 placed on hold at the time when the file transfer is attempted.
1170 max_transfer_output_mb = <ClassAd Integer Expression>
1176 This integer expression specifies the maximum allowed total size in
1178 MiB of the output files that are transferred for a job. This expres‐
1179 sion does notapply to grid universe, standard universe, or files
1180 transferred via file transfer plug-ins. The expression may refer to
1181 attributes of the job. The special value -1 indicates no limit. If
1182 not set, the value set by configuration variable MAX_TRANSFER_OUT‐
1183 PUT_MBis used. If the total size of the job's output files to be
1184 transferred is larger than the limit, the job will be placed on hold
1185 with a HoldReasonCodevalue of 33. The output will be transferred up
1186 to the point when the limit is hit, so some files may be fully
1187 transferred, some partially, and some not at all.
1188 output_destination = <destination-URL>
1194 When present, defines a URL that specifies both a plug-in and a des‐
1196 tination for the transfer of the entire output sandbox or a subset
1197 of output files as specified by the submit command transfer_out‐
1198 put_files. The plug-in does the transfer of files, and no files are
1199 sent back to the submit machine. The HTCondor Administrator's manual
1200 has full details.
1201 should_transfer_files = <YES | NO | IF_NEEDED >
1207 The should_transfer_filessetting is used to define if HTCondor
1209 should transfer files to and from the remote machine where the job
1210 runs. The file transfer mechanism is used to run jobs which are not
1211 in the standard universe (and can therefore use remote system calls
1212 for file access) on machines which do not have a shared file system
1213 with the submit machine. should_transfer_filesequal to YESwill cause
1214 HTCondor to always transfer files for the job. NOdisables HTCondor's
1215 file transfer mechanism. IF_NEEDEDwill not transfer files for the
1216 job if it is matched with a resource in the same FileSystemDomainas
1217 the submit machine (and therefore, on a machine with the same shared
1218 file system). If the job is matched with a remote resource in a dif‐
1219 ferent FileSystemDomain, HTCondor will transfer the necessary files.
1220 For more information about this and other settings related to trans‐
1222 ferring files, see the HTCondor User's manual section on the file
1223 transfer mechanism.
1224 Note that should_transfer_filesis not supported for jobs submitted
1226 to the grid universe.
1227 skip_filechecks = <True | False>
1233 When True, file permission checks for the submitted job are dis‐
1235 abled. When False, file permissions are checked; this is the behav‐
1236 ior when this command is not present in the submit description file.
1237 File permissions are checked for read permissions on all input
1238 files, such as those defined by commands inputand trans‐
1239 fer_input_files, and for write permission to output files, such as a
1240 log file defined by logand output files defined with outputor trans‐
1241 fer_output_files.
1242 stream_error = <True | False>
1248 If True, then stderris streamed back to the machine from which the
1250 job was submitted. If False, stderris stored locally and transferred
1251 back when the job completes. This command is ignored if the job
1252 ClassAd attribute TransferErris False. The default value is False.
1253 This command must be used in conjunction with error, otherwise
1254 stderrwill sent to /dev/nullon Unix machines and ignored on Windows
1255 machines.
1256 stream_input = <True | False>
1262 If True, then stdinis streamed from the machine on which the job was
1264 submitted. The default value is False. The command is only relevant
1265 for jobs submitted to the vanilla or java universes, and it is
1266 ignored by the grid universe. This command must be used in conjunc‐
1267 tion with input, otherwise stdinwill be /dev/nullon Unix machines
1268 and ignored on Windows machines.
1269 stream_output = <True | False>
1275 If True, then stdoutis streamed back to the machine from which the
1277 job was submitted. If False, stdoutis stored locally and transferred
1278 back when the job completes. This command is ignored if the job
1279 ClassAd attribute TransferOutis False. The default value is False.
1280 This command must be used in conjunction with output, otherwise std‐
1281 outwill sent to /dev/nullon Unix machines and ignored on Windows
1282 machines.
1283 transfer_executable = <True | False>
1289 This command is applicable to jobs submitted to the grid and vanilla
1291 universes. If transfer_executableis set to False, then HTCondor
1292 looks for the executable on the remote machine, and does not trans‐
1293 fer the executable over. This is useful for an already pre-staged
1294 executable; HTCondor behaves more like rsh. The default value is
1295 True.
1296 transfer_input_files = < file1,file2,file... >
1302 A comma-delimited list of all the files and directories to be trans‐
1304 ferred into the working directory for the job, before the job is
1305 started. By default, the file specified in the executablecommand and
1306 any file specified in the inputcommand (for example, stdin) are
1307 transferred.
1308 When a path to an input file or directory is specified, this speci‐
1310 fies the path to the file on the submit side. The file is placed in
1311 the job's temporary scratch directory on the execute side, and it is
1312 named using the base name of the original path. For example,
1313 /path/to/input_filebecomes input_filein the job's scratch directory.
1314 A directory may be specified by appending the forward slash charac‐
1316 ter (/) as a trailing path separator. This syntax is used for both
1317 Windows and Linux submit hosts. A directory example using a trailing
1318 path separator is input_data/. When a directory is specified with
1319 the trailing path separator, the contents of the directory are
1320 transferred, but the directory itself is not transferred. It is as
1321 if each of the items within the directory were listed in the trans‐
1322 fer list. When there is no trailing path separator, the directory is
1323 transferred, its contents are transferred, and these contents are
1324 placed inside the transferred directory.
1325 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1327 tories is not currently supported.
1328 Symbolic links to files are transferred as the files they point to.
1330 Transfer of symbolic links to directories is not currently sup‐
1331 ported.
1332 For vanilla and vm universe jobs only, a file may be specified by
1334 giving a URL, instead of a file name. The implementation for URL
1335 transfers requires both configuration and available plug-in.
1336 transfer_output_files = < file1,file2,file... >
1342 This command forms an explicit list of output files and directories
1344 to be transferred back from the temporary working directory on the
1345 execute machine to the submit machine. If there are multiple files,
1346 they must be delimited with commas. Setting transfer_output_filesto
1347 the empty string ( "" ) means that no files are to be transferred.
1348 For HTCondor-C jobs and all other non-grid universe jobs, if trans‐
1350 fer_output_filesis not specified, HTCondor will automatically trans‐
1351 fer back all files in the job's temporary working directory which
1352 have been modified or created by the job. Subdirectories are not
1353 scanned for output, so if output from subdirectories is desired, the
1354 output list must be explicitly specified. For grid universe jobs
1355 other than HTCondor-C, desired output files must also be explicitly
1356 listed. Another reason to explicitly list output files is for a job
1357 that creates many files, and the user wants only a subset trans‐
1358 ferred back.
1359 For grid universe jobs other than with grid type condor, to have
1361 files other than standard output and standard error transferred from
1362 the execute machine back to the submit machine, do use transfer_out‐
1363 put_files, listing all files to be transferred. These files are
1364 found on the execute machine in the working directory of the job.
1365 When a path to an output file or directory is specified, it speci‐
1367 fies the path to the file on the execute side. As a destination on
1368 the submit side, the file is placed in the job's initial working
1369 directory, and it is named using the base name of the original path.
1370 For example, path/to/output_filebecomes output_filein the job's ini‐
1371 tial working directory. The name and path of the file that is writ‐
1372 ten on the submit side may be modified by using transfer_out‐
1373 put_remaps. Note that this remap function only works with files but
1374 not with directories.
1375 A directory may be specified using a trailing path separator. An
1377 example of a trailing path separator is the slash character on Unix
1378 platforms; a directory example using a trailing path separator is
1379 input_data/. When a directory is specified with a trailing path sep‐
1380 arator, the contents of the directory are transferred, but the
1381 directory itself is not transferred. It is as if each of the items
1382 within the directory were listed in the transfer list. When there is
1383 no trailing path separator, the directory is transferred, its con‐
1384 tents are transferred, and these contents are placed inside the
1385 transferred directory.
1386 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1388 tories is not currently supported.
1389 Symbolic links to files are transferred as the files they point to.
1391 Transfer of symbolic links to directories is not currently sup‐
1392 ported.
1393 transfer_output_remaps = < “ name = newname ; name2 = newname2
1399 ... ”>
1400 This specifies the name (and optionally path) to use when download‐
1402 ing output files from the completed job. Normally, output files are
1403 transferred back to the initial working directory with the same name
1404 they had in the execution directory. This gives you the option to
1405 save them with a different path or name. If you specify a relative
1406 path, the final path will be relative to the job's initial working
1407 directory.
1408 namedescribes an output file name produced by your job, and new‐
1410 namedescribes the file name it should be downloaded to. Multiple
1411 remaps can be specified by separating each with a semicolon. If you
1412 wish to remap file names that contain equals signs or semicolons,
1413 these special characters may be escaped with a backslash. You cannot
1414 specify directories to be remapped.
1415 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT >
1421 Setting when_to_transfer_outputequal to ON_EXITwill cause HTCondor
1425 to transfer the job's output files back to the submitting machine
1426 only when the job completes (exits on its own).
1427 The ON_EXIT_OR_EVICToption is intended for fault tolerant jobs which
1429 periodically save their own state and can restart where they left
1430 off. In this case, files are spooled to the submit machine any time
1431 the job leaves a remote site, either because it exited on its own,
1432 or was evicted by the HTCondor system for any reason prior to job
1433 completion. The files spooled back are placed in a directory defined
1434 by the value of the SPOOLconfiguration variable. Any output files
1435 transferred back to the submit machine are automatically sent back
1436 out again as input files if the job restarts.
1437 POLICY COMMANDS
1443 max_retries = <integer>
1451 The maximum number of retries allowed for this job (must be non-neg‐
1453 ative). If the job fails (does not exit with the success_exit_code‐
1454 exit code) it will be retried up to max_retriestimes (unless retries
1455 are ceased because of the retry_untilcommand). If max_retriesis not
1456 defined, and either retry_untilor success_exit_codeis, the value of
1457 DEFAULT_JOB_MAX_RETRIESwill be used for the maximum number of
1458 retries.
1459 The combination of the max_retries, retry_until, and suc‐
1461 cess_exit_codecommands causes an appropriate OnExitRemoveexpression
1462 to be automatically generated. If retry command(s) and
1463 on_exit_removeare both defined, the OnExitRemoveexpression will be
1464 generated by OR'ing the expression specified in OnExitRemoveand the
1465 expression generated by the retry commands.
1466 retry_until <Integer | ClassAd Boolean Expression>
1472 An integer value or boolean expression that prevents further retries
1474 from taking place, even if max_retrieshave not been exhausted. If
1475 retry_untilis an integer, the job exiting with that exit code will
1476 cause retries to cease. If retry_untilis a ClassAd expression, the
1477 expression evaluating to Truewill cause retries to cease.
1478 success_exit_code = <integer>
1484 The exit code that is considered successful for this job. Defaults
1486 to 0 if not defined.
1487 Note: non-zero values of success_exit_codeshould generally not be
1489 used for DAG node jobs.At the present time, condor_dagmandoes not
1490 take into account the value of success_exit_code. This means that,
1491 if success_exit_codeis set to a non-zero value, condor_dagmanwill
1492 consider the job failed when it actually succeeds. For single-proc
1493 DAG node jobs, this can be overcome by using a POST script that
1494 takes into account the value of success_exit_code(although this is
1495 not recommended). For multi-proc DAG node jobs, there is currently
1496 no way to overcome this limitation.
1497 hold = <True | False>
1503 If holdis set to True, then the submitted job will be placed into
1505 the Hold state. Jobs in the Hold state will not run until released
1506 by condor_release. Defaults to False.
1507 keep_claim_idle = <integer>
1513 An integer number of seconds that a job requests the condor_scheddto
1515 wait before releasing its claim after the job exits or after the job
1516 is removed.
1517 The process by which the condor_scheddclaims a condor_startdis some‐
1519 what time-consuming. To amortize this cost, the condor_scheddtries
1520 to reuse claims to run subsequent jobs, after a job using a claim is
1521 done. However, it can only do this if there is an idle job in the
1522 queue at the moment the previous job completes. Sometimes, and espe‐
1523 cially for the node jobs when using DAGMan, there is a subsequent
1524 job about to be submitted, but it has not yet arrived in the queue
1525 when the previous job completes. As a result, the condor_sched‐
1526 dreleases the claim, and the next job must wait an entire negotia‐
1527 tion cycle to start. When this submit command is defined with a non-
1528 negative integer, when the job exits, the condor_scheddtries as
1529 usual to reuse the claim. If it cannot, instead of releasing the
1530 claim, the condor_scheddkeeps the claim until either the number of
1531 seconds given as a parameter, or a new job which matches that claim
1532 arrives, whichever comes first. The condor_startdin question will
1533 remain in the Claimed/Idle state, and the original job will be
1534 "charged" (in terms of priority) for the time in this state.
1535 leave_in_queue = <ClassAd Boolean Expression>
1541 When the ClassAd Expression evaluates to True, the job is not
1543 removed from the queue upon completion. This allows the user of a
1544 remotely spooled job to retrieve output files in cases where HTCon‐
1545 dor would have removed them as part of the cleanup associated with
1546 completion. The job will only exit the queue once it has been marked
1547 for removal (via condor_rm, for example) and the leave_in_queueex‐
1548 pression has become False. leave_in_queuedefaults to False.
1549 As an example, if the job is to be removed once the output is
1551 retrieved with condor_transfer_data, then use
1552 leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED)
1554 ||\
1555 (StageOutFinish == 0))
1556 next_job_start_delay = <ClassAd Boolean Expression>
1562 This expression specifies the number of seconds to delay after
1564 starting up this job before the next job is started. The maximum
1565 allowed delay is specified by the HTCondor configuration variable
1566 MAX_NEXT_JOB_START_DELAY, which defaults to 10 minutes. This command
1567 does not apply to scheduleror localuniverse jobs.
1568 This command has been historically used to implement a form of job
1570 start throttling from the job submitter's perspective. It was effec‐
1571 tive for the case of multiple job submission where the transfer of
1572 extremely large input data sets to the execute machine caused
1573 machine performance to suffer. This command is no longer useful, as
1574 throttling should be accomplished through configuration of the con‐
1575 dor_schedddaemon.
1576 on_exit_hold = <ClassAd Boolean Expression>
1582 The ClassAd expression is checked when the job exits, and if True,
1584 places the job into the Hold state. If False(the default value when
1585 not defined), then nothing happens and the on_exit_removeexpression
1586 is checked to determine if that needs to be applied.
1587 For example: Suppose a job is known to run for a minimum of an hour.
1589 If the job exits after less than an hour, the job should be placed
1590 on hold and an e-mail notification sent, instead of being allowed to
1591 leave the queue.
1592 on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1596 This expression places the job on hold if it exits for any reason
1598 before running for an hour. An e-mail will be sent to the user
1599 explaining that the job was placed on hold because this expression
1600 became True.
1601 periodic_*expressions take precedence over on_exit_*expressions, and
1603 *_holdexpressions take precedence over a *_removeexpressions.
1604 Only job ClassAd attributes will be defined for use by this ClassAd
1606 expression. This expression is available for the vanilla, java, par‐
1607 allel, grid, local and scheduler universes. It is additionally
1608 available, when submitted from a Unix machine, for the standard uni‐
1609 verse.
1610 on_exit_hold_reason = <ClassAd String Expression>
1616 When the job is placed on hold due to the on_exit_holdexpression
1618 becoming True, this expression is evaluated to set the value of Hol‐
1619 dReasonin the job ClassAd. If this expression is UNDEFINEDor pro‐
1620 duces an empty or invalid string, a default description is used.
1621 on_exit_hold_subcode = <ClassAd Integer Expression>
1627 When the job is placed on hold due to the on_exit_holdexpression
1629 becoming True, this expression is evaluated to set the value of Hol‐
1630 dReasonSubCodein the job ClassAd. The default subcode is 0. The Hol‐
1631 dReasonCodewill be set to 3, which indicates that the job went on
1632 hold due to a job policy expression.
1633 on_exit_remove = <ClassAd Boolean Expression>
1639 The ClassAd expression is checked when the job exits, and if
1641 True(the default value when undefined), then it allows the job to
1642 leave the queue normally. If False, then the job is placed back into
1643 the Idle state. If the user job runs under the vanilla universe,
1644 then the job restarts from the beginning. If the user job runs under
1645 the standard universe, then it continues from where it left off,
1646 using the last checkpoint.
1647 For example, suppose a job occasionally segfaults, but chances are
1649 that the job will finish successfully if the job is run again with
1650 the same data. The on_exit_removeexpression can cause the job to run
1651 again with the following command. Assume that the signal identifier
1652 for the segmentation fault is 11 on the platform where the job will
1653 be running.
1654 on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)
1656 This expression lets the job leave the queue if the job was not
1658 killed by a signal or if it was killed by a signal other than 11,
1659 representing segmentation fault in this example. So, if the exited
1660 due to signal 11, it will stay in the job queue. In any other case
1661 of the job exiting, the job will leave the queue as it normally
1662 would have done.
1663 As another example, if the job should only leave the queue if it
1665 exited on its own with status 0, this on_exit_removeexpression works
1666 well:
1667 on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
1671 If the job was killed by a signal or exited with a non-zero exit
1673 status, HTCondor would leave the job in the queue to run again.
1674 periodic_*expressions take precedence over on_exit_*expressions, and
1676 *_holdexpressions take precedence over a *_removeexpressions.
1677 Only job ClassAd attributes will be defined for use by this ClassAd
1679 expression.
1680 periodic_hold = <ClassAd Boolean Expression>
1686 This expression is checked periodically when the job is not in the
1688 Held state. If it becomes True, the job will be placed on hold. If
1689 unspecified, the default value is False.
1690 periodic_*expressions take precedence over on_exit_*expressions, and
1692 *_holdexpressions take precedence over a *_removeexpressions.
1693 Only job ClassAd attributes will be defined for use by this ClassAd
1695 expression. Note that, by default, this expression is only checked
1696 once every 60 seconds. The period of these evaluations can be
1697 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1698 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1699 periodic_hold_reason = <ClassAd String Expression>
1705 When the job is placed on hold due to the periodic_holdexpression
1707 becoming True, this expression is evaluated to set the value of Hol‐
1708 dReasonin the job ClassAd. If this expression is UNDEFINEDor pro‐
1709 duces an empty or invalid string, a default description is used.
1710 periodic_hold_subcode = <ClassAd Integer Expression>
1716 When the job is placed on hold due to the periodic_holdexpression
1718 becoming true, this expression is evaluated to set the value of Hol‐
1719 dReasonSubCodein the job ClassAd. The default subcode is 0. The Hol‐
1720 dReasonCodewill be set to 3, which indicates that the job went on
1721 hold due to a job policy expression.
1722 periodic_release = <ClassAd Boolean Expression>
1728 This expression is checked periodically when the job is in the Held
1730 state. If the expression becomes True, the job will be released.
1731 Only job ClassAd attributes will be defined for use by this ClassAd
1733 expression. Note that, by default, this expression is only checked
1734 once every 60 seconds. The period of these evaluations can be
1735 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1736 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1737 periodic_remove = <ClassAd Boolean Expression>
1743 This expression is checked periodically. If it becomes True, the job
1745 is removed from the queue. If unspecified, the default value is
1746 False.
1747 See the Examples section of this manual page for an example of a
1749 periodic_removeexpression.
1750 periodic_*expressions take precedence over on_exit_*expressions, and
1752 *_holdexpressions take precedence over a *_removeexpressions. So,
1753 the periodic_removeexpression takes precedent over the
1754 on_exit_removeexpression, if the two describe conflicting actions.
1755 Only job ClassAd attributes will be defined for use by this ClassAd
1757 expression. Note that, by default, this expression is only checked
1758 once every 60 seconds. The period of these evaluations can be
1759 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1760 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1761 COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
1767 allow_startup_script = <True | False>
1775 If True, a standard universe job will execute a script instead of
1777 submitting the job, and the consistency check to see if the exe‐
1778 cutable has been linked using condor_compileis omitted. The exe‐
1779 cutablecommand within the submit description file specifies the name
1780 of the script. The script is used to do preprocessing before the job
1781 is submitted. The shell script ends with an execof the job exe‐
1782 cutable, such that the process id of the executable is the same as
1783 that of the shell script. Here is an example script that gets a copy
1784 of a machine-specific executable before the exec.
1785 #! /bin/sh
1787 # get the host name of the machine
1789 $host=`uname -n`
1790 # grab a standard universe executable designed specifically
1792 # for this host
1793 scp elsewhere@cs.wisc.edu:${host} executable
1794 # The PID MUST stay the same, so exec the new standard universe
1796 process.
1797 exec executable ${1+"$@"} If this command is not present (defined),
1798 then the value defaults to false.
1799 append_files = file1, file2, ...
1805 If your job attempts to access a file mentioned in this list, HTCon‐
1809 dor will force all writes to that file to be appended to the end.
1810 Furthermore, condor_submit will not truncate it. This list uses the
1811 same syntax as compress_files, shown above.
1812 This option may yield some surprising results. If several jobs
1814 attempt to write to the same file, their output may be intermixed.
1815 If a job is evicted from one or more machines during the course of
1816 its lifetime, such an output file might contain several copies of
1817 the results. This option should be only be used when you wish a cer‐
1818 tain file to be treated as a running log instead of a precise
1819 result.
1820 This option only applies to standard-universe jobs.
1822 buffer_files = < “ name = (size,block-size) ; name2 =
1828 (size,block-size) ... ” >
1829 buffer_size = <bytes-in-buffer>
1835 buffer_block_size = <bytes-in-block>
1841 HTCondor keeps a buffer of recently-used data for each file a job
1843 accesses. This buffer is used both to cache commonly-used data and
1844 to consolidate small reads and writes into larger operations that
1845 get better throughput. The default settings should produce reason‐
1846 able results for most programs.
1847 These options only apply to standard-universe jobs.
1849 If needed, you may set the buffer controls individually for each
1851 file using the buffer_files option. For example, to set the buffer
1852 size to 1 MiB and the block size to 256 KiB for the file input.data,
1853 use this command:
1854 buffer_files = "input.data=(1000000,256000)"
1858 Alternatively, you may use these two options to set the default
1860 sizes for all files used by your job:
1861 buffer_size = 1000000
1865 buffer_block_size = 256000
1866 If you do not set these, HTCondor will use the values given by these
1868 two configuration file macros:
1869 DEFAULT_IO_BUFFER_SIZE = 1000000
1873 DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
1874 Finally, if no other settings are present, HTCondor will use a buf‐
1876 fer of 512 KiB and a block size of 32 KiB.
1877 compress_files = file1, file2, ...
1883 If your job attempts to access any of the files mentioned in this
1887 list, HTCondor will automatically compress them (if writing) or
1888 decompress them (if reading). The compress format is the same as
1889 used by GNU gzip.
1890 The files given in this list may be simple file names or complete
1892 paths and may include as a wild card. For example, this list causes
1893 the file /tmp/data.gz, any file named event.gz, and any file ending
1894 in .gzip to be automatically compressed or decompressed as needed:
1895 compress_files = /tmp/data.gz, event.gz, *.gzip
1899 Due to the nature of the compression format, compressed files must
1901 only be accessed sequentially. Random access reading is allowed but
1902 is very slow, while random access writing is simply not possible.
1903 This restriction may be avoided by using both compress_files and
1904 fetch_files at the same time. When this is done, a file is kept in
1905 the decompressed state at the execution machine, but is compressed
1906 for transfer to its original location.
1907 This option only applies to standard universe jobs.
1909 fetch_files = file1, file2, ...
1915 If your job attempts to access a file mentioned in this list, HTCon‐
1917 dor will automatically copy the whole file to the executing machine,
1918 where it can be accessed quickly. When your job closes the file, it
1919 will be copied back to its original location. This list uses the
1920 same syntax as compress_files, shown above.
1921 This option only applies to standard universe jobs.
1923 file_remaps = < “ name = newname ; name2 = newname2 ... ”>
1929 Directs HTCondor to use a new file name in place of an old one.
1933 namedescribes a file name that your job may attempt to open, and
1934 newnamedescribes the file name it should be replaced with. new‐
1935 namemay include an optional leading access specifier, local: or
1936 remote: . If left unspecified, the default access specifier is
1937 remote: . Multiple remaps can be specified by separating each with a
1938 semicolon.
1939 This option only applies to standard universe jobs.
1941 If you wish to remap file names that contain equals signs or semi‐
1943 colons, these special characters may be escaped with a backslash.
1944 Example One:
1948 Suppose that your job reads a file named dataset.1. To instruct
1950 HTCondor to force your job to read other.datasetinstead, add this
1951 to the submit file:
1952 file_remaps = "dataset.1=other.dataset"
1954 Example Two:
1958 Suppose that your run many jobs which all read in the same large
1960 file, called very.big. If this file can be found in the same
1961 place on a local disk in every machine in the pool, (say
1962 /bigdisk/bigfile,) you can instruct HTCondor of this fact by
1963 remapping very.bigto /bigdisk/bigfileand specifying that the file
1964 is to be read locally, which will be much faster than reading
1965 over the network.
1966 file_remaps = "very.big = local:/bigdisk/bigfile"
1968 Example Three:
1972 Several remaps can be applied at once by separating each with a
1974 semicolon.
1975 file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 =
1977 other.dataset"
1978 local_files = file1, file2, ...
1986 If your job attempts to access a file mentioned in this list, HTCon‐
1990 dor will cause it to be read or written at the execution machine.
1991 This is most useful for temporary files not used for input or out‐
1992 put. This list uses the same syntax as compress_files, shown above.
1993 local_files = /tmp/*
1997 This option only applies to standard universe jobs.
1999 want_remote_io = <True | False>
2005 This option controls how a file is opened and manipulated in a stan‐
2007 dard universe job. If this option is true, which is the default,
2008 then the condor_shadowmakes all decisions about how each and every
2009 file should be opened by the executing job. This entails a network
2010 round trip (or more) from the job to the condor_shadowand back again
2011 for every single open()in addition to other needed information about
2012 the file. If set to false, then when the job queries the con‐
2013 dor_shadowfor the first time about how to open a file, the con‐
2014 dor_shadowwill inform the job to automatically perform all of its
2015 file manipulation on the local file system on the execute machine
2016 and any file remapping will be ignored. This means that there mustbe
2017 a shared file system (such as NFS or AFS) between the execute
2018 machine and the submit machine and that ALLpaths that the job could
2019 open on the execute machine must be valid. The ability of the stan‐
2020 dard universe job to checkpoint, possibly to a checkpoint server, is
2021 not affected by this attribute. However, when the job resumes it
2022 will be expecting the same file system conditions that were present
2023 when the job checkpointed.
2024 COMMANDS FOR THE GRID
2030 azure_admin_key = <pathname>
2038 For grid type azurejobs, specifies the path and file name of a file
2040 that contains an SSH public key. This key can be used to log into
2041 the administrator account of the instance via SSH.
2042 azure_admin_username = <account name>
2048 For grid type azurejobs, specifies the name of an administrator
2050 account to be created in the instance. This account can be logged
2051 into via SSH.
2052 azure_auth_file = <pathname>
2058 For grid type azurejobs, specifies a path and file name of the
2060 authorization file that grants permission for HTCondor to use the
2061 Azure account. If it's not defined, then HTCondor will attempt to
2062 use the default credentials of the Azure CLI tools.
2063 azure_image = <image id>
2069 For grid type azurejobs, identifies the disk image to be used for
2071 the boot disk of the instance. This image must already be registered
2072 within Azure.
2073 azure_location = <image id>
2079 For grid type azurejobs, identifies the location within Azure where
2081 the instance should be run. As an example, one current location is
2082 centralus.
2083 azure_size = <machine type>
2089 For grid type azurejobs, the hardware configuration that the virtual
2091 machine instance is to run on.
2092 batch_queue = <queuename>
2098 Used for pbs, lsf, and sgegrid universe jobs. Specifies the name of
2100 the PBS/LSF/SGE job queue into which the job should be submitted. If
2101 not specified, the default queue is used.
2102 boinc_authenticator_file = <pathname>
2108 For grid type boincjobs, specifies a path and file name of the
2110 authorization file that grants permission for HTCondor to use the
2111 BOINC service. There is no default value when not specified.
2112 cream_attributes = <name=value;...;name=value>
2118 Provides a list of attribute/value pairs to be set in a CREAM job
2120 description of a grid universe job destined for the CREAM grid sys‐
2121 tem. The pairs are separated by semicolons, and written in New Clas‐
2122 sAd syntax.
2123 delegate_job_GSI_credentials_lifetime = <seconds>
2129 Specifies the maximum number of seconds for which delegated proxies
2131 should be valid. The default behavior when this command is not spec‐
2132 ified is determined by the configuration variable DELE‐
2133 GATE_JOB_GSI_CREDENTIALS_LIFETIME, which defaults to one day. A
2134 value of 0 indicates that the delegated proxy should be valid for as
2135 long as allowed by the credential used to create the proxy. This
2136 setting currently only applies to proxies delegated for non-grid
2137 jobs and for HTCondor-C jobs. It does not currently apply to globus
2138 grid jobs, which always behave as though this setting were 0. This
2139 variable has no effect if the configuration variable DELE‐
2140 GATE_JOB_GSI_CREDENTIALSis False, because in that case the job proxy
2141 is copied rather than delegated.
2142 ec2_access_key_id = <pathname>
2148 For grid type ec2jobs, identifies the file containing the access
2150 key.
2151 ec2_ami_id = <EC2 xMI ID>
2157 For grid type ec2jobs, identifies the machine image. Services com‐
2159 patible with the EC2 Query API may refer to these with abbreviations
2160 other than AMI, for example EMIis valid for Eucalyptus.
2161 ec2_availability_zone = <zone name>
2167 For grid type ec2jobs, specifies the Availability Zone that the
2169 instance should be run in. This command is optional, unless
2170 ec2_ebs_volumesis set. As an example, one current zone is us-
2171 east-1b.
2172 ec2_block_device_mapping = <block-device>:<kernel-device>,<block-
2178 device>:<kernel-device>, ...
2179 For grid type ec2jobs, specifies the block device to kernel device
2181 mapping. This command is optional.
2182 ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device name>,...
2188 For grid type ec2jobs, optionally specifies a list of Elastic Block
2190 Store (EBS) volumes to be made available to the instance and the
2191 device names they should have in the instance.
2192 ec2_elastic_ip = <elastic IP address>
2198 For grid type ec2jobs, and optional specification of an Elastic IP
2200 address that should be assigned to this instance.
2201 ec2_iam_profile_arn = <IAM profile ARN>
2207 For grid type ec2jobs, an Amazon Resource Name (ARN) identifying
2209 which Identity and Access Management (IAM) (instance) profile to as‐
2210 sociate with the instance.
2211 ec2_iam_profile_name= <IAM profile name>
2217 For grid type ec2jobs, a name identifying which Identity and Access
2219 Management (IAM) (instance) profile to associate with the instance.
2220 ec2_instance_type = <instance type>
2226 For grid type ec2jobs, identifies the instance type. Different ser‐
2228 vices may offer different instance types, so no default value is
2229 set.
2230 ec2_keypair = <ssh key-pair name>
2236 For grid type ec2jobs, specifies the name of an SSH key-pair that is
2238 already registered with the EC2 service. The associated private key
2239 can be used to sshinto the virtual machine once it is running.
2240 ec2_keypair_file = <pathname>
2246 For grid type ec2jobs, specifies the complete path and file name of
2248 a file into which HTCondor will write an SSH key for use with ec2
2249 jobs. The key can be used to sshinto the virtual machine once it is
2250 running. If ec2_keypairis specified for a job, ec2_keypair_fileis
2251 ignored.
2252 ec2_parameter_names = ParameterName1, ParameterName2, ...
2258 For grid type ec2jobs, a space or comma separated list of the names
2260 of additional parameters to pass when instantiating an instance.
2261 ec2_parameter_<name> = <value>
2267 For grid type ec2jobs, specifies the value for the correspondingly
2269 named (instance instantiation) parameter. <name>is the parameter
2270 name specified in the submit command ec2_parameter_names, but with
2271 any periods replaced by underscores.
2272 ec2_secret_access_key = <pathname>
2278 For grid type ec2jobs, specifies the path and file name containing
2280 the secret access key.
2281 ec2_security_groups = group1, group2, ...
2287 For grid type ec2jobs, defines the list of EC2 security groups which
2289 should be associated with the job.
2290 ec2_security_ids = id1, id2, ...
2296 For grid type ec2jobs, defines the list of EC2 security group IDs
2298 which should be associated with the job.
2299 ec2_spot_price = <bid>
2305 For grid type ec2jobs, specifies the spot instance bid, which is the
2307 most that the job submitter is willing to pay per hour to run this
2308 job.
2309 ec2_tag_names = <name0,name1,name...>
2315 For grid type ec2jobs, specifies the case of tag names that will be
2317 associated with the running instance. This is only necessary if a
2318 tag name case matters. By default the list will be automatically
2319 generated.
2320 ec2_tag_<name> = <value>
2326 For grid type ec2jobs, specifies a tag to be associated with the
2328 running instance. The tag name will be lower-cased, use
2329 ec2_tag_namesto change the case.
2330 WantNameTag = <True | False>
2336 For grid type ec2jobs, a job may request that its 'name' tag be
2338 (not) set by HTCondor. If the job does not otherwise specify any
2339 tags, not setting its name tag will eliminate a call by the EC2
2340 GAHP, improving performance.
2341 ec2_user_data = <data>
2347 For grid type ec2jobs, provides a block of data that can be accessed
2349 by the virtual machine. If both ec2_user_dataand
2350 ec2_user_data_fileare specified for a job, the two blocks of data
2351 are concatenated, with the data from this ec2_user_datasubmit com‐
2352 mand occurring first.
2353 ec2_user_data_file = <pathname>
2359 For grid type ec2jobs, specifies a path and file name whose contents
2361 can be accessed by the virtual machine. If both ec2_user_dataand
2362 ec2_user_data_fileare specified for a job, the two blocks of data
2363 are concatenated, with the data from that ec2_user_datasubmit com‐
2364 mand occurring first.
2365 ec2_vpc_ip = <a.b.c.d>
2371 For grid type ec2jobs, that are part of a Virtual Private Cloud
2373 (VPC), an optional specification of the IP address that this
2374 instance should have within the VPC.
2375 ec2_vpc_subnet = <subnet specification string>
2381 For grid type ec2jobs, an optional specification of the Virtual Pri‐
2383 vate Cloud (VPC) that this instance should be a part of.
2384 gce_account = <account name>
2390 For grid type gcejobs, specifies the Google cloud services account
2392 to use. If this submit command isn't specified, then a random
2393 account from the authorization file given by gce_auth_filewill be
2394 used.
2395 gce_auth_file = <pathname>
2401 For grid type gcejobs, specifies a path and file name of the autho‐
2403 rization file that grants permission for HTCondor to use the Google
2404 account. If this command is not specified, then the default file of
2405 the Google command-line tools will be used.
2406 gce_image = <image id>
2412 For grid type gcejobs, the identifier of the virtual machine image
2414 representing the HTCondor job to be run. This virtual machine image
2415 must already be register with GCE and reside in Google's Cloud Stor‐
2416 age service.
2417 gce_json_file = <pathname>
2423 For grid type gcejobs, specifies the path and file name of a file
2425 that contains JSON elements that should be added to the instance
2426 description submitted to the GCE service.
2427 gce_machine_type = <machine type>
2433 For grid type gcejobs, the long form of the URL that describes the
2435 machine configuration that the virtual machine instance is to run
2436 on.
2437 gce_metadata = <name=value,...,name=value>
2443 For grid type gcejobs, a comma separated list of name and value
2445 pairs that define metadata for a virtual machine instance that is an
2446 HTCondor job.
2447 gce_metadata_file = <pathname>
2453 For grid type gcejobs, specifies a path and file name of the file
2455 that contains metadata for a virtual machine instance that is an
2456 HTCondor job. Within the file, each name and value pair is on its
2457 own line; so, the pairs are separated by the newline character.
2458 gce_preemptible = <True | False>
2464 For grid type gcejobs, specifies whether the virtual machine
2466 instance should be preemptible. The default is for the instance to
2467 not be preemptible.
2468 globus_rematch = <ClassAd Boolean Expression>
2474 This expression is evaluated by the condor_gridmanagerwhenever:
2476 1. the globus_resubmitexpression evaluates to True
2478 2. the condor_gridmanagerdecides it needs to retry a submission
2480 (as when a previous submission failed to commit) If
2481 globus_rematchevaluates to True, then beforethe job is submitted
2482 again to globus, the condor_gridmanagerwill request that the con‐
2483 dor_schedddaemon renegotiate with the matchmaker (the con‐
2484 dor_negotiator). The result is this job will be matched again.
2485 globus_resubmit = <ClassAd Boolean Expression>
2491 The expression is evaluated by the condor_gridmanagereach time the
2493 condor_gridmanagergets a job ad to manage. Therefore, the expression
2494 is evaluated:
2495 1. when a grid universe job is first submitted to HTCondor-G
2497 2. when a grid universe job is released from the hold state
2499 3. when HTCondor-G is restarted (specifically, whenever the con‐
2501 dor_gridmanageris restarted) If the expression evaluates to True,
2502 then any previous submission to the grid universe will be forgot‐
2503 ten and this job will be submitted again as a fresh submission to
2504 the grid universe. This may be useful if there is a desire to
2505 give up on a previous submission and try again. Note that this
2506 may result in the same job running more than once. Do not treat
2507 this operation lightly.
2508 globus_rsl = <RSL-string>
2514 Used to provide any additional Globus RSL string attributes which
2516 are not covered by other submit description file commands or job
2517 attributes. Used for griduniversejobs, where the grid resource has a
2518 grid-type-stringof gt2.
2519 grid_resource = <grid-type-string> <grid-specific-parameter-list>
2525 For each grid-type-stringvalue, there are further type-specific val‐
2527 ues that must specified. This submit description file command allows
2528 each to be given in a space-separated list. Allowable grid-type-
2529 stringvalues are batch, condor, cream, ec2, gt2, gt5, lsf, nor‐
2530 dugrid, pbs, sge, and unicore. The HTCondor manual chapter on Grid
2531 Computing details the variety of grid types.
2532 For a grid-type-stringof batch, the single parameter is the name of
2534 the local batch system, and will be one of pbs, lsf, or sge.
2535 For a grid-type-stringof condor, the first parameter is the name of
2537 the remote condor_schedddaemon. The second parameter is the name of
2538 the pool to which the remote condor_schedddaemon belongs.
2539 For a grid-type-stringof cream, there are three parameters. The
2541 first parameter is the web services address of the CREAM server. The
2542 second parameter is the name of the batch system that sits behind
2543 the CREAM server. The third parameter identifies a site-specific
2544 queue within the batch system.
2545 For a grid-type-stringof ec2, one additional parameter specifies the
2547 EC2 URL.
2548 For a grid-type-stringof gt2, the single parameter is the name of
2550 the pre-WS GRAM resource to be used.
2551 For a grid-type-stringof gt5, the single parameter is the name of
2553 the pre-WS GRAM resource to be used, which is the same as for the
2554 grid-type-stringof gt2.
2555 For a grid-type-stringof lsf, no additional parameters are used.
2557 For a grid-type-stringof nordugrid, the single parameter is the name
2559 of the NorduGrid resource to be used.
2560 For a grid-type-stringof pbs, no additional parameters are used.
2562 For a grid-type-stringof sge, no additional parameters are used.
2564 For a grid-type-stringof unicore, the first parameter is the name of
2566 the Unicore Usite to be used. The second parameter is the name of
2567 the Unicore Vsite to be used.
2568 keystore_alias = <name>
2574 A string to locate the certificate in a Java keystore file, as used
2576 for a unicorejob.
2577 keystore_file = <pathname>
2583 The complete path and file name of the Java keystore file containing
2585 the certificate to be used for a unicorejob.
2586 keystore_passphrase_file = <pathname>
2592 The complete path and file name to the file containing the
2594 passphrase protecting a Java keystore file containing the certifi‐
2595 cate. Relevant for a unicorejob.
2596 MyProxyCredentialName = <symbolic name>
2602 The symbolic name that identifies a credential to the MyProxyserver.
2604 This symbolic name is set as the credential is initially stored on
2605 the server (using myproxy-init).
2606 MyProxyHost = <host>:<port>
2612 The Internet address of the host that is the MyProxyserver. The
2614 hostmay be specified by either a host name (as in head.example.com)
2615 or an IP address (of the form 123.456.7.8). The portnumber is an
2616 integer.
2617 MyProxyNewProxyLifetime = <number-of-minutes>
2623 The new lifetime (in minutes) of the proxy after it is refreshed.
2625 MyProxyPassword = <password>
2631 The password needed to refresh a credential on the MyProxyserver.
2633 This password is set when the user initially stores credentials on
2634 the server (using myproxy-init). As an alternative to using MyProxy‐
2635 Passwordin the submit description file, the password may be speci‐
2636 fied as a command line argument to condor_submitwith the -passwor‐
2637 dargument.
2638 MyProxyRefreshThreshold = <number-of-seconds>
2644 The time (in seconds) before the expiration of a proxy that the
2646 proxy should be refreshed. For example, if MyProxyRefreshThresholdis
2647 set to the value 600, the proxy will be refreshed 10 minutes before
2648 it expires.
2649 MyProxyServerDN = <credential subject>
2655 A string that specifies the expected Distinguished Name (credential
2657 subject, abbreviated DN) of the MyProxyserver. It must be specified
2658 when the MyProxyserver DN does not follow the conventional naming
2659 scheme of a host credential. This occurs, for example, when the
2660 MyProxyserver DN begins with a user credential.
2661 nordugrid_rsl = <RSL-string>
2667 Used to provide any additional RSL string attributes which are not
2669 covered by regular submit description file parameters. Used when the
2670 universeis grid, and the type of grid system is nordugrid.
2671 transfer_error = <True | False>
2677 For jobs submitted to the grid universe only. If True, then the
2679 error output (from stderr) from the job is transferred from the
2680 remote machine back to the submit machine. The name of the file
2681 after transfer is given by the errorcommand. If False, no transfer
2682 takes place (from the remote machine to submit machine), and the
2683 name of the file is given by the errorcommand. The default value is
2684 True.
2685 transfer_input = <True | False>
2691 For jobs submitted to the grid universe only. If True, then the job
2693 input (stdin) is transferred from the machine where the job was sub‐
2694 mitted to the remote machine. The name of the file that is trans‐
2695 ferred is given by the inputcommand. If False, then the job's input
2696 is taken from a pre-staged file on the remote machine, and the name
2697 of the file is given by the inputcommand. The default value is True.
2698 For transferring files other than stdin, see transfer_input_files.
2700 transfer_output = <True | False>
2706 For jobs submitted to the grid universe only. If True, then the out‐
2708 put (from stdout) from the job is transferred from the remote
2709 machine back to the submit machine. The name of the file after
2710 transfer is given by the outputcommand. If False, no transfer takes
2711 place (from the remote machine to submit machine), and the name of
2712 the file is given by the outputcommand. The default value is True.
2713 For transferring files other than stdout, see transfer_output_files.
2715 use_x509userproxy = <True | False>
2721 Set this command to Trueto indicate that the job requires an X.509
2723 user proxy. If x509userproxyis set, then that file is used for the
2724 proxy. Otherwise, the proxy is looked for in the standard locations.
2725 If x509userproxyis set or if the job is a grid universe job of grid
2726 type gt2, gt5, cream, or nordugrid, then the value of use_x509user‐
2727 proxyis forced to True. Defaults to False.
2728 x509userproxy = <full-pathname>
2734 Used to override the default path name for X.509 user certificates.
2736 The default location for X.509 proxies is the /tmpdirectory, which
2737 is generally a local file system. Setting this value would allow
2738 HTCondor to access the proxy in a shared file system (for example,
2739 AFS). HTCondor will use the proxy specified in the submit descrip‐
2740 tion file first. If nothing is specified in the submit description
2741 file, it will use the environment variable X509_USER_PROXY. If that
2742 variable is not present, it will search in the default location.
2743 Note that proxies are only valid for a limited time. Condor_submit
2744 will not submit a job with an expired proxy, it will return an
2745 error. Also, if the configuration parameter CRED_MIN_TIME_LEFT is
2746 set to some number of seconds, and if the proxy will expire before
2747 that many seconds, condor_submit will also refuse to submit the job.
2748 That is, if CRED_MIN_TIME_LEFT is set to 60, condor_submit will
2749 refuse to submit a job whose proxy will expire 60 seconds from the
2750 time of submission.
2751 x509userproxyis relevant when the universeis vanilla, or when the
2753 universeis gridand the type of grid system is one of gt2, gt5, con‐
2754 dor, cream, or nordugrid. Defining a value causes the proxy to be
2755 delegated to the execute machine. Further, VOMS attributes defined
2756 in the proxy will appear in the job ClassAd.
2757 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2763 hold_kill_sig = <signal-number>
2771 For the scheduler universe only, signal-numberis the signal deliv‐
2773 ered to the job when the job is put on hold with condor_hold. sig‐
2774 nal-numbermay be either the platform-specific name or value of the
2775 signal. If this command is not present, the value of kill_sigis
2776 used.
2777 jar_files = <file_list>
2783 Specifies a list of additional JAR files to include when using the
2785 Java universe. JAR files will be transferred along with the exe‐
2786 cutable and automatically added to the classpath.
2787 java_vm_args = <argument_list>
2793 Specifies a list of additional arguments to the Java VM itself, When
2795 HTCondor runs the Java program, these are the arguments that go
2796 before the class name. This can be used to set VM-specific arguments
2797 like stack size, garbage-collector arguments and initial property
2798 values.
2799 machine_count = <max>
2805 For the parallel universe, a single value (max) is required. It is
2807 neither a maximum or minimum, but the number of machines to be dedi‐
2808 cated toward running the job.
2809 remove_kill_sig = <signal-number>
2815 For the scheduler universe only, signal-numberis the signal deliv‐
2817 ered to the job when the job is removed with condor_rm. signal-num‐
2818 bermay be either the platform-specific name or value of the signal.
2819 This example shows it both ways for a Linux signal:
2820 remove_kill_sig = SIGUSR1
2822 remove_kill_sig = 10
2823 If this command is not present, the value of kill_sigis used.
2825 COMMANDS FOR THE VM UNIVERSE
2831 vm_disk = file1:device1:permission1, file2:device2:permission2:format2,
2839 ...
2840 A list of comma separated disk files. Each disk file is specified by
2842 4 colon separated fields. The first field is the path and file name
2843 of the disk file. The second field specifies the device. The third
2844 field specifies permissions, and the optional fourth field specifies
2845 the image format. If a disk file will be transferred by HTCondor,
2846 then the first field should just be the simple file name (no path
2847 information).
2848 An example that specifies two disk files:
2850 vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2852 vm_checkpoint = <True | False>
2858 A boolean value specifying whether or not to take checkpoints. If
2860 not specified, the default value is False. In the current implemen‐
2861 tation, setting both vm_checkpointand vm_networkingto Truedoes not
2862 yet work in all cases. Networking cannot be used if a vm universe
2863 job uses a checkpoint in order to continue execution after migration
2864 to another machine.
2865 vm_macaddr = <MACAddr>
2871 Defines that MAC address that the virtual machine's network inter‐
2873 face should have, in the standard format of six groups of two hexa‐
2874 decimal digits separated by colons.
2875 vm_memory = <MBytes-of-memory>
2881 The amount of memory in MBytes that a vm universe job requires.
2883 vm_networking = <True | False>
2889 Specifies whether to use networking or not. In the current implemen‐
2891 tation, setting both vm_checkpointand vm_networkingto Truedoes not
2892 yet work in all cases. Networking cannot be used if a vm universe
2893 job uses a checkpoint in order to continue execution after migration
2894 to another machine.
2895 vm_networking_type = <nat | bridge >
2901 When vm_networkingis True, this definition augments the job's
2903 requirements to match only machines with the specified networking.
2904 If not specified, then either networking type matches.
2905 vm_no_output_vm = <True | False>
2911 When True, prevents HTCondor from transferring output files back to
2913 the machine from which the vm universe job was submitted. If not
2914 specified, the default value is False.
2915 vm_type = <vmware | xen | kvm>
2921 Specifies the underlying virtual machine software that this job
2923 expects.
2924 vmware_dir = <pathname>
2930 The complete path and name of the directory where VMware-specific
2932 files and applications such as the VMDK (Virtual Machine Disk For‐
2933 mat) and VMX (Virtual Machine Configuration) reside. This command is
2934 optional; when not specified, all relevant VMware image files are to
2935 be listed using transfer_input_files.
2936 vmware_should_transfer_files = <True | False>
2942 Specifies whether HTCondor will transfer VMware-specific files
2944 located as specified by vmware_dirto the execute machine (True) or
2945 rely on access through a shared file system (False). Omission of
2946 this required command (for VMware vm universe jobs) results in an
2947 error message from condor_submit, and the job will not be submitted.
2948 vmware_snapshot_disk = <True | False>
2954 When True, causes HTCondor to utilize a VMware snapshot disk for new
2956 or modified files. If not specified, the default value is True.
2957 xen_initrd = <image-file>
2963 When xen_kernelgives a file name for the kernel image to use, this
2965 optional command may specify a path to a ramdisk (initrd) image
2966 file. If the image file will be transferred by HTCondor, then the
2967 value should just be the simple file name (no path information).
2968 xen_kernel = <included | path-to-kernel>
2974 A value of includedspecifies that the kernel is included in the disk
2976 file. If not one of these values, then the value is a path and file
2977 name of the kernel to be used. If a kernel file will be transferred
2978 by HTCondor, then the value should just be the simple file name (no
2979 path information).
2980 xen_kernel_params = <string>
2986 A string that is appended to the Xen kernel command line.
2988 xen_root = <string>
2994 A string that is appended to the Xen kernel command line to specify
2996 the root device. This string is required when xen_kernelgives a path
2997 to a kernel. Omission for this required case results in an error
2998 message during submission.
2999 COMMANDS FOR THE DOCKER UNIVERSE
3005 docker_image = < image-name >
3013 Defines the name of the Docker image that is the basis for the
3015 docker container.
3016 ADVANCED COMMANDS
3022 accounting_group = <accounting-group-name>
3030 Causes jobs to negotiate under the given accounting group. This
3032 value is advertised in the job ClassAd as AcctGroup. The HTCondor
3033 Administrator's manual contains more information about accounting
3034 groups.
3035 accounting_group_user = <accounting-group-user-name>
3041 Sets the user name associated with the accounting group name for
3043 resource usage accounting purposes. If not set, defaults to the
3044 value of the job ClassAd attribute Owner. This value is advertised
3045 in the job ClassAd as AcctGroupUser. If an accounting group has not
3046 been set with the accounting_groupcommand, this command is ignored.
3047 concurrency_limits = <string-list>
3053 A list of resources that this job needs. The resources are presumed
3055 to have concurrency limits placed upon them, thereby limiting the
3056 number of concurrent jobs in execution which need the named
3057 resource. Commas and space characters delimit the items in the list.
3058 Each item in the list is a string that identifies the limit, or it
3059 is a ClassAd expression that evaluates to a string, and it is evalu‐
3060 ated in the context of machine ClassAd being considered as a match.
3061 Each item in the list also may specify a numerical value identifying
3062 the integer number of resources required for the job. The syntax
3063 follows the resource name by a colon character ( : ) and the numeri‐
3064 cal value. Details on concurrency limits are in the HTCondor Admin‐
3065 istrator's manual.
3066 concurrency_limits_expr = <ClassAd String Expression>
3072 A ClassAd expression that represents the list of resources that this
3074 job needs after evaluation. The ClassAd expression may specify
3075 machine ClassAd attributes that are evaluated against a matched
3076 machine. After evaluation, the list sets concurrency_limits.
3077 copy_to_spool = <True | False>
3083 If copy_to_spoolis True, then condor_submitcopies the executable to
3085 the local spool directory before running it on a remote host. As
3086 copying can be quite time consuming and unnecessary, the default
3087 value is Falsefor all job universes other than the standard uni‐
3088 verse. When False, condor_submitdoes not copy the executable to a
3089 local spool directory. The default is Truein standard universe,
3090 because resuming execution from a checkpoint can only be guaranteed
3091 to work using precisely the same executable that created the check‐
3092 point.
3093 coresize = <size>
3099 Should the user's program abort and produce a core file, coresize‐
3101 specifies the maximum size in bytes of the core file which the user
3102 wishes to keep. If coresizeis not specified in the command file, the
3103 system's user resource limit coredumpsizeis used (note that core‐
3104 dumpsizeis not an HTCondor parameter - it is an operating system
3105 parameter that can be viewed with the limitor ulimitcommand on Unix
3106 and in the Registry on Windows). A value of -1 results in no limits
3107 being applied to the core file size. If HTCondor is running as root,
3108 a coresizesetting greater than the system coredumpsizelimit will
3109 override the system setting; if HTCondor is notrunning as root, the
3110 system coredumpsizelimit will override coresize.
3111 cron_day_of_month = <Cron-evaluated Day>
3117 The set of days of the month for which a deferral time applies. The
3119 HTCondor User's manual section on Time Scheduling for Job Execution
3120 has further details.
3121 cron_day_of_week = <Cron-evaluated Day>
3127 The set of days of the week for which a deferral time applies. The
3129 HTCondor User's manual section on Time Scheduling for Job Execution
3130 has further details.
3131 cron_hour = <Cron-evaluated Hour>
3137 The set of hours of the day for which a deferral time applies. The
3139 HTCondor User's manual section on Time Scheduling for Job Execution
3140 has further details.
3141 cron_minute = <Cron-evaluated Minute>
3147 The set of minutes within an hour for which a deferral time applies.
3149 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3150 tion has further details.
3151 cron_month = <Cron-evaluated Month>
3157 The set of months within a year for which a deferral time applies.
3159 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3160 tion has further details.
3161 cron_prep_time = <ClassAd Integer Expression>
3167 Analogous to deferral_prep_time. The number of seconds prior to a
3169 job's deferral time that the job may be matched and sent to an exe‐
3170 cution machine.
3171 cron_window = <ClassAd Integer Expression>
3177 Analogous to the submit command deferral_window. It allows cron jobs
3179 that miss their deferral time to begin execution.
3180 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3182 tion has further details.
3183 dagman_log = <pathname>
3189 DAGMan inserts this command to specify an event log that it watches
3191 to maintain the state of the DAG. If the logcommand is not specified
3192 in the submit file, DAGMan uses the logcommand to specify the event
3193 log.
3194 deferral_prep_time = <ClassAd Integer Expression>
3200 The number of seconds prior to a job's deferral time that the job
3202 may be matched and sent to an execution machine.
3203 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3205 tion has further details.
3206 deferral_time = <ClassAd Integer Expression>
3212 Allows a job to specify the time at which its execution is to begin,
3214 instead of beginning execution as soon as it arrives at the execu‐
3215 tion machine. The deferral time is an expression that evaluates to a
3216 Unix Epoch timestamp (the number of seconds elapsed since 00:00:00
3217 on January 1, 1970, Coordinated Universal Time). Deferral time is
3218 evaluated with respect to the execution machine. This option delays
3219 the start of execution, but not the matching and claiming of a
3220 machine for the job. If the job is not available and ready to begin
3221 execution at the deferral time, it has missed its deferral time. A
3222 job that misses its deferral time will be put on hold in the queue.
3223 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3225 tion has further details.
3226 Due to implementation details, a deferral time may not be used for
3228 scheduler universe jobs.
3229 deferral_window = <ClassAd Integer Expression>
3235 The deferral window is used in conjunction with the defer‐
3237 ral_timecommand to allow jobs that miss their deferral time to begin
3238 execution.
3239 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3241 tion has further details.
3242 description = <string>
3248 A string that sets the value of the job ClassAd attribute JobDe‐
3250 scription. When set, tools which display the executable such as con‐
3251 dor_qwill instead use this string.
3252 email_attributes = <list-of-job-ad-attributes>
3258 A comma-separated list of attributes from the job ClassAd. These
3260 attributes and their values will be included in the e-mail notifica‐
3261 tion of job completion.
3262 image_size = <size>
3268 Advice to HTCondor specifying the maximum virtual image size to
3270 which the job will grow during its execution. HTCondor will then
3271 execute the job only on machines which have enough resources, (such
3272 as virtual memory), to support executing the job. If not specified,
3273 HTCondor will automatically make a (reasonably accurate) estimate
3274 about the job's size and adjust this estimate as the program runs.
3275 If specified and underestimated, the job may crash due to the
3276 inability to acquire more address space; for example, if mal‐
3277 loc()fails. If the image size is overestimated, HTCondor may have
3278 difficulty finding machines which have the required resources.
3279 sizeis specified in KiB. For example, for an image size of 8 MiB,
3280 sizeshould be 8000.
3281 initialdir = <directory-path>
3287 Used to give jobs a directory with respect to file input and output.
3289 Also provides a directory (on the machine from which the job is sub‐
3290 mitted) for the job event log, when a full path is not specified.
3291 For vanilla universe jobs where there is a shared file system, it is
3293 the current working directory on the machine where the job is exe‐
3294 cuted.
3295 For vanilla or grid universe jobs where file transfer mechanisms are
3297 utilized (there is nota shared file system), it is the directory on
3298 the machine from which the job is submitted where the input files
3299 come from, and where the job's output files go to.
3300 For standard universe jobs, it is the directory on the machine from
3302 which the job is submitted where the condor_shadowdaemon runs; the
3303 current working directory for file input and output accomplished
3304 through remote system calls.
3305 For scheduler universe jobs, it is the directory on the machine from
3307 which the job is submitted where the job runs; the current working
3308 directory for file input and output with respect to relative path
3309 names.
3310 Note that the path to the executable is notrelative to initialdir;
3312 if it is a relative path, it is relative to the directory in which
3313 the condor_submitcommand is run.
3314 job_ad_information_attrs = <attribute-list>
3320 A comma-separated list of job ClassAd attribute names. The named
3322 attributes and their values are written to the job event log when‐
3323 ever any event is being written to the log. This implements the same
3324 thing as the configuration variable EVENT_LOG_INFORMATION_ATTRS(see
3325 page ), but it applies to the job event log, instead of the system
3326 event log.
3327 JobBatchName = <batch_name>
3333 Set the batch name for this submit. The batch name is displayed by
3335 condor_q-batch. It is intended for use by users to give meaningful
3336 names to their jobs and to influence how condor_qgroups jobs for
3337 display. This value in a submit file can be overridden by specifying
3338 the -batch-nameargument on the condor_submitcommand line.
3339 job_lease_duration = <number-of-seconds>
3345 For vanilla, parallel, VM, and java universe jobs only, the duration
3347 in seconds of a job lease. The default value is 2,400, or forty min‐
3348 utes. If a job lease is not desired, the value can be explicitly set
3349 to 0 to disable the job lease semantics. The value can also be a
3350 ClassAd expression that evaluates to an integer. The HTCondor User's
3351 manual section on Special Environment Considerations has further
3352 details.
3353 job_machine_attrs = <attr1, attr2, ...>
3359 A comma and/or space separated list of machine attribute names that
3361 should be recorded in the job ClassAd in addition to the ones speci‐
3362 fied by the condor_schedddaemon's system configuration variable SYS‐
3363 TEM_JOB_MACHINE_ATTRS. When there are multiple run attempts, history
3364 of machine attributes from previous run attempts may be kept. The
3365 number of run attempts to store may be extended beyond the system-
3366 specified history length by using the submit file command
3367 job_machine_attrs_history_length. A machine attribute named Xwill be
3368 inserted into the job ClassAd as an attribute named MachineAttrX0.
3369 The previous value of this attribute will be named MachineAttrX1,
3370 the previous to that will be named MachineAttrX2, and so on, up to
3371 the specified history length. A history of length 1 means that only
3372 MachineAttrX0will be recorded. The value recorded in the job ClassAd
3373 is the evaluation of the machine attribute in the context of the job
3374 ClassAd when the condor_schedddaemon initiates the start up of the
3375 job. If the evaluation results in an Undefinedor Errorresult, the
3376 value recorded in the job ad will be Undefinedor Error, respec‐
3377 tively.
3378 want_graceful_removal = <boolean expression>
3384 When True, this causes a graceful shutdown of the job when the job
3386 is removed or put on hold, giving it time to clean up or save state.
3387 Otherwise, the job is abruptly killed. The default is false.
3388 kill_sig = <signal-number>
3394 When HTCondor needs to kick a job off of a machine, it will send the
3396 job the signal specified by signal-number. signal-numberneeds to be
3397 an integer which represents a valid signal on the execution machine.
3398 For jobs submitted to the standard universe, the default value is
3399 the number for SIGTSTP which tells the HTCondor libraries to initi‐
3400 ate a checkpoint of the process. For jobs submitted to other uni‐
3401 verses, the default value, when not defined, is SIGTERM , which is
3402 the standard way to terminate a program in Unix.
3403 kill_sig_timeout = <seconds>
3409 This submit command should no longer be used as of HTCondor version
3411 7.7.3; use job_max_vacate_timeinstead. If job_max_vacate_timeis not
3412 defined, this defines the number of seconds that HTCondor should
3413 wait following the sending of the kill signal defined by kill_sigand
3414 forcibly killing the job. The actual amount of time between sending
3415 the signal and forcibly killing the job is the smallest of this
3416 value and the configuration variable KILLING_TIMEOUT, as defined on
3417 the execute machine.
3418 load_profile = <True | False>
3424 When True, loads the account profile of the dedicated run account
3426 for Windows jobs. May not be used with run_as_owner.
3427 match_list_length = <integer value>
3433 Defaults to the value zero (0). When match_list_lengthis defined
3435 with an integer value greater than zero (0), attributes are inserted
3436 into the job ClassAd. The maximum number of attributes defined is
3437 given by the integer value. The job ClassAds introduced are given as
3438 LastMatchName0 = "most-recent-Name"
3440 LastMatchName1 = "next-most-recent-Name"
3441 The value for each introduced ClassAd is given by the value of the
3443 Nameattribute from the machine ClassAd of a previous execution
3444 (match). As a job is matched, the definitions for these attributes
3445 will roll, with LastMatchName1 becoming LastMatchName2 , Last‐
3446 MatchName0 becoming LastMatchName1 , and LastMatchName0 being set
3447 by the most recent value of the Nameattribute.
3448 An intended use of these job attributes is in the requirements
3450 expression. The requirements can allow a job to prefer a match with
3451 either the same or a different resource than a previous match.
3452 job_max_vacate_time = <integer expression>
3458 An integer-valued expression (in seconds) that may be used to adjust
3460 the time given to an evicted job for gracefully shutting down. If
3461 the job's setting is less than the machine's, the job's is used. If
3462 the job's setting is larger than the machine's, the result depends
3463 on whether the job has any excess retirement time. If the job has
3464 more retirement time left than the machine's max vacate time set‐
3465 ting, then retirement time will be converted into vacating time, up
3466 to the amount requested by the job.
3467 Setting this expression does not affect the job's resource require‐
3469 ments or preferences. For a job to only run on a machine with a min‐
3470 imum MachineMaxVacateTime, or to preferentially run on such
3471 machines, explicitly specify this in the requirements and/or rank
3472 expressions.
3473 max_job_retirement_time = <integer expression>
3479 An integer-valued expression (in seconds) that does nothing unless
3481 the machine that runs the job has been configured to provide retire‐
3482 ment time. Retirement time is a grace period given to a job to fin‐
3483 ish when a resource claim is about to be preempted. The default
3484 behavior in many cases is to take as much retirement time as the
3485 machine offers, so this command will rarely appear in a submit
3486 description file.
3487 When a resource claim is to be preempted, this expression in the
3489 submit file specifies the maximum run time of the job (in seconds,
3490 since the job started). This expression has no effect, if it is
3491 greater than the maximum retirement time provided by the machine
3492 policy. If the resource claim is notpreempted, this expression and
3493 the machine retirement policy are irrelevant. If the resource claim
3494 ispreempted the job will be allowed to run until the retirement time
3495 expires, at which point it is hard-killed. The job will be soft-
3496 killed when it is getting close to the end of retirement in order to
3497 give it time to gracefully shut down. The amount of lead-time for
3498 soft-killing is determined by the maximum vacating time granted to
3499 the job.
3500 Standard universe jobs and any jobs running with nice_userpriority
3502 have a default max_job_retirement_timeof 0, so no retirement time is
3503 utilized by default. In all other cases, no default value is pro‐
3504 vided, so the maximum amount of retirement time is utilized by
3505 default.
3506 Setting this expression does not affect the job's resource require‐
3508 ments or preferences. For a job to only run on a machine with a min‐
3509 imum MaxJobRetirementTime, or to preferentially run on such
3510 machines, explicitly specify this in the requirements and/or rank
3511 expressions.
3512 nice_user = <True | False>
3518 Normally, when a machine becomes available to HTCondor, HTCondor
3520 decides which job to run based upon user and job priorities. Setting
3521 nice_userequal to Truetells HTCondor not to use your regular user
3522 priority, but that this job should have last priority among all
3523 users and all jobs. So jobs submitted in this fashion run only on
3524 machines which no other non-nice_user job wants -- a true bottom-
3525 feeder job! This is very handy if a user has some jobs they wish to
3526 run, but do not wish to use resources that could instead be used to
3527 run other people's HTCondor jobs. Jobs submitted in this fashion
3528 have "nice-user."prepended to the owner name when viewed from con‐
3529 dor_qor condor_userprio. The default value is False.
3530 noop_job = <ClassAd Boolean Expression>
3536 When this boolean expression is True, the job is immediately removed
3538 from the queue, and HTCondor makes no attempt at running the job.
3539 The log file for the job will show a job submitted event and a job
3540 terminated event, along with an exit code of 0, unless the user
3541 specifies a different signal or exit code.
3542 noop_job_exit_code = <return value>
3548 When noop_jobis in the submit description file and evaluates to
3550 True, this command allows the job to specify the return value as
3551 shown in the job's log file job terminated event. If not specified,
3552 the job will show as having terminated with status 0. This overrides
3553 any value specified with noop_job_exit_signal.
3554 noop_job_exit_signal = <signal number>
3560 When noop_jobis in the submit description file and evaluates to
3562 True, this command allows the job to specify the signal number that
3563 the job's log event will show the job having terminated with.
3564 remote_initialdir = <directory-path>
3570 The path specifies the directory in which the job is to be executed
3572 on the remote machine. This is currently supported in all universes
3573 except for the standard universe.
3574 rendezvousdir = <directory-path>
3580 Used to specify the shared file system directory to be used for file
3582 system authentication when submitting to a remote scheduler. Should
3583 be a path to a preexisting directory.
3584 run_as_owner = <True | False>
3590 A boolean value that causes the job to be run under the login of the
3592 submitter, if supported by the joint configuration of the submit and
3593 execute machines. On Unix platforms, this defaults to True, and on
3594 Windows platforms, it defaults to False. May not be used with
3595 load_profile. See the HTCondor manual Platform-Specific Information
3596 chapter for administrative details on configuring Windows to support
3597 this option.
3598 stack_size = <size in bytes>
3604 This command applies only to Linux platform jobs that are not stan‐
3606 dard universe jobs. An integer number of bytes, representing the
3607 amount of stack space to be allocated for the job. This value
3608 replaces the default allocation of stack space, which is unlimited
3609 in size.
3610 submit_event_notes = <note>
3616 A string that is appended to the submit event in the job's log file.
3618 For DAGMan jobs, the string DAG Node:and the node's name is automat‐
3619 ically defined for submit_event_notes, causing the logged submit
3620 event to identify the DAG node job submitted.
3621 +<attribute> = <value>
3627 A line that begins with a '+' (plus) character instructs condor_sub‐
3629 mitto insert the given attributeinto the job ClassAd with the given
3630 value. Note that setting an attribute should not be used in place of
3631 one of the specific commands listed above. Often, the command name
3632 does not directly correspond to an attribute name; furthermore, many
3633 submit commands result in actions more complex than simply setting
3634 an attribute or attributes. See for a list of HTCondor job
3635 attributes.
3636 MACROS AND COMMENTS
3642 In addition to commands, the submit description file can contain macros
3644 and comments.
3645 Macros
3647 Parameterless macros in the form of $(macro_name:default initial
3649 value)may be used anywhere in HTCondor submit description files to
3650 provide textual substitution at submit time. Macros can be defined
3651 by lines in the form of
3652 <macro_name> = <string>
3654 Two pre-defined macros are supplied by the submit description file
3656 parser. The $(Cluster)or $(ClusterId)macro supplies the value of the
3657 ClusterIdjob ClassAd attribute, and the $(Process)or $(ProcId)macro
3658 supplies the value of the ProcIdjob ClassAd attribute. These macros
3659 are intended to aid in the specification of input/output files,
3660 arguments, etc., for clusters with lots of jobs, and/or could be
3661 used to supply an HTCondor process with its own cluster and process
3662 numbers on the command line.
3663 The $(Node)macro is defined for parallel universe jobs, and is espe‐
3665 cially relevant for MPI applications. It is a unique value assigned
3666 for the duration of the job that essentially identifies the machine
3667 (slot) on which a program is executing. Values assigned start at 0
3668 and increase monotonically. The values are assigned as the parallel
3669 job is about to start.
3670 Recursive definition of macros is permitted. An example of a con‐
3672 struction that works is the following:
3673 foo = bar
3675 foo = snap $(foo)
3676 As a result, foo = snap bar.
3678 Note that both left- and right- recursion works, so
3680 foo = bar
3682 foo = $(foo) snap
3683 has as its result foo = bar snap.
3685 The construction
3687 foo = $(foo) bar
3689 by itself will notwork, as it does not have an initial base case.
3691 Mutually recursive constructions such as:
3692 B = bar
3694 C = $(B)
3695 B = $(C) boo
3696 will notwork, and will fill memory with expansions.
3698 A default value may be specified, for use if the macro has no defi‐
3700 nition. Consider the example
3701 D = $(E:24)
3703 Where Eis not defined within the submit description file, the
3705 default value 24 is used, resulting in
3706 D = 24
3708 This is of limited value, as the scope of macro substitution is the
3710 submit description file. Thus, either the macro is or is not defined
3711 within the submit description file. If the macro is defined, then
3712 the default value is useless. If the macro is not defined, then
3713 there is no point in using it in a submit command.
3714 To use the dollar sign character ( $ ) as a literal, without macro
3716 expansion, use
3717 $(DOLLAR)
3719 In addition to the normal macro, there is also a special kind of
3721 macro called a substitution macrothat allows the substitution of a
3722 machine ClassAd attribute value defined on the resource machine
3723 itself (gotten after a match to the machine has been made) into spe‐
3724 cific commands within the submit description file. The substitution
3725 macro is of the form:
3726 $$(attribute)
3728 As this form of the substitution macro is only evaluated within the
3730 context of the machine ClassAd, use of a scope resolution prefix
3731 TARGET.or MY.is not allowed.
3732 A common use of this form of the substitution macro is for the het‐
3734 erogeneous submission of an executable:
3735 executable = povray.$$(OpSys).$$(Arch)
3737 Values for the OpSysand Archattributes are substituted at match time
3739 for any given resource. This example allows HTCondor to automati‐
3740 cally choose the correct executable for the matched machine.
3741 An extension to the syntax of the substitution macro provides an
3743 alternative string to use if the machine attribute within the sub‐
3744 stitution macro is undefined. The syntax appears as:
3745 $$(attribute:string_if_attribute_undefined)
3747 An example using this extended syntax provides a path name to a
3749 required input file. Since the file can be placed in different loca‐
3750 tions on different machines, the file's path name is given as an
3751 argument to the program.
3752 arguments = $$(input_file_path:/usr/foo)
3754 On the machine, if the attribute input_file_pathis not defined, then
3756 the path /usr/foois used instead.
3757 A further extension to the syntax of the substitution macro allows
3759 the evaluation of a ClassAd expression to define the value. In this
3760 form, the expression may refer to machine attributes by prefacing
3761 them with the TARGET.scope resolution prefix. To place a ClassAd
3762 expression into the substitution macro, square brackets are added to
3763 delimit the expression. The syntax appears as:
3764 $$([ClassAd expression])
3766 An example of a job that uses this syntax may be one that wants to
3768 know how much memory it can use. The application cannot detect this
3769 itself, as it would potentially use all of the memory on a multi-
3770 slot machine. So the job determines the memory per slot, reducing it
3771 by 10% to account for miscellaneous overhead, and passes this as a
3772 command line argument to the application. In the submit description
3773 file will be
3774 arguments = --memory $$([TARGET.Memory * 0.9])
3776 To insert two dollar sign characters ( $$ ) as literals into a Clas‐
3778 sAd string, use
3779 $$(DOLLARDOLLAR)
3781 The environment macro, $ENV, allows the evaluation of an environment
3783 variable to be used in setting a submit description file command.
3784 The syntax used is
3785 $ENV(variable)
3787 An example submit description file command that uses this function‐
3789 ality evaluates the submitter's home directory in order to set the
3790 path and file name of a log file:
3791 log = $ENV(HOME)/jobs/logfile
3793 The environment variable is evaluated when the submit description
3795 file is processed.
3796 The $RANDOM_CHOICE macro allows a random choice to be made from a
3798 given list of parameters at submission time. For an expression, if
3799 some randomness needs to be generated, the macro may appear as
3800 $RANDOM_CHOICE(0,1,2,3,4,5,6)
3802 When evaluated, one of the parameters values will be chosen.
3804 Comments
3810 Blank lines and lines beginning with a pound sign ('#') character
3812 are ignored by the submit description file parser.
3813
3819 While processing the queuecommand in a submit file or from the command
3820 line, condor_submitwill set the values of several automatic submit
3821 variables so that they can be referred to by statements in the submit
3822 file. With the exception of Cluster and Process, if these variables are
3823 set by the submit file, they will not be modified during queueprocess‐
3824 ing.
3825 ClusterId
3827 Set to the integer value that the ClusterIdattribute that the job
3829 ClassAd will have when the job is submitted. All jobs in a single
3830 submit will normally have the same value for the ClusterId. If the
3831 -dry-runargument is specified, The value will be 1.
3832 Cluster
3838 Alternate name for the ClusterId submit variable. Before HTCondor
3840 version 8.4 this was the only name.
3841 ProcId
3847 Set to the integer value that the ProcIdattribute of the job ClassAd
3849 will have when the job is submitted. The value will start at 0 and
3850 increment by 1 for each job submitted.
3851 Process
3857 Alternate name for the ProcId submit variable. Before HTCondor ver‐
3859 sion 8.4 this was the only name.
3860 Node
3866 For parallel universes, set to the value #pArAlLeLnOdE# or #MpInOdE#
3868 depending on the parallel universe type For other universes it is
3869 set to nothing.
3870 Step
3876 Set to the step value as it varies from 0 to N-1 where N is the num‐
3878 ber provided on the queueargument. This variable changes at the same
3879 rate as ProcId when it changes at all. For submit files that don't
3880 make use of the queue number option, Step will always be 0. For sub‐
3881 mit files that don't make use of any of the foreach options, Step
3882 and ProcId will always be the same.
3883 ItemIndex
3889 Set to the index within the item list being processed by the various
3891 queue foreach options. For submit files that don't make use of any
3892 queue foreach list, ItemIndex will always be 0 For submit files that
3893 make use of a slice to select only some items in a foreach list,
3894 ItemIndex will only be set to selected values.
3895 Row
3901 Alternate name for ItemIndex.
3903 Item
3909 when a queue foreach option is used and no variable list is sup‐
3911 plied, this variable will be set to the value of the current item.
3912 The automatic variables below are set before parsing the submit file,
3916 and will not vary during processing unless the submit file itself sets
3917 them.
3918 ARCH
3920 Set to the CPU architecture of the machine running condor_submit.
3922 The value will be the same as the automatic configuration variable
3923 of the same name.
3924 OPSYS
3930 Set to the name of the operating system on the machine running con‐
3932 dor_submit. The value will be the same as the automatic configura‐
3933 tion variable of the same name.
3934 OPSYSANDVER
3940 Set to the name and major version of the operating system on the
3942 machine running condor_submit. The value will be the same as the
3943 automatic configuration variable of the same name.
3944 OPSYSMAJORVER
3950 Set to the major version of the operating system on the machine run‐
3952 ning condor_submit. The value will be the same as the automatic con‐
3953 figuration variable of the same name.
3954 OPSYSVER
3960 Set to the version of the operating system on the machine running
3962 condor_submit. The value will be the same as the automatic configu‐
3963 ration variable of the same name.
3964 SPOOL
3970 Set to the full path of the HTCondor spool directory. The value will
3972 be the same as the automatic configuration variable of the same
3973 name.
3974 IsLinux
3980 Set to true if the operating system of the machine running con‐
3982 dor_submitis a Linux variant. Set to false otherwise.
3983 IsWindows
3989 Set to true if the operating system of the machine running con‐
3991 dor_submitis a Microsoft Windows variant. Set to false otherwise.
3992 SUBMIT_FILE
3998 Set to the full pathname of the submit file being processed by con‐
4000 dor_submit. If submit statements are read from standard input, it is
4001 set to nothing.
4002
4006 condor_submitwill exit with a status value of 0 (zero) upon success,
4007 and a non-zero value upon failure.
4008
4010 * Submit Description File Example 1: This example queues three jobs
4011 for execution by HTCondor. The first will be given command line
4012 arguments of 15and 2000, and it will write its standard output to
4013 foo.out1. The second will be given command line arguments of 30and
4014 2000, and it will write its standard output to foo.out2. Similarly
4015 the third will have arguments of 45and 6000, and it will use
4016 foo.out3for its standard output. Standard error output (if any) from
4017 all three programs will appear in foo.error.
4018 ####################
4022 #
4023 # submit description file
4024 # Example 1: queuing multiple jobs with differing
4025 # command line arguments and output files.
4026 #
4027 ####################
4028 Executable = foo
4030 Universe = vanilla
4031 Arguments = 15 2000
4033 Output = foo.out0
4034 Error = foo.err0
4035 Queue
4036 Arguments = 30 2000
4038 Output = foo.out1
4039 Error = foo.err1
4040 Queue
4041 Arguments = 45 6000
4043 Output = foo.out2
4044 Error = foo.err2
4045 Queue
4046 Or you can get the same results as the above submit file by using a
4048 list of arguments with the Queue statement
4049 ####################
4053 #
4054 # submit description file
4055 # Example 1b: queuing multiple jobs with differing
4056 # command line arguments and output files, alternate syntax
4057 #
4058 ####################
4059 Executable = foo
4061 Universe = vanilla
4062 # generate different output and error filenames for each process
4064 Output = foo.out$(Process)
4065 Error = foo.err$(Process)
4066 Queue Arguments From (
4068 15 2000
4069 30 2000
4070 45 6000
4071 )
4072 * Submit Description File Example 2: This submit description file
4076 example queues 150 runs of program foowhich must have been compiled
4077 and linked for an Intel x86 processor running RHEL 3. HTCondor will
4078 not attempt to run the processes on machines which have less than 32
4079 Megabytes of physical memory, and it will run them on machines which
4080 have at least 64 Megabytes, if such machines are available. Stdin,
4081 stdout, and stderr will refer to in.0, out.0, and err.0for the first
4082 run of this program (process 0). Stdin, stdout, and stderr will
4083 refer to in.1, out.1, and err.1for process 1, and so forth. A log
4084 file containing entries about where and when HTCondor runs, takes
4085 checkpoints, and migrates processes in this cluster will be written
4086 into file foo.log.
4087 ####################
4091 #
4092 # Example 2: Show off some fancy features including
4093 # use of pre-defined macros and logging.
4094 #
4095 ####################
4096 Executable = foo
4098 Universe = standard
4099 Requirements = OpSys == "LINUX" && Arch =="INTEL"
4100 Rank = Memory >= 64
4101 Request_Memory = 32 Mb
4102 Image_Size = 28 Mb
4103 Error = err.$(Process)
4105 Input = in.$(Process)
4106 Output = out.$(Process)
4107 Log = foo.log
4108 Queue 150
4109 * Submit Description File Example 3: This example targets the
4113 /bin/sleepprogram to run only on a platform running a RHEL 6 operat‐
4114 ing system. The example presumes that the pool contains machines
4115 running more than one version of Linux, and this job needs the par‐
4116 ticular operating system to run correctly.
4117 ####################
4121 #
4122 # Example 3: Run on a RedHat 6 machine
4123 #
4124 ####################
4125 Universe = vanilla
4126 Executable = /bin/sleep
4127 Arguments = 30
4128 Requirements = (OpSysAndVer == "RedHat6")
4129 Error = err.$(Process)
4131 Input = in.$(Process)
4132 Output = out.$(Process)
4133 Log = sleep.log
4134 Queue
4135 * Command Line example: The following command uses the -appendoption
4139 to add two commands before the job(s) is queued. A log file and an
4140 error log file are specified. The submit description file is
4141 unchanged.
4142 condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
4144 Note that each of the added commands is contained within quote marks
4146 because there are space characters within the command.
4147 * periodic_removeexample: A job should be removed from the queue, if
4151 the total suspension time of the job is more than half of the run
4152 time of the job.
4153 Including the command
4155 periodic_remove = CumulativeSuspensionTime >
4157 ((RemoteWallClockTime - CumulativeSuspensionTime)
4158 / 2.0)
4159 in the submit description file causes this to happen.
4161
4165 * For security reasons, HTCondor will refuse to run any jobs submit‐
4166 ted by user root (UID = 0) or by a user whose default group is group
4167 wheel (GID = 0). Jobs submitted by user root or a user with a
4168 default group of wheel will appear to sit forever in the queue in an
4169 idle state.
4170 * All path names specified in the submit description file must be
4174 less than 256 characters in length, and command line arguments must
4175 be less than 4096 characters in length; otherwise, condor_submit‐
4176 gives a warning message but the jobs will not execute properly.
4177 * Somewhat understandably, behavior gets bizarre if the user makes
4181 the mistake of requesting multiple HTCondor jobs to write to the
4182 same file, and/or if the user alters any files that need to be
4183 accessed by an HTCondor job which is still in the queue. For exam‐
4184 ple, the compressing of data or output files before an HTCondor job
4185 has completed is a common mistake.
4186 * To disable checkpointing for Standard Universe jobs, include the
4190 line:
4191 +WantCheckpoint = False
4193 in the submit description file before the queue command(s).
4195
4197 HTCondor User Manual
4198
4200 Center for High Throughput Computing, University of Wisconsin-Madison
4201
4203 Copyright (C) 1990-2019 Center for High Throughput Computing, Computer
4204 Sciences Department, University of Wisconsin-Madison, Madison, WI. All
4205 Rights Reserved. Licensed under the Apache License, Version 2.0.
4206 date condor_submit(1)