1condor_submit(1) General Commands Manual condor_submit(1)
2
6 condor_submitQueue jobs for execution under HTCondor
7
9 condor_submit[-terse] [-verbose] [-unused] [-file submit_file] [-name
10 schedd_name] [-remote schedd_name] [-addr <ip:port>] [-pool pool_name]
11 [-disable] [-password passphrase] [-debug] [-appendcommand...] [-batch-
12 name batch_name] [-spool] [-dump filename] [-interactive] [-allow-crlf-
13 script] [-dry-run] [-maxjobs number-of-jobs] [-single-cluster] [-stm
14 method] [<submit-variable>=<value>] [submit description file] [-queue
15 queue_arguments]
16
18 condor_submitis the program for submitting jobs for execution under
19 HTCondor. condor_submitrequires one or more submit description commands
20 to direct the queuing of jobs. These commands may come from a file,
21 standard input, the command line, or from some combination of these.
22 One submit description may contain specifications for the queuing of
23 many HTCondor jobs at once. A single invocation of condor_submitmay
24 cause one or more clusters. A cluster is a set of jobs specified in the
25 submit description between queuecommands for which the executable is
26 not changed. It is advantageous to submit multiple jobs as a single
27 cluster because:
28 * Much less memory is used by the scheduler to hold the same number
30 of jobs.
31 * Only one copy of the checkpoint file is needed to represent all
33 jobs in a cluster until they begin execution.
34 * There is much less overhead involved for HTCondor to start the
36 next job in a cluster than for HTCondor to start a new cluster. This
37 can make a big difference when submitting lots of short jobs.
38 Multiple clusters may be specified within a single submit description.
40 Each cluster must specify a single executable.
41 The job ClassAd attribute ClusterIdidentifies a cluster.
43 The submit description fileargument is the path and file name of the
45 submit description file. If this optional argument is the dash charac‐
46 ter (-), then the commands are taken from standard input. If -is speci‐
47 fied for the submit description file, -verboseis implied; this can be
48 overridden by specifying -terse.
49 If no submit discription fileargument is given, and no -queueargument
51 is given, commands are taken automatically from standard input.
52 Note that submission of jobs from a Windows machine requires a stashed
54 password to allow HTCondor to impersonate the user submitting the job.
55 To stash a password, use the condor_store_credcommand. See the manual
56 page for details.
57 For lengthy lines within the submit description file, the backslash ( \
59 ) is a line continuation character. Placing the backslash at the end of
60 a line causes the current line's command to be continued with the next
61 line of the file. Submit description files may contain comments. A com‐
62 ment is any line beginning with a pound character ( # ).
63
65 -terse
66 Terse output - display JobId ranges only.
68 -verbose
74 Verbose output - display the created job ClassAd
76 -unused
82 As a default, causes no warnings to be issued about user-defined
84 macros not being used within the submit description file. The mean‐
85 ing reverses (toggles) when the configuration variable
86 WARN_ON_UNUSED_SUBMIT_FILE_MACROSis set to the non default value of
87 False. Printing the warnings can help identify spelling errors of
88 submit description file commands. The warnings are sent to stderr.
89 -file submit_file
95 Use submit_fileas the submit discription file. This is equivalent to
97 providing submit_fileas an argument without the preceeding -file.
98 -name schedd_name
104 Submit to the specified condor_schedd. Use this option to submit to
106 a condor_scheddother than the default local one. schedd_nameis the
107 value of the NameClassAd attribute on the machine where the con‐
108 dor_schedddaemon runs.
109 -remote schedd_name
115 Submit to the specified condor_schedd, spooling all required input
117 files over the network connection. schedd_nameis the value of the
118 NameClassAd attribute on the machine where the condor_schedddaemon
119 runs. This option is equivalent to using both -nameand -spool.
120 -addr <ip:port>
126 Submit to the condor_scheddat the IP address and port given by the
128 sinful stringargument <ip:port>.
129 -pool pool_name
135 Look in the specified pool for the condor_scheddto submit to. This
137 option is used with -nameor -remote.
138 -disable
144 Disable file permission checks when submitting a job for read per‐
146 missions on all input files, such as those defined by commands
147 inputand transfer_input_files, as well as write permission to output
148 files, such as a log file defined by logand output files defined
149 with outputor transfer_output_files.
150 -password passphrase
156 Specify a password to the MyProxyserver.
158 -debug
164 Cause debugging information to be sent to stderr, based on the value
166 of the configuration variable TOOL_DEBUG.
167 -append command
173 Augment the commands in the submit description file with the given
175 command. This command will be considered to immediately precede the
176 queuecommand within the submit description file, and come after all
177 other previous commands. If the commandspecifies a queuecommand, as
178 in the example
179 condor_submit mysubmitfile -append "queue input in A, B, C"
181 then the entire -appendcommand line option and its arguments are
183 converted to
184 condor_submit mysubmitfile -queue input in A, B, C
186 The submit description file is not modified. Multiple commands are
188 specified by using the -appendoption multiple times. Each new com‐
189 mand is given in a separate -appendoption. Commands with spaces in
190 them will need to be enclosed in double quote marks.
191 -batch-name batch_name
197 Set the batch name for this submit. The batch name is displayed by
199 condor_q-batch. It is intended for use by users to give meaningful
200 names to their jobs and to influence how condor_qgroups jobs for
201 display. Use of this argument takes precedence over a batch name
202 specified in the submit description file itself.
203 -spool
209 Spool all required input files, job event log, and proxy over the
211 connection to the condor_schedd. After submission, modify local
212 copies of the files without affecting your jobs. Any output files
213 for completed jobs need to be retrieved with condor_transfer_data.
214 -dump filename
220 Sends all ClassAds to the specified file, instead of to the con‐
222 dor_schedd.
223 -interactive
229 Indicates that the user wants to run an interactive shell on an exe‐
231 cute machine in the pool. This is equivalent to creating a submit
232 description file of a vanilla universe sleep job, and then running
233 condor_ssh_to_jobby hand. Without any additional arguments, con‐
234 dor_submitwith the -interactive flag creates a dummy vanilla uni‐
235 verse job that sleeps, submits it to the local scheduler, waits for
236 the job to run, and then launches condor_ssh_to_jobto run a shell.
237 If the user would like to run the shell on a machine that matches a
238 particular requirementsexpression, the submit description file is
239 specified, and it will contain the expression. Note that all policy
240 expressions specified in the submit description file are honored,
241 but any executableor universecommands are overwritten to be sleep
242 and vanilla. The job ClassAd attribute InteractiveJobis set to
243 Trueto identify interactive jobs for condor_startdpolicy usage.
244 -allow-crlf-script
250 Changes the check for an invalid line ending on the executable
252 script's #!line from an ERROR to a WARNING. The #!line will be
253 ignored by Windows, so it won't matter if it is invalid; but Unix
254 and Linux will not run a script that has a Windows/DOS line ending
255 on the first line of the script. So condor_submitwill not allow such
256 a script to be submitted as the job's executable unless this option
257 is supplied.
258 -dry-run file
264 Parse the submit description file, sending the resulting job ClassAd
266 to the file given by file, but do not submit the job(s). This per‐
267 mits observation of the job specification, and it facilitates debug‐
268 ging the submit description file contents. If fileis -, the output
269 is written to stdout.
270 -maxjobs number-of-jobs
276 If the total number of jobs specified by the submit description file
278 is more than the integer value given by number-of-jobs, then no jobs
279 are submitted for execution and an error message is generated. A 0
280 or negative value for the number-of-jobscauses no limit to be
281 imposed.
282 -single-cluster
288 If the jobs specified by the submit description file causes more
290 than a single cluster value to be assigned, then no jobs are submit‐
291 ted for execution and an error message is generated.
292 -stm method
298 Specify the method use to move a sandbox into HTCondor. methodis one
300 of stm_use_schedd_onlyor stm_use_transferd.
301 <submit-variable>=<value>
307 Defines a submit command or submit variable with a value, and parses
309 it as if it was placed at the beginning of the submit description
310 file. The submit description file is not changed. To correctly parse
311 the condor_submitcommand line, this option must be specified without
312 white space characters before and after the equals sign (=), or the
313 entire option must be surrounded by double quote marks.
314 -queue queue_arguments
320 A command line specification of how many jobs to queue, which is
322 only permitted if the submit description file does not have a
323 queuecommand. The queue_argumentsare the same as may be within a
324 submit description file. The parsing of the queue_argumentsfinishes
325 at the end of the line or when a dash character (-) is encountered.
326 Therefore, its best placement within the command line will be at the
327 end of the command line.
328 On a Unix command line, the shell expands file globs before parsing
330 occurs.
331
337 Note: more information on submitting HTCondor jobs can be found here: .
338 As of version 8.5.6, the condor_submitlanguage supports multi-line val‐
340 ues in commands. The syntax is the same as the configuration language
341 (see more details here: ).
342 Each submit description file describes one or more clusters of jobs to
344 be placed in the HTCondor execution pool. All jobs in a cluster must
345 share the same executable, but they may have different input and output
346 files, and different program arguments. The submit description file is
347 generally the last command-line argument to condor_submit. If the sub‐
348 mit description file argument is omitted, condor_submitwill read the
349 submit description from standard input.
350 The submit description file must contain at least one executablecommand
352 and at least one queuecommand. All of the other commands have default
353 actions.
354 Note that a submit file that contains more than one executable command
356 will produce multiple clusters when submitted. This is not generally
357 recommended, and is not allowed for submit files that are run as DAG
358 node jobs by condor_dagman.
359 The commands which can appear in the submit description file are numer‐
361 ous. They are listed here in alphabetical order by category.
362 BASIC COMMANDS
364 arguments = <argument_list>
372 List of arguments to be supplied to the executable as part of the
374 command line.
375 In the javauniverse, the first argument must be the name of the
377 class containing main.
378 There are two permissible formats for specifying arguments, identi‐
380 fied as the old syntax and the new syntax. The old syntax supports
381 white space characters within arguments only in special circum‐
382 stances; when used, the command line arguments are represented in
383 the job ClassAd attribute Args. The new syntax supports uniform
384 quoting of white space characters within arguments; when used, the
385 command line arguments are represented in the job ClassAd attribute
386 Arguments.
387 Old Syntax
389 In the old syntax, individual command line arguments are delimited
391 (separated) by space characters. To allow a double quote mark in an
392 argument, it is escaped with a backslash; that is, the two character
393 sequence \" becomes a single double quote mark within an argument.
394 Further interpretation of the argument string differs depending on
396 the operating system. On Windows, the entire argument string is
397 passed verbatim (other than the backslash in front of double quote
398 marks) to the Windows application. Most Windows applications will
399 allow spaces within an argument value by surrounding the argument
400 with double quotes marks. In all other cases, there is no further
401 interpretation of the arguments.
402 Example:
404 arguments = one \"two\" 'three'
408 Produces in Unix vanilla universe:
410 argument 1: one
414 argument 2: "two"
415 argument 3: 'three'
416 New Syntax
418 Here are the rules for using the new syntax:
420 1. The entire string representing the command line arguments is
424 surrounded by double quote marks. This permits the white space
425 characters of spaces and tabs to potentially be embedded within a
426 single argument. Putting the double quote mark within the argu‐
427 ments is accomplished by escaping it with another double quote
428 mark.
429 2. The white space characters of spaces or tabs delimit argu‐
433 ments.
434 3. To embed white space characters of spaces or tabs within a
438 single argument, surround the entire argument with single quote
439 marks.
440 4. To insert a literal single quote mark, escape it within an
444 argument already delimited by single quote marks by adding
445 another single quote mark.
446 Example:
450 arguments = "3 simple arguments"
452 Produces:
454 argument 1: 3
456 argument 2: simple
457 argument 3: arguments
458 Another example:
460 arguments = "one 'two with spaces' 3"
462 Produces:
464 argument 1: one
466 argument 2: two with spaces
467 argument 3: 3
468 And yet another example:
470 arguments = "one ""two"" 'spacey ”quoted” argument'"
472 Produces:
474 argument 1: one
476 argument 2: "two"
477 argument 3: spacey 'quoted' argument
478 Notice that in the new syntax, the backslash has no special meaning.
480 This is for the convenience of Windows users.
481 environment = <parameter_list>
487 List of environment variables.
489 There are two different formats for specifying the environment vari‐
491 ables: the old format and the new format. The old format is retained
492 for backward-compatibility. It suffers from a platform-dependent
493 syntax and the inability to insert some special characters into the
494 environment.
495 The new syntax for specifying environment values:
497 1. Put double quote marks around the entire argument string. This
501 distinguishes the new syntax from the old. The old syntax does
502 not have double quote marks around it. Any literal double quote
503 marks within the string must be escaped by repeating the double
504 quote mark.
505 2. Each environment entry has the form
509 <name>=<value>
513 3. Use white space (space or tab characters) to separate environ‐
517 ment entries.
518 4. To put any white space in an environment entry, surround the
522 space and as much of the surrounding entry as desired with single
523 quote marks.
524 5. To insert a literal single quote mark, repeat the single quote
528 mark anywhere inside of a section surrounded by single quote
529 marks.
530 Example:
534 environment = "one=1 two=""2"" three='spacey ”quoted”
538 value'"
539 Produces the following environment entries:
541 one=1
545 two="2"
546 three=spacey 'quoted' value
547 Under the old syntax, there are no double quote marks surrounding
549 the environment specification. Each environment entry remains of the
550 form
551 <name>=<value>
553 Under Unix, list multiple environment entries by separating them
555 with a semicolon (;). Under Windows, separate multiple entries with
556 a vertical bar (|). There is no way to insert a literal semicolon
557 under Unix or a literal vertical bar under Windows. Note that spaces
558 are accepted, but rarely desired, characters within parameter names
559 and values, because they are treated as literal characters, not sep‐
560 arators or ignored white space. Place spaces within the parameter
561 list only if required.
562 A Unix example:
564 environment = one=1;two=2;three="quotes have no 'special' meaning"
568 This produces the following:
570 one=1
574 two=2
575 three="quotes have no 'special' meaning"
576 If the environment is set with the environmentcommand andgetenvis
578 also set to true, values specified with environmentoverride values
579 in the submitter's environment (regardless of the order of the envi‐
580 ronmentand getenvcommands).
581 error = <pathname>
587 A path and file name used by HTCondor to capture any error messages
589 the program would normally write to the screen (that is, this file
590 becomes stderr). A path is given with respect to the file system of
591 the machine on which the job is submitted. The file is written (by
592 the job) in the remote scratch directory of the machine where the
593 job is executed. When the job exits, the resulting file is trans‐
594 ferred back to the machine where the job was submitted, and the path
595 is utilized for file placement. If not specified, the default value
596 of /dev/nullis used for submission to a Unix machine. If not speci‐
597 fied, error messages are ignored for submission to a Windows
598 machine. More than one job should not use the same error file, since
599 this will cause one job to overwrite the errors of another. If
600 HTCondor detects that the error and output files for a job are the
601 same, it will run the job such that the output and error data is
602 merged.
603 executable = <pathname>
609 An optional path and a required file name of the executable file for
611 this job cluster. Only one executablecommand within a submit
612 description file is guaranteed to work properly. More than one often
613 works.
614 If no path or a relative path is used, then the executable file is
616 presumed to be relative to the current working directory of the user
617 as the condor_submitcommand is issued.
618 If submitting into the standard universe, then the named executable
620 must have been re-linked with the HTCondor libraries (such as via
621 the condor_compilecommand). If submitting into the vanilla universe
622 (the default), then the named executable need not be re-linked and
623 can be any process which can run in the background (shell scripts
624 work fine as well). If submitting into the Java universe, then the
625 argument must be a compiled .classfile.
626 getenv = <True | False>
632 If getenvis set to True, then condor_submitwill copy all of the
634 user's current shell environment variables at the time of job sub‐
635 mission into the job ClassAd. The job will therefore execute with
636 the same set of environment variables that the user had at submit
637 time. Defaults to False.
638 If the environment is set with the environmentcommand andgetenvis
640 also set to true, values specified with environmentoverride values
641 in the submitter's environment (regardless of the order of the envi‐
642 ronmentand getenvcommands).
643 input = <pathname>
649 HTCondor assumes that its jobs are long-running, and that the user
651 will not wait at the terminal for their completion. Because of this,
652 the standard files which normally access the terminal, (stdin, std‐
653 out, and stderr), must refer to files. Thus, the file name specified
654 with inputshould contain any keyboard input the program requires
655 (that is, this file becomes stdin). A path is given with respect to
656 the file system of the machine on which the job is submitted. The
657 file is transferred before execution to the remote scratch directory
658 of the machine where the job is executed. If not specified, the
659 default value of /dev/nullis used for submission to a Unix machine.
660 If not specified, input is ignored for submission to a Windows
661 machine. For grid universe jobs, inputmay be a URL that the Globus
662 tool globus_url_copyunderstands.
663 Note that this command does notrefer to the command-line arguments
665 of the program. The command-line arguments are specified by the
666 argumentscommand.
667 log = <pathname>
673 Use logto specify a file name where HTCondor will write a log file
675 of what is happening with this job cluster, called a job event log.
676 For example, HTCondor will place a log entry into this file when and
677 where the job begins running, when the job produces a checkpoint, or
678 moves (migrates) to another machine, and when the job completes.
679 Most users find specifying a logfile to be handy; its use is recom‐
680 mended. If no logentry is specified, HTCondor does not create a log
681 for this cluster. If a relative path is specified, it is relative to
682 the current working directory as the job is submitted or the direc‐
683 tory specified by submit command initialdiron the submit machine.
684 log_xml = <True | False>
690 If log_xmlis True, then the job event log file will be written in
692 ClassAd XML. If not specified, XML is not used. Note that the file
693 is an XML fragment; it is missing the file header and footer. Do not
694 mix XML and non-XML within a single file. If multiple jobs write to
695 a single job event log file, ensure that all of the jobs specify
696 this option in the same way.
697 notification = <Always | Complete | Error | Never>
703 Owners of HTCondor jobs are notified by e-mail when certain events
705 occur. If defined by Always, the owner will be notified whenever the
706 job produces a checkpoint, as well as when the job completes. If
707 defined by Complete, the owner will be notified when the job termi‐
708 nates. If defined by Error, the owner will only be notified if the
709 job terminates abnormally, (as defined by JobSuccessExitCode, if
710 defined) or if the job is placed on hold because of a failure, and
711 not by user request. If defined by Never(the default), the owner
712 will not receive e-mail, regardless to what happens to the job. The
713 HTCondor User's manual documents statistics included in the e-mail.
714 notify_user = <email-address>
720 Used to specify the e-mail address to use when HTCondor sends e-mail
722 about a job. If not specified, HTCondor defaults to using the e-mail
723 address defined by
724 job-owner@UID_DOMAIN
726 where the configuration variable UID_DOMAINis specified by the
728 HTCondor site administrator. If UID_DOMAINhas not been specified,
729 HTCondor sends the e-mail to:
730 job-owner@submit-machine-name
732 output = <pathname>
738 The outputfile captures any information the program would ordinarily
740 write to the screen (that is, this file becomes stdout). A path is
741 given with respect to the file system of the machine on which the
742 job is submitted. The file is written (by the job) in the remote
743 scratch directory of the machine where the job is executed. When the
744 job exits, the resulting file is transferred back to the machine
745 where the job was submitted, and the path is utilized for file
746 placement. If not specified, the default value of /dev/nullis used
747 for submission to a Unix machine. If not specified, output is
748 ignored for submission to a Windows machine. Multiple jobs should
749 not use the same output file, since this will cause one job to over‐
750 write the output of another. If HTCondor detects that the error and
751 output files for a job are the same, it will run the job such that
752 the output and error data is merged.
753 Note that if a program explicitly opens and writes to a file, that
755 file should notbe specified as the outputfile.
756 priority = <integer>
762 An HTCondor job priority can be any integer, with 0 being the
764 default. Jobs with higher numerical priority will run before jobs
765 with lower numerical priority. Note that this priority is on a per
766 user basis. One user with many jobs may use this command to order
767 his/her own jobs, and this will have no effect on whether or not
768 these jobs will run ahead of another user's jobs.
769 Note that the priority setting in an HTCondor submit file will be
771 overridden by condor_dagmanif the submit file is used for a node in
772 a DAG, and the priority of the node within the DAG is non-zero (see
773 for more details).
774 queue [<int expr>]
780 Places zero or more copies of the job into the HTCondor queue.
782 queue
786 [<int expr>] [<varname>] in[slice] <list of items>Places zero or
788 more copies of the job in the queue based on items in a <list of
789 items>
790 queue
794 [<int expr>] [<varname>] matching[files | dirs] [slice] <list of
796 items with file globbing>] Places zero or more copies of the job in
797 the queue based on files that match a <list of items with file glob‐
798 bing>
799 queue
803 [<int expr>] [<list of varnames>] from[slice] <file name> | <list of
805 items>] Places zero or more copies of the job in the queue based on
806 lines from the submit file or from <file name>
807 The optional argument <int expr>specifies how many times to repeat
809 the job submission for a given set of arguments. It may be an inte‐
810 ger or an expression that evaluates to an integer, and it defaults
811 to 1. All but the first form of this command are various ways of
812 specifying a list of items. When these forms are used <int expr>jobs
813 will be queued for each item in the list. The in, matchingand
814 fromkeyword indicates how the list will be specified.
815 * inThe list of items is an explicit comma and/or space separated
817 <list of items>. If the <list of items>begins with an open paren,
818 and the close paren is not on the same line as the open, then the
819 list continues until a line that begins with a close paren is
820 read from the submit file.
821 * matchingEach item in the <list of items with file globbing>will
823 be matched against the names of files and directories relative to
824 the current directory, the set of matching names is the resulting
825 list of items.
826 * filesOnly filenames will matched.
828 * dirsOnly directory names will be matched.
830 * from<file name> | <list of items>Each line from <file name>or
832 <list of items>is a single item, this allows for multiple vari‐
833 ables to be set for each item. Lines from <file name>or <list of
834 items>will be split on comma and/or space until there are values
835 for each of the variables specified in <list of varnames>. The
836 last variable will contain the remainder of the line. When the
837 <list of items>form is used, the list continues until the first
838 line that begins with a close paren, and lines beginning with
839 pound sign ('#') will be skipped. When using the <file name>form,
840 if the <file name>ends with |, then it will be executed as a
841 script whatever the script writes to stdoutwill be the list of
842 items. The optional argument <varname>or <list of varnames>is the
843 name or names of of variables that will be set to the value of
844 the current item when queuing the job. If no <varname>is speci‐
845 fied the variable ITEM will be used. Leading and trailing white‐
846 space be trimmed. The optional argument <slice>is a python style
847 slice selecting only some of the items in the list of items. Neg‐
848 ative step values are not supported.
849 A submit file may contain more than one queuestatement, and if
851 desired, any commands may be placed between subsequent queuecom‐
852 mands, such as new input, output, error, initialdir, or argu‐
853 mentscommands. This is handy when submitting multiple runs into one
854 cluster with one submit description file.
855 universe = <vanilla | standard | scheduler | local | grid | java | vm |
861 parallel | docker>
862 Specifies which HTCondor universe to use when running this job. The
864 HTCondor universe specifies an HTCondor execution environment.
865 The vanillauniverse is the default (except where the configuration
867 variable DEFAULT_UNIVERSEdefines it otherwise), and is an execution
868 environment for jobs which do not use HTCondor's mechanisms for tak‐
869 ing checkpoints; these are ones that have not been linked with the
870 HTCondor libraries. Use the vanillauniverse to submit shell scripts
871 to HTCondor.
872 The standarduniverse tells HTCondor that this job has been re-linked
874 via condor_compilewith the HTCondor libraries and therefore supports
875 taking checkpoints and remote system calls.
876 The scheduleruniverse is for a job that is to run on the machine
878 where the job is submitted. This universe is intended for a job that
879 acts as a metascheduler and will not be preempted.
880 The localuniverse is for a job that is to run on the machine where
882 the job is submitted. This universe runs the job immediately and
883 will not preempt the job.
884 The griduniverse forwards the job to an external job management sys‐
886 tem. Further specification of the griduniverse is done with the
887 grid_resourcecommand.
888 The javauniverse is for programs written to the Java Virtual
890 Machine.
891 The vmuniverse facilitates the execution of a virtual machine.
893 The paralleluniverse is for parallel jobs (e.g. MPI) that require
895 multiple machines in order to run.
896 The dockeruniverse runs a docker container as an HTCondor job.
898 COMMANDS FOR MATCHMAKING
904 rank = <ClassAd Float Expression>
912 A ClassAd Floating-Point expression that states how to rank machines
914 which have already met the requirements expression. Essentially,
915 rank expresses preference. A higher numeric value equals better
916 rank. HTCondor will give the job the machine with the highest rank.
917 For example,
918 request_memory = max({60, Target.TotalSlotMemory})
920 rank = Memory
921 asks HTCondor to find all available machines with more than 60
923 megabytes of memory and give to the job the machine with the most
924 amount of memory. The HTCondor User's Manual contains complete
925 information on the syntax and available attributes that can be used
926 in the ClassAd expression.
927 request_cpus = <num-cpus>
933 A requested number of CPUs (cores). If not specified, the number
935 requested will be 1. If specified, the expression
936 && (RequestCpus <= Target.Cpus)
938 is appended to the requirementsexpression for the job.
940 For pools that enable dynamic condor_startdprovisioning, specifies
942 the minimum number of CPUs requested for this job, resulting in a
943 dynamic slot being created with this many cores.
944 request_disk = <quantity>
950 The requested amount of disk space in KiB requested for this job. If
952 not specified, it will be set to the job ClassAd attribute
953 DiskUsage. The expression
954 && (RequestDisk <= Target.Disk)
956 is appended to the requirementsexpression for the job.
958 For pools that enable dynamic condor_startdprovisioning, a dynamic
960 slot will be created with at least this much disk space.
961 Characters may be appended to a numerical value to indicate units.
963 Kor KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers
964 of bytes. Gor GBindicates GiB, numbers of bytes. Tor TBindicates
965 TiB, numbers of bytes.
966 request_memory = <quantity>
972 The required amount of memory in MiB that this job needs to avoid
974 excessive swapping. If not specified and the submit command vm_memo‐
975 ryis specified, then the value specified for vm_memorydefines
976 request_memory. If neither request_memorynor vm_memoryis specified,
977 the value is set by the configuration variable JOB_DEFAULT_REQUEST‐
978 MEMORY. The actual amount of memory used by a job is represented by
979 the job ClassAd attribute MemoryUsage.
980 For pools that enable dynamic condor_startdprovisioning, a dynamic
982 slot will be created with at least this much RAM.
983 The expression
985 && (RequestMemory <= Target.Memory)
987 is appended to the requirementsexpression for the job.
989 Characters may be appended to a numerical value to indicate units.
991 Kor KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers
992 of bytes. Gor GBindicates GiB, numbers of bytes. Tor TBindicates
993 TiB, numbers of bytes.
994 request_<name> = <quantity>
1000 The required amount of the custom machine resource identified by
1002 <name>that this job needs. The custom machine resource is defined in
1003 the machine's configuration. Machines that have available GPUs will
1004 define <name>to be GPUs.
1005 requirements = <ClassAd Boolean Expression>
1011 The requirements command is a boolean ClassAd expression which uses
1013 C-like operators. In order for any job in this cluster to run on a
1014 given machine, this requirements expression must evaluate to true on
1015 the given machine.
1016 For scheduler and local universe jobs, the requirements expression
1018 is evaluated against the SchedulerClassAd which represents the the
1019 condor_schedddaemon running on the submit machine, rather than a
1020 remote machine. Like all commands in the submit description file, if
1021 multiple requirements commands are present, all but the last one are
1022 ignored. By default, condor_submitappends the following clauses to
1023 the requirements expression:
1024 1. Arch and OpSys are set equal to the Arch and OpSys of the sub‐
1026 mit machine. In other words: unless you request otherwise, HTCon‐
1027 dor will give your job machines with the same architecture and
1028 operating system version as the machine running condor_submit.
1029 2. Cpus >= RequestCpus, if the job ClassAd attribute RequestCpu‐
1031 sis defined.
1032 3. Disk >= RequestDisk, if the job ClassAd attribute Request‐
1034 Diskis defined. Otherwise, Disk >= DiskUsage is appended to the
1035 requirements. The DiskUsageattribute is initialized to the size
1036 of the executable plus the size of any files specified in a
1037 transfer_input_filescommand. It exists to ensure there is enough
1038 disk space on the target machine for HTCondor to copy over both
1039 the executable and needed input files. The DiskUsageattribute
1040 represents the maximum amount of total disk space required by the
1041 job in kilobytes. HTCondor automatically updates the DiskUsageat‐
1042 tribute approximately every 20 minutes while the job runs with
1043 the amount of space being used by the job on the execute machine.
1044 4. Memory >= RequestMemory, if the job ClassAd attribute Request‐
1046 Memoryis defined.
1047 5. If Universe is set to Vanilla, FileSystemDomain is set equal
1049 to the submit machine's FileSystemDomain. View the requirements
1050 of a job which has already been submitted (along with everything
1051 else about the job ClassAd) with the command condor_q -l; see the
1052 command reference for condor_qon page . Also, see the HTCondor
1053 Users Manual for complete information on the syntax and available
1054 attributes that can be used in the ClassAd expression.
1055 FILE TRANSFER COMMANDS
1061 dont_encrypt_input_files = < file1,file2,file... >
1069 A comma and/or space separated list of input files that are notto be
1071 network encrypted when transferred with the file transfer mechanism.
1072 Specification of files in this manner overrides configuration that
1073 would use encryption. Each input file must also be in the list given
1074 by transfer_input_files. When a path to an input file or directory
1075 is specified, this specifies the path to the file on the submit
1076 side. A single wild card character (*) may be used in each file
1077 name.
1078 dont_encrypt_output_files = < file1,file2,file... >
1084 A comma and/or space separated list of output files that are notto
1086 be network encrypted when transferred back with the file transfer
1087 mechanism. Specification of files in this manner overrides configu‐
1088 ration that would use encryption. The output file(s) must also
1089 either be in the list given by transfer_output_filesor be discovered
1090 and to be transferred back with the file transfer mechanism. When a
1091 path to an output file or directory is specified, this specifies the
1092 path to the file on the execute side. A single wild card character
1093 (*) may be used in each file name.
1094 encrypt_execute_directory = <True | False>
1100 Defaults to False. If set to True, HTCondor will encrypt the con‐
1102 tents of the remote scratch directory of the machine where the job
1103 is executed. This encryption is transparent to the job itself, but
1104 ensures that files left behind on the local disk of the execute
1105 machine, perhaps due to a system crash, will remain private. In
1106 addition, condor_submitwill append to the job's requirementsexpres‐
1107 sion
1108 && (TARGET.HasEncryptExecuteDirectory)
1110 to ensure the job is matched to a machine that is capable of
1112 encrypting the contents of the execute directory. This support is
1113 limited to Windows platforms that use the NTFS file system and Linux
1114 platforms with the ecryptfs-utilspackage installed.
1115 encrypt_input_files = < file1,file2,file... >
1121 A comma and/or space separated list of input files that are to be
1123 network encrypted when transferred with the file transfer mechanism.
1124 Specification of files in this manner overrides configuration that
1125 would notuse encryption. Each input file must also be in the list
1126 given by transfer_input_files. When a path to an input file or
1127 directory is specified, this specifies the path to the file on the
1128 submit side. A single wild card character (*) may be used in each
1129 file name. The method of encryption utilized will be as agreed upon
1130 in security negotiation; if that negotiation failed, then the file
1131 transfer mechanism must also fail for files to be network encrypted.
1132 encrypt_output_files = < file1,file2,file... >
1138 A comma and/or space separated list of output files that are to be
1140 network encrypted when transferred back with the file transfer mech‐
1141 anism. Specification of files in this manner overrides configuration
1142 that would notuse encryption. The output file(s) must also either be
1143 in the list given by transfer_output_filesor be discovered and to be
1144 transferred back with the file transfer mechanism. When a path to an
1145 output file or directory is specified, this specifies the path to
1146 the file on the execute side. A single wild card character (*) may
1147 be used in each file name. The method of encryption utilized will be
1148 as agreed upon in security negotiation; if that negotiation failed,
1149 then the file transfer mechanism must also fail for files to be net‐
1150 work encrypted.
1151 max_transfer_input_mb = <ClassAd Integer Expression>
1157 This integer expression specifies the maximum allowed total size in
1159 MiB of the input files that are transferred for a job. This expres‐
1160 sion does notapply to grid universe, standard universe, or files
1161 transferred via file transfer plug-ins. The expression may refer to
1162 attributes of the job. The special value -1 indicates no limit. If
1163 not defined, the value set by configuration variable MAX_TRANS‐
1164 FER_INPUT_MBis used. If the observed size of all input files at sub‐
1165 mit time is larger than the limit, the job will be immediately
1166 placed on hold with a HoldReasonCodevalue of 32. If the job passes
1167 this initial test, but the size of the input files increases or the
1168 limit decreases so that the limit is violated, the job will be
1169 placed on hold at the time when the file transfer is attempted.
1170 max_transfer_output_mb = <ClassAd Integer Expression>
1176 This integer expression specifies the maximum allowed total size in
1178 MiB of the output files that are transferred for a job. This expres‐
1179 sion does notapply to grid universe, standard universe, or files
1180 transferred via file transfer plug-ins. The expression may refer to
1181 attributes of the job. The special value -1 indicates no limit. If
1182 not set, the value set by configuration variable MAX_TRANSFER_OUT‐
1183 PUT_MBis used. If the total size of the job's output files to be
1184 transferred is larger than the limit, the job will be placed on hold
1185 with a HoldReasonCodevalue of 33. The output will be transferred up
1186 to the point when the limit is hit, so some files may be fully
1187 transferred, some partially, and some not at all.
1188 output_destination = <destination-URL>
1194 When present, defines a URL that specifies both a plug-in and a des‐
1196 tination for the transfer of the entire output sandbox or a subset
1197 of output files as specified by the submit command transfer_out‐
1198 put_files. The plug-in does the transfer of files, and no files are
1199 sent back to the submit machine. The HTCondor Administrator's manual
1200 has full details.
1201 should_transfer_files = <YES | NO | IF_NEEDED >
1207 The should_transfer_filessetting is used to define if HTCondor
1209 should transfer files to and from the remote machine where the job
1210 runs. The file transfer mechanism is used to run jobs which are not
1211 in the standard universe (and can therefore use remote system calls
1212 for file access) on machines which do not have a shared file system
1213 with the submit machine. should_transfer_filesequal to YESwill cause
1214 HTCondor to always transfer files for the job. NOdisables HTCondor's
1215 file transfer mechanism. IF_NEEDEDwill not transfer files for the
1216 job if it is matched with a resource in the same FileSystemDomainas
1217 the submit machine (and therefore, on a machine with the same shared
1218 file system). If the job is matched with a remote resource in a dif‐
1219 ferent FileSystemDomain, HTCondor will transfer the necessary files.
1220 For more information about this and other settings related to trans‐
1222 ferring files, see the HTCondor User's manual section on the file
1223 transfer mechanism.
1224 Note that should_transfer_filesis not supported for jobs submitted
1226 to the grid universe.
1227 skip_filechecks = <True | False>
1233 When True, file permission checks for the submitted job are dis‐
1235 abled. When False, file permissions are checked; this is the behav‐
1236 ior when this command is not present in the submit description file.
1237 File permissions are checked for read permissions on all input
1238 files, such as those defined by commands inputand trans‐
1239 fer_input_files, and for write permission to output files, such as a
1240 log file defined by logand output files defined with outputor trans‐
1241 fer_output_files.
1242 stream_error = <True | False>
1248 If True, then stderris streamed back to the machine from which the
1250 job was submitted. If False, stderris stored locally and transferred
1251 back when the job completes. This command is ignored if the job
1252 ClassAd attribute TransferErris False. The default value is False.
1253 This command must be used in conjunction with error, otherwise
1254 stderrwill sent to /dev/nullon Unix machines and ignored on Windows
1255 machines.
1256 stream_input = <True | False>
1262 If True, then stdinis streamed from the machine on which the job was
1264 submitted. The default value is False. The command is only relevant
1265 for jobs submitted to the vanilla or java universes, and it is
1266 ignored by the grid universe. This command must be used in conjunc‐
1267 tion with input, otherwise stdinwill be /dev/nullon Unix machines
1268 and ignored on Windows machines.
1269 stream_output = <True | False>
1275 If True, then stdoutis streamed back to the machine from which the
1277 job was submitted. If False, stdoutis stored locally and transferred
1278 back when the job completes. This command is ignored if the job
1279 ClassAd attribute TransferOutis False. The default value is False.
1280 This command must be used in conjunction with output, otherwise std‐
1281 outwill sent to /dev/nullon Unix machines and ignored on Windows
1282 machines.
1283 transfer_executable = <True | False>
1289 This command is applicable to jobs submitted to the grid and vanilla
1291 universes. If transfer_executableis set to False, then HTCondor
1292 looks for the executable on the remote machine, and does not trans‐
1293 fer the executable over. This is useful for an already pre-staged
1294 executable; HTCondor behaves more like rsh. The default value is
1295 True.
1296 transfer_input_files = < file1,file2,file... >
1302 A comma-delimited list of all the files and directories to be trans‐
1304 ferred into the working directory for the job, before the job is
1305 started. By default, the file specified in the executablecommand and
1306 any file specified in the inputcommand (for example, stdin) are
1307 transferred.
1308 When a path to an input file or directory is specified, this speci‐
1310 fies the path to the file on the submit side. The file is placed in
1311 the job's temporary scratch directory on the execute side, and it is
1312 named using the base name of the original path. For example,
1313 /path/to/input_filebecomes input_filein the job's scratch directory.
1314 A directory may be specified by appending the forward slash charac‐
1316 ter (/) as a trailing path separator. This syntax is used for both
1317 Windows and Linux submit hosts. A directory example using a trailing
1318 path separator is input_data/. When a directory is specified with
1319 the trailing path separator, the contents of the directory are
1320 transferred, but the directory itself is not transferred. It is as
1321 if each of the items within the directory were listed in the trans‐
1322 fer list. When there is no trailing path separator, the directory is
1323 transferred, its contents are transferred, and these contents are
1324 placed inside the transferred directory.
1325 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1327 tories is not currently supported.
1328 Symbolic links to files are transferred as the files they point to.
1330 Transfer of symbolic links to directories is not currently sup‐
1331 ported.
1332 For vanilla and vm universe jobs only, a file may be specified by
1334 giving a URL, instead of a file name. The implementation for URL
1335 transfers requires both configuration and available plug-in.
1336 transfer_output_files = < file1,file2,file... >
1342 This command forms an explicit list of output files and directories
1344 to be transferred back from the temporary working directory on the
1345 execute machine to the submit machine. If there are multiple files,
1346 they must be delimited with commas. Setting transfer_output_filesto
1347 the empty string ( "" ) means that no files are to be transferred.
1348 For HTCondor-C jobs and all other non-grid universe jobs, if trans‐
1350 fer_output_filesis not specified, HTCondor will automatically trans‐
1351 fer back all files in the job's temporary working directory which
1352 have been modified or created by the job. Subdirectories are not
1353 scanned for output, so if output from subdirectories is desired, the
1354 output list must be explicitly specified. For grid universe jobs
1355 other than HTCondor-C, desired output files must also be explicitly
1356 listed. Another reason to explicitly list output files is for a job
1357 that creates many files, and the user wants only a subset trans‐
1358 ferred back.
1359 For grid universe jobs other than with grid type condor, to have
1361 files other than standard output and standard error transferred from
1362 the execute machine back to the submit machine, do use transfer_out‐
1363 put_files, listing all files to be transferred. These files are
1364 found on the execute machine in the working directory of the job.
1365 When a path to an output file or directory is specified, it speci‐
1367 fies the path to the file on the execute side. As a destination on
1368 the submit side, the file is placed in the job's initial working
1369 directory, and it is named using the base name of the original path.
1370 For example, path/to/output_filebecomes output_filein the job's ini‐
1371 tial working directory. The name and path of the file that is writ‐
1372 ten on the submit side may be modified by using transfer_out‐
1373 put_remaps. Note that this remap function only works with files but
1374 not with directories.
1375 A directory may be specified using a trailing path separator. An
1377 example of a trailing path separator is the slash character on Unix
1378 platforms; a directory example using a trailing path separator is
1379 input_data/. When a directory is specified with a trailing path sep‐
1380 arator, the contents of the directory are transferred, but the
1381 directory itself is not transferred. It is as if each of the items
1382 within the directory were listed in the transfer list. When there is
1383 no trailing path separator, the directory is transferred, its con‐
1384 tents are transferred, and these contents are placed inside the
1385 transferred directory.
1386 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1388 tories is not currently supported.
1389 Symbolic links to files are transferred as the files they point to.
1391 Transfer of symbolic links to directories is not currently sup‐
1392 ported.
1393 transfer_output_remaps = < “ name = newname ; name2 = newname2
1399 ... ”>
1400 This specifies the name (and optionally path) to use when download‐
1402 ing output files from the completed job. Normally, output files are
1403 transferred back to the initial working directory with the same name
1404 they had in the execution directory. This gives you the option to
1405 save them with a different path or name. If you specify a relative
1406 path, the final path will be relative to the job's initial working
1407 directory.
1408 namedescribes an output file name produced by your job, and new‐
1410 namedescribes the file name it should be downloaded to. Multiple
1411 remaps can be specified by separating each with a semicolon. If you
1412 wish to remap file names that contain equals signs or semicolons,
1413 these special characters may be escaped with a backslash. You cannot
1414 specify directories to be remapped.
1415 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT >
1421 Setting when_to_transfer_outputequal to ON_EXITwill cause HTCondor
1425 to transfer the job's output files back to the submitting machine
1426 only when the job completes (exits on its own).
1427 The ON_EXIT_OR_EVICToption is intended for fault tolerant jobs which
1429 periodically save their own state and can restart where they left
1430 off. In this case, files are spooled to the submit machine any time
1431 the job leaves a remote site, either because it exited on its own,
1432 or was evicted by the HTCondor system for any reason prior to job
1433 completion. The files spooled back are placed in a directory defined
1434 by the value of the SPOOLconfiguration variable. Any output files
1435 transferred back to the submit machine are automatically sent back
1436 out again as input files if the job restarts.
1437 POLICY COMMANDS
1443 max_retries = <integer>
1451 The maximum number of retries allowed for this job (must be non-neg‐
1453 ative). If the job fails (does not exit with the success_exit_code‐
1454 exit code) it will be retried up to max_retriestimes (unless retries
1455 are ceased because of the retry_untilcommand). If max_retriesis not
1456 defined, and either retry_untilor success_exit_codeis, the value of
1457 DEFAULT_JOB_MAX_RETRIESwill be used for the maximum number of
1458 retries.
1459 The combination of the max_retries, retry_until, and suc‐
1461 cess_exit_codecommands causes an appropriate OnExitRemoveexpression
1462 to be automatically generated. If retry command(s) and
1463 on_exit_removeare both defined, the OnExitRemoveexpression will be
1464 generated by OR'ing the expression specified in OnExitRemoveand the
1465 expression generated by the retry commands.
1466 retry_until <Integer | ClassAd Boolean Expression>
1472 An integer value or boolean expression that prevents further retries
1474 from taking place, even if max_retrieshave not been exhausted. If
1475 retry_untilis an integer, the job exiting with that exit code will
1476 cause retries to cease. If retry_untilis a ClassAd expression, the
1477 expression evaluating to Truewill cause retries to cease.
1478 success_exit_code = <integer>
1484 The exit code that is considered successful for this job. Defaults
1486 to 0 if not defined.
1487 Note: non-zero values of success_exit_codeshould generally not be
1489 used for DAG node jobs.At the present time, condor_dagmandoes not
1490 take into account the value of success_exit_code. This means that,
1491 if success_exit_codeis set to a non-zero value, condor_dagmanwill
1492 consider the job failed when it actually succeeds. For single-proc
1493 DAG node jobs, this can be overcome by using a POST script that
1494 takes into account the value of success_exit_code(although this is
1495 not recommended). For multi-proc DAG node jobs, there is currently
1496 no way to overcome this limitation.
1497 hold = <True | False>
1503 If holdis set to True, then the submitted job will be placed into
1505 the Hold state. Jobs in the Hold state will not run until released
1506 by condor_release. Defaults to False.
1507 keep_claim_idle = <integer>
1513 An integer number of seconds that a job requests the condor_scheddto
1515 wait before releasing its claim after the job exits or after the job
1516 is removed.
1517 The process by which the condor_scheddclaims a condor_startdis some‐
1519 what time-consuming. To amortize this cost, the condor_scheddtries
1520 to reuse claims to run subsequent jobs, after a job using a claim is
1521 done. However, it can only do this if there is an idle job in the
1522 queue at the moment the previous job completes. Sometimes, and espe‐
1523 cially for the node jobs when using DAGMan, there is a subsequent
1524 job about to be submitted, but it has not yet arrived in the queue
1525 when the previous job completes. As a result, the condor_sched‐
1526 dreleases the claim, and the next job must wait an entire negotia‐
1527 tion cycle to start. When this submit command is defined with a non-
1528 negative integer, when the job exits, the condor_scheddtries as
1529 usual to reuse the claim. If it cannot, instead of releasing the
1530 claim, the condor_scheddkeeps the claim until either the number of
1531 seconds given as a parameter, or a new job which matches that claim
1532 arrives, whichever comes first. The condor_startdin question will
1533 remain in the Claimed/Idle state, and the original job will be
1534 "charged" (in terms of priority) for the time in this state.
1535 leave_in_queue = <ClassAd Boolean Expression>
1541 When the ClassAd Expression evaluates to True, the job is not
1543 removed from the queue upon completion. This allows the user of a
1544 remotely spooled job to retrieve output files in cases where HTCon‐
1545 dor would have removed them as part of the cleanup associated with
1546 completion. The job will only exit the queue once it has been marked
1547 for removal (via condor_rm, for example) and the leave_in_queueex‐
1548 pression has become False. leave_in_queuedefaults to False.
1549 As an example, if the job is to be removed once the output is
1551 retrieved with condor_transfer_data, then use
1552 leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED)
1554 ||\
1555 (StageOutFinish == 0))
1556 next_job_start_delay = <ClassAd Boolean Expression>
1562 This expression specifies the number of seconds to delay after
1564 starting up this job before the next job is started. The maximum
1565 allowed delay is specified by the HTCondor configuration variable
1566 MAX_NEXT_JOB_START_DELAY, which defaults to 10 minutes. This command
1567 does not apply to scheduleror localuniverse jobs.
1568 This command has been historically used to implement a form of job
1570 start throttling from the job submitter's perspective. It was effec‐
1571 tive for the case of multiple job submission where the transfer of
1572 extremely large input data sets to the execute machine caused
1573 machine performance to suffer. This command is no longer useful, as
1574 throttling should be accomplished through configuration of the con‐
1575 dor_schedddaemon.
1576 on_exit_hold = <ClassAd Boolean Expression>
1582 The ClassAd expression is checked when the job exits, and if True,
1584 places the job into the Hold state. If False(the default value when
1585 not defined), then nothing happens and the on_exit_removeexpression
1586 is checked to determine if that needs to be applied.
1587 For example: Suppose a job is known to run for a minimum of an hour.
1589 If the job exits after less than an hour, the job should be placed
1590 on hold and an e-mail notification sent, instead of being allowed to
1591 leave the queue.
1592 on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1596 This expression places the job on hold if it exits for any reason
1598 before running for an hour. An e-mail will be sent to the user
1599 explaining that the job was placed on hold because this expression
1600 became True.
1601 periodic_*expressions take precedence over on_exit_*expressions, and
1603 *_holdexpressions take precedence over a *_removeexpressions.
1604 Only job ClassAd attributes will be defined for use by this ClassAd
1606 expression. This expression is available for the vanilla, java, par‐
1607 allel, grid, local and scheduler universes. It is additionally
1608 available, when submitted from a Unix machine, for the standard uni‐
1609 verse.
1610 on_exit_hold_reason = <ClassAd String Expression>
1616 When the job is placed on hold due to the on_exit_holdexpression
1618 becoming True, this expression is evaluated to set the value of Hol‐
1619 dReasonin the job ClassAd. If this expression is UNDEFINEDor pro‐
1620 duces an empty or invalid string, a default description is used.
1621 on_exit_hold_subcode = <ClassAd Integer Expression>
1627 When the job is placed on hold due to the on_exit_holdexpression
1629 becoming True, this expression is evaluated to set the value of Hol‐
1630 dReasonSubCodein the job ClassAd. The default subcode is 0. The Hol‐
1631 dReasonCodewill be set to 3, which indicates that the job went on
1632 hold due to a job policy expression.
1633 on_exit_remove = <ClassAd Boolean Expression>
1639 The ClassAd expression is checked when the job exits, and if
1641 True(the default value when undefined), then it allows the job to
1642 leave the queue normally. If False, then the job is placed back into
1643 the Idle state. If the user job runs under the vanilla universe,
1644 then the job restarts from the beginning. If the user job runs under
1645 the standard universe, then it continues from where it left off,
1646 using the last checkpoint.
1647 For example, suppose a job occasionally segfaults, but chances are
1649 that the job will finish successfully if the job is run again with
1650 the same data. The on_exit_removeexpression can cause the job to run
1651 again with the following command. Assume that the signal identifier
1652 for the segmentation fault is 11 on the platform where the job will
1653 be running.
1654 on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11) This
1656 expression lets the job leave the queue if the job was not killed by a
1657 signal or if it was killed by a signal other than 11, representing seg‐
1658 mentation fault in this example. So, if the exited due to signal 11, it
1659 will stay in the job queue. In any other case of the job exiting, the
1660 job will leave the queue as it normally would have done.
1661 As another example, if the job should only leave the queue if it
1663 exited on its own with status 0, this on_exit_removeexpression works
1664 well:
1665 on_exit_remove = (ExitBySignal == False) && (ExitCode == 0) If the
1669 job was killed by a signal or exited with a non-zero exit status,
1670 HTCondor would leave the job in the queue to run again.
1671 periodic_*expressions take precedence over on_exit_*expressions, and
1673 *_holdexpressions take precedence over a *_removeexpressions.
1674 Only job ClassAd attributes will be defined for use by this ClassAd
1676 expression.
1677 periodic_hold = <ClassAd Boolean Expression>
1683 This expression is checked periodically when the job is not in the
1685 Held state. If it becomes True, the job will be placed on hold. If
1686 unspecified, the default value is False.
1687 periodic_*expressions take precedence over on_exit_*expressions, and
1689 *_holdexpressions take precedence over a *_removeexpressions.
1690 Only job ClassAd attributes will be defined for use by this ClassAd
1692 expression. Note that, by default, this expression is only checked
1693 once every 60 seconds. The period of these evaluations can be
1694 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1695 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1696 periodic_hold_reason = <ClassAd String Expression>
1702 When the job is placed on hold due to the periodic_holdexpression
1704 becoming True, this expression is evaluated to set the value of Hol‐
1705 dReasonin the job ClassAd. If this expression is UNDEFINEDor pro‐
1706 duces an empty or invalid string, a default description is used.
1707 periodic_hold_subcode = <ClassAd Integer Expression>
1713 When the job is placed on hold due to the periodic_holdexpression
1715 becoming true, this expression is evaluated to set the value of Hol‐
1716 dReasonSubCodein the job ClassAd. The default subcode is 0. The Hol‐
1717 dReasonCodewill be set to 3, which indicates that the job went on
1718 hold due to a job policy expression.
1719 periodic_release = <ClassAd Boolean Expression>
1725 This expression is checked periodically when the job is in the Held
1727 state. If the expression becomes True, the job will be released.
1728 Only job ClassAd attributes will be defined for use by this ClassAd
1730 expression. Note that, by default, this expression is only checked
1731 once every 60 seconds. The period of these evaluations can be
1732 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1733 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1734 periodic_remove = <ClassAd Boolean Expression>
1740 This expression is checked periodically. If it becomes True, the job
1742 is removed from the queue. If unspecified, the default value is
1743 False.
1744 See the Examples section of this manual page for an example of a
1746 periodic_removeexpression.
1747 periodic_*expressions take precedence over on_exit_*expressions, and
1749 *_holdexpressions take precedence over a *_removeexpressions. So,
1750 the periodic_removeexpression takes precedent over the
1751 on_exit_removeexpression, if the two describe conflicting actions.
1752 Only job ClassAd attributes will be defined for use by this ClassAd
1754 expression. Note that, by default, this expression is only checked
1755 once every 60 seconds. The period of these evaluations can be
1756 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1757 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1758 COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
1764 allow_startup_script = <True | False>
1772 If True, a standard universe job will execute a script instead of
1774 submitting the job, and the consistency check to see if the exe‐
1775 cutable has been linked using condor_compileis omitted. The exe‐
1776 cutablecommand within the submit description file specifies the name
1777 of the script. The script is used to do preprocessing before the job
1778 is submitted. The shell script ends with an execof the job exe‐
1779 cutable, such that the process id of the executable is the same as
1780 that of the shell script. Here is an example script that gets a copy
1781 of a machine-specific executable before the exec.
1782 #! /bin/sh
1784 # get the host name of the machine
1786 $host=`uname -n`
1787 # grab a standard universe executable designed specifically
1789 # for this host
1790 scp elsewhere@cs.wisc.edu:${host} executable
1791 # The PID MUST stay the same, so exec the new standard universe
1793 process.
1794 exec executable ${1+"$@"} If this command is not present (defined),
1795 then the value defaults to false.
1796 append_files = file1, file2, ...
1802 If your job attempts to access a file mentioned in this list, HTCon‐
1806 dor will force all writes to that file to be appended to the end.
1807 Furthermore, condor_submit will not truncate it. This list uses the
1808 same syntax as compress_files, shown above.
1809 This option may yield some surprising results. If several jobs
1811 attempt to write to the same file, their output may be intermixed.
1812 If a job is evicted from one or more machines during the course of
1813 its lifetime, such an output file might contain several copies of
1814 the results. This option should be only be used when you wish a cer‐
1815 tain file to be treated as a running log instead of a precise
1816 result.
1817 This option only applies to standard-universe jobs.
1819 buffer_files = < “ name = (size,block-size) ; name2 =
1825 (size,block-size) ... ” >
1826 buffer_size = <bytes-in-buffer>
1832 buffer_block_size = <bytes-in-block>
1838 HTCondor keeps a buffer of recently-used data for each file a job
1840 accesses. This buffer is used both to cache commonly-used data and
1841 to consolidate small reads and writes into larger operations that
1842 get better throughput. The default settings should produce reason‐
1843 able results for most programs.
1844 These options only apply to standard-universe jobs.
1846 If needed, you may set the buffer controls individually for each
1848 file using the buffer_files option. For example, to set the buffer
1849 size to 1 MiB and the block size to 256 KiB for the file input.data,
1850 use this command:
1851 buffer_files = "input.data=(1000000,256000)"
1855 Alternatively, you may use these two options to set the default
1857 sizes for all files used by your job:
1858 buffer_size = 1000000
1862 buffer_block_size = 256000
1863 If you do not set these, HTCondor will use the values given by these
1865 two configuration file macros:
1866 DEFAULT_IO_BUFFER_SIZE = 1000000
1870 DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
1871 Finally, if no other settings are present, HTCondor will use a buf‐
1873 fer of 512 KiB and a block size of 32 KiB.
1874 compress_files = file1, file2, ...
1880 If your job attempts to access any of the files mentioned in this
1884 list, HTCondor will automatically compress them (if writing) or
1885 decompress them (if reading). The compress format is the same as
1886 used by GNU gzip.
1887 The files given in this list may be simple file names or complete
1889 paths and may include as a wild card. For example, this list causes
1890 the file /tmp/data.gz, any file named event.gz, and any file ending
1891 in .gzip to be automatically compressed or decompressed as needed:
1892 compress_files = /tmp/data.gz, event.gz, *.gzip
1896 Due to the nature of the compression format, compressed files must
1898 only be accessed sequentially. Random access reading is allowed but
1899 is very slow, while random access writing is simply not possible.
1900 This restriction may be avoided by using both compress_files and
1901 fetch_files at the same time. When this is done, a file is kept in
1902 the decompressed state at the execution machine, but is compressed
1903 for transfer to its original location.
1904 This option only applies to standard universe jobs.
1906 fetch_files = file1, file2, ...
1912 If your job attempts to access a file mentioned in this list, HTCon‐
1914 dor will automatically copy the whole file to the executing machine,
1915 where it can be accessed quickly. When your job closes the file, it
1916 will be copied back to its original location. This list uses the
1917 same syntax as compress_files, shown above.
1918 This option only applies to standard universe jobs.
1920 file_remaps = < “ name = newname ; name2 = newname2 ... ”>
1926 Directs HTCondor to use a new file name in place of an old one.
1930 namedescribes a file name that your job may attempt to open, and
1931 newnamedescribes the file name it should be replaced with. new‐
1932 namemay include an optional leading access specifier, local: or
1933 remote: . If left unspecified, the default access specifier is
1934 remote: . Multiple remaps can be specified by separating each with a
1935 semicolon.
1936 This option only applies to standard universe jobs.
1938 If you wish to remap file names that contain equals signs or semi‐
1940 colons, these special characters may be escaped with a backslash.
1941 Example One:
1945 Suppose that your job reads a file named dataset.1. To instruct
1947 HTCondor to force your job to read other.datasetinstead, add this
1948 to the submit file:
1949 file_remaps = "dataset.1=other.dataset"
1951 Example Two:
1955 Suppose that your run many jobs which all read in the same large
1957 file, called very.big. If this file can be found in the same
1958 place on a local disk in every machine in the pool, (say
1959 /bigdisk/bigfile,) you can instruct HTCondor of this fact by
1960 remapping very.bigto /bigdisk/bigfileand specifying that the file
1961 is to be read locally, which will be much faster than reading
1962 over the network.
1963 file_remaps = "very.big = local:/bigdisk/bigfile"
1965 Example Three:
1969 Several remaps can be applied at once by separating each with a
1971 semicolon.
1972 file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 =
1974 other.dataset"
1975 local_files = file1, file2, ...
1983 If your job attempts to access a file mentioned in this list, HTCon‐
1987 dor will cause it to be read or written at the execution machine.
1988 This is most useful for temporary files not used for input or out‐
1989 put. This list uses the same syntax as compress_files, shown above.
1990 local_files = /tmp/*
1994 This option only applies to standard universe jobs.
1996 want_remote_io = <True | False>
2002 This option controls how a file is opened and manipulated in a stan‐
2004 dard universe job. If this option is true, which is the default,
2005 then the condor_shadowmakes all decisions about how each and every
2006 file should be opened by the executing job. This entails a network
2007 round trip (or more) from the job to the condor_shadowand back again
2008 for every single open()in addition to other needed information about
2009 the file. If set to false, then when the job queries the con‐
2010 dor_shadowfor the first time about how to open a file, the con‐
2011 dor_shadowwill inform the job to automatically perform all of its
2012 file manipulation on the local file system on the execute machine
2013 and any file remapping will be ignored. This means that there mustbe
2014 a shared file system (such as NFS or AFS) between the execute
2015 machine and the submit machine and that ALLpaths that the job could
2016 open on the execute machine must be valid. The ability of the stan‐
2017 dard universe job to checkpoint, possibly to a checkpoint server, is
2018 not affected by this attribute. However, when the job resumes it
2019 will be expecting the same file system conditions that were present
2020 when the job checkpointed.
2021 COMMANDS FOR THE GRID
2027 azure_admin_key = <pathname>
2035 For grid type azurejobs, specifies the path and file name of a file
2037 that contains an SSH public key. This key can be used to log into
2038 the administrator account of the instance via SSH.
2039 azure_admin_username = <account name>
2045 For grid type azurejobs, specifies the name of an administrator
2047 account to be created in the instance. This account can be logged
2048 into via SSH.
2049 azure_auth_file = <pathname>
2055 For grid type azurejobs, specifies a path and file name of the
2057 authorization file that grants permission for HTCondor to use the
2058 Azure account. If it's not defined, then HTCondor will attempt to
2059 use the default credentials of the Azure CLI tools.
2060 azure_image = <image id>
2066 For grid type azurejobs, identifies the disk image to be used for
2068 the boot disk of the instance. This image must already be registered
2069 within Azure.
2070 azure_location = <image id>
2076 For grid type azurejobs, identifies the location within Azure where
2078 the instance should be run. As an example, one current location is
2079 centralus.
2080 azure_size = <machine type>
2086 For grid type azurejobs, the hardware configuration that the virtual
2088 machine instance is to run on.
2089 batch_queue = <queuename>
2095 Used for pbs, lsf, and sgegrid universe jobs. Specifies the name of
2097 the PBS/LSF/SGE job queue into which the job should be submitted. If
2098 not specified, the default queue is used.
2099 boinc_authenticator_file = <pathname>
2105 For grid type boincjobs, specifies a path and file name of the
2107 authorization file that grants permission for HTCondor to use the
2108 BOINC service. There is no default value when not specified.
2109 cream_attributes = <name=value;...;name=value>
2115 Provides a list of attribute/value pairs to be set in a CREAM job
2117 description of a grid universe job destined for the CREAM grid sys‐
2118 tem. The pairs are separated by semicolons, and written in New Clas‐
2119 sAd syntax.
2120 delegate_job_GSI_credentials_lifetime = <seconds>
2126 Specifies the maximum number of seconds for which delegated proxies
2128 should be valid. The default behavior when this command is not spec‐
2129 ified is determined by the configuration variable DELE‐
2130 GATE_JOB_GSI_CREDENTIALS_LIFETIME, which defaults to one day. A
2131 value of 0 indicates that the delegated proxy should be valid for as
2132 long as allowed by the credential used to create the proxy. This
2133 setting currently only applies to proxies delegated for non-grid
2134 jobs and for HTCondor-C jobs. It does not currently apply to globus
2135 grid jobs, which always behave as though this setting were 0. This
2136 variable has no effect if the configuration variable DELE‐
2137 GATE_JOB_GSI_CREDENTIALSis False, because in that case the job proxy
2138 is copied rather than delegated.
2139 ec2_access_key_id = <pathname>
2145 For grid type ec2jobs, identifies the file containing the access
2147 key.
2148 ec2_ami_id = <EC2 xMI ID>
2154 For grid type ec2jobs, identifies the machine image. Services com‐
2156 patible with the EC2 Query API may refer to these with abbreviations
2157 other than AMI, for example EMIis valid for Eucalyptus.
2158 ec2_availability_zone = <zone name>
2164 For grid type ec2jobs, specifies the Availability Zone that the
2166 instance should be run in. This command is optional, unless
2167 ec2_ebs_volumesis set. As an example, one current zone is us-
2168 east-1b.
2169 ec2_block_device_mapping = <block-device>:<kernel-device>,<block-
2175 device>:<kernel-device>, ...
2176 For grid type ec2jobs, specifies the block device to kernel device
2178 mapping. This command is optional.
2179 ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device name>,...
2185 For grid type ec2jobs, optionally specifies a list of Elastic Block
2187 Store (EBS) volumes to be made available to the instance and the
2188 device names they should have in the instance.
2189 ec2_elastic_ip = <elastic IP address>
2195 For grid type ec2jobs, and optional specification of an Elastic IP
2197 address that should be assigned to this instance.
2198 ec2_iam_profile_arn = <IAM profile ARN>
2204 For grid type ec2jobs, an Amazon Resource Name (ARN) identifying
2206 which Identity and Access Management (IAM) (instance) profile to as‐
2207 sociate with the instance.
2208 ec2_iam_profile_name= <IAM profile name>
2214 For grid type ec2jobs, a name identifying which Identity and Access
2216 Management (IAM) (instance) profile to associate with the instance.
2217 ec2_instance_type = <instance type>
2223 For grid type ec2jobs, identifies the instance type. Different ser‐
2225 vices may offer different instance types, so no default value is
2226 set.
2227 ec2_keypair = <ssh key-pair name>
2233 For grid type ec2jobs, specifies the name of an SSH key-pair that is
2235 already registered with the EC2 service. The associated private key
2236 can be used to sshinto the virtual machine once it is running.
2237 ec2_keypair_file = <pathname>
2243 For grid type ec2jobs, specifies the complete path and file name of
2245 a file into which HTCondor will write an SSH key for use with ec2
2246 jobs. The key can be used to sshinto the virtual machine once it is
2247 running. If ec2_keypairis specified for a job, ec2_keypair_fileis
2248 ignored.
2249 ec2_parameter_names = ParameterName1, ParameterName2, ...
2255 For grid type ec2jobs, a space or comma separated list of the names
2257 of additional parameters to pass when instantiating an instance.
2258 ec2_parameter_<name> = <value>
2264 For grid type ec2jobs, specifies the value for the correspondingly
2266 named (instance instantiation) parameter. <name>is the parameter
2267 name specified in the submit command ec2_parameter_names, but with
2268 any periods replaced by underscores.
2269 ec2_secret_access_key = <pathname>
2275 For grid type ec2jobs, specifies the path and file name containing
2277 the secret access key.
2278 ec2_security_groups = group1, group2, ...
2284 For grid type ec2jobs, defines the list of EC2 security groups which
2286 should be associated with the job.
2287 ec2_security_ids = id1, id2, ...
2293 For grid type ec2jobs, defines the list of EC2 security group IDs
2295 which should be associated with the job.
2296 ec2_spot_price = <bid>
2302 For grid type ec2jobs, specifies the spot instance bid, which is the
2304 most that the job submitter is willing to pay per hour to run this
2305 job.
2306 ec2_tag_names = <name0,name1,name...>
2312 For grid type ec2jobs, specifies the case of tag names that will be
2314 associated with the running instance. This is only necessary if a
2315 tag name case matters. By default the list will be automatically
2316 generated.
2317 ec2_tag_<name> = <value>
2323 For grid type ec2jobs, specifies a tag to be associated with the
2325 running instance. The tag name will be lower-cased, use
2326 ec2_tag_namesto change the case.
2327 WantNameTag = <True | False>
2333 For grid type ec2jobs, a job may request that its 'name' tag be
2335 (not) set by HTCondor. If the job does not otherwise specify any
2336 tags, not setting its name tag will eliminate a call by the EC2
2337 GAHP, improving performance.
2338 ec2_user_data = <data>
2344 For grid type ec2jobs, provides a block of data that can be accessed
2346 by the virtual machine. If both ec2_user_dataand
2347 ec2_user_data_fileare specified for a job, the two blocks of data
2348 are concatenated, with the data from this ec2_user_datasubmit com‐
2349 mand occurring first.
2350 ec2_user_data_file = <pathname>
2356 For grid type ec2jobs, specifies a path and file name whose contents
2358 can be accessed by the virtual machine. If both ec2_user_dataand
2359 ec2_user_data_fileare specified for a job, the two blocks of data
2360 are concatenated, with the data from that ec2_user_datasubmit com‐
2361 mand occurring first.
2362 ec2_vpc_ip = <a.b.c.d>
2368 For grid type ec2jobs, that are part of a Virtual Private Cloud
2370 (VPC), an optional specification of the IP address that this
2371 instance should have within the VPC.
2372 ec2_vpc_subnet = <subnet specification string>
2378 For grid type ec2jobs, an optional specification of the Virtual Pri‐
2380 vate Cloud (VPC) that this instance should be a part of.
2381 gce_account = <account name>
2387 For grid type gcejobs, specifies the Google cloud services account
2389 to use. If this submit command isn't specified, then a random
2390 account from the authorization file given by gce_auth_filewill be
2391 used.
2392 gce_auth_file = <pathname>
2398 For grid type gcejobs, specifies a path and file name of the autho‐
2400 rization file that grants permission for HTCondor to use the Google
2401 account. If this command is not specified, then the default file of
2402 the Google command-line tools will be used.
2403 gce_image = <image id>
2409 For grid type gcejobs, the identifier of the virtual machine image
2411 representing the HTCondor job to be run. This virtual machine image
2412 must already be register with GCE and reside in Google's Cloud Stor‐
2413 age service.
2414 gce_json_file = <pathname>
2420 For grid type gcejobs, specifies the path and file name of a file
2422 that contains JSON elements that should be added to the instance
2423 description submitted to the GCE service.
2424 gce_machine_type = <machine type>
2430 For grid type gcejobs, the long form of the URL that describes the
2432 machine configuration that the virtual machine instance is to run
2433 on.
2434 gce_metadata = <name=value,...,name=value>
2440 For grid type gcejobs, a comma separated list of name and value
2442 pairs that define metadata for a virtual machine instance that is an
2443 HTCondor job.
2444 gce_metadata_file = <pathname>
2450 For grid type gcejobs, specifies a path and file name of the file
2452 that contains metadata for a virtual machine instance that is an
2453 HTCondor job. Within the file, each name and value pair is on its
2454 own line; so, the pairs are separated by the newline character.
2455 gce_preemptible = <True | False>
2461 For grid type gcejobs, specifies whether the virtual machine
2463 instance should be preemptible. The default is for the instance to
2464 not be preemptible.
2465 globus_rematch = <ClassAd Boolean Expression>
2471 This expression is evaluated by the condor_gridmanagerwhenever:
2473 1. the globus_resubmitexpression evaluates to True
2475 2. the condor_gridmanagerdecides it needs to retry a submission
2477 (as when a previous submission failed to commit) If
2478 globus_rematchevaluates to True, then beforethe job is submitted
2479 again to globus, the condor_gridmanagerwill request that the con‐
2480 dor_schedddaemon renegotiate with the matchmaker (the con‐
2481 dor_negotiator). The result is this job will be matched again.
2482 globus_resubmit = <ClassAd Boolean Expression>
2488 The expression is evaluated by the condor_gridmanagereach time the
2490 condor_gridmanagergets a job ad to manage. Therefore, the expression
2491 is evaluated:
2492 1. when a grid universe job is first submitted to HTCondor-G
2494 2. when a grid universe job is released from the hold state
2496 3. when HTCondor-G is restarted (specifically, whenever the con‐
2498 dor_gridmanageris restarted) If the expression evaluates to True,
2499 then any previous submission to the grid universe will be forgot‐
2500 ten and this job will be submitted again as a fresh submission to
2501 the grid universe. This may be useful if there is a desire to
2502 give up on a previous submission and try again. Note that this
2503 may result in the same job running more than once. Do not treat
2504 this operation lightly.
2505 globus_rsl = <RSL-string>
2511 Used to provide any additional Globus RSL string attributes which
2513 are not covered by other submit description file commands or job
2514 attributes. Used for griduniversejobs, where the grid resource has a
2515 grid-type-stringof gt2.
2516 grid_resource = <grid-type-string> <grid-specific-parameter-list>
2522 For each grid-type-stringvalue, there are further type-specific val‐
2524 ues that must specified. This submit description file command allows
2525 each to be given in a space-separated list. Allowable grid-type-
2526 stringvalues are batch, condor, cream, ec2, gt2, gt5, lsf, nor‐
2527 dugrid, pbs, sge, and unicore. The HTCondor manual chapter on Grid
2528 Computing details the variety of grid types.
2529 For a grid-type-stringof batch, the single parameter is the name of
2531 the local batch system, and will be one of pbs, lsf, or sge.
2532 For a grid-type-stringof condor, the first parameter is the name of
2534 the remote condor_schedddaemon. The second parameter is the name of
2535 the pool to which the remote condor_schedddaemon belongs.
2536 For a grid-type-stringof cream, there are three parameters. The
2538 first parameter is the web services address of the CREAM server. The
2539 second parameter is the name of the batch system that sits behind
2540 the CREAM server. The third parameter identifies a site-specific
2541 queue within the batch system.
2542 For a grid-type-stringof ec2, one additional parameter specifies the
2544 EC2 URL.
2545 For a grid-type-stringof gt2, the single parameter is the name of
2547 the pre-WS GRAM resource to be used.
2548 For a grid-type-stringof gt5, the single parameter is the name of
2550 the pre-WS GRAM resource to be used, which is the same as for the
2551 grid-type-stringof gt2.
2552 For a grid-type-stringof lsf, no additional parameters are used.
2554 For a grid-type-stringof nordugrid, the single parameter is the name
2556 of the NorduGrid resource to be used.
2557 For a grid-type-stringof pbs, no additional parameters are used.
2559 For a grid-type-stringof sge, no additional parameters are used.
2561 For a grid-type-stringof unicore, the first parameter is the name of
2563 the Unicore Usite to be used. The second parameter is the name of
2564 the Unicore Vsite to be used.
2565 keystore_alias = <name>
2571 A string to locate the certificate in a Java keystore file, as used
2573 for a unicorejob.
2574 keystore_file = <pathname>
2580 The complete path and file name of the Java keystore file containing
2582 the certificate to be used for a unicorejob.
2583 keystore_passphrase_file = <pathname>
2589 The complete path and file name to the file containing the
2591 passphrase protecting a Java keystore file containing the certifi‐
2592 cate. Relevant for a unicorejob.
2593 MyProxyCredentialName = <symbolic name>
2599 The symbolic name that identifies a credential to the MyProxyserver.
2601 This symbolic name is set as the credential is initially stored on
2602 the server (using myproxy-init).
2603 MyProxyHost = <host>:<port>
2609 The Internet address of the host that is the MyProxyserver. The
2611 hostmay be specified by either a host name (as in head.example.com)
2612 or an IP address (of the form 123.456.7.8). The portnumber is an
2613 integer.
2614 MyProxyNewProxyLifetime = <number-of-minutes>
2620 The new lifetime (in minutes) of the proxy after it is refreshed.
2622 MyProxyPassword = <password>
2628 The password needed to refresh a credential on the MyProxyserver.
2630 This password is set when the user initially stores credentials on
2631 the server (using myproxy-init). As an alternative to using MyProxy‐
2632 Passwordin the submit description file, the password may be speci‐
2633 fied as a command line argument to condor_submitwith the -passwor‐
2634 dargument.
2635 MyProxyRefreshThreshold = <number-of-seconds>
2641 The time (in seconds) before the expiration of a proxy that the
2643 proxy should be refreshed. For example, if MyProxyRefreshThresholdis
2644 set to the value 600, the proxy will be refreshed 10 minutes before
2645 it expires.
2646 MyProxyServerDN = <credential subject>
2652 A string that specifies the expected Distinguished Name (credential
2654 subject, abbreviated DN) of the MyProxyserver. It must be specified
2655 when the MyProxyserver DN does not follow the conventional naming
2656 scheme of a host credential. This occurs, for example, when the
2657 MyProxyserver DN begins with a user credential.
2658 nordugrid_rsl = <RSL-string>
2664 Used to provide any additional RSL string attributes which are not
2666 covered by regular submit description file parameters. Used when the
2667 universeis grid, and the type of grid system is nordugrid.
2668 transfer_error = <True | False>
2674 For jobs submitted to the grid universe only. If True, then the
2676 error output (from stderr) from the job is transferred from the
2677 remote machine back to the submit machine. The name of the file
2678 after transfer is given by the errorcommand. If False, no transfer
2679 takes place (from the remote machine to submit machine), and the
2680 name of the file is given by the errorcommand. The default value is
2681 True.
2682 transfer_input = <True | False>
2688 For jobs submitted to the grid universe only. If True, then the job
2690 input (stdin) is transferred from the machine where the job was sub‐
2691 mitted to the remote machine. The name of the file that is trans‐
2692 ferred is given by the inputcommand. If False, then the job's input
2693 is taken from a pre-staged file on the remote machine, and the name
2694 of the file is given by the inputcommand. The default value is True.
2695 For transferring files other than stdin, see transfer_input_files.
2697 transfer_output = <True | False>
2703 For jobs submitted to the grid universe only. If True, then the out‐
2705 put (from stdout) from the job is transferred from the remote
2706 machine back to the submit machine. The name of the file after
2707 transfer is given by the outputcommand. If False, no transfer takes
2708 place (from the remote machine to submit machine), and the name of
2709 the file is given by the outputcommand. The default value is True.
2710 For transferring files other than stdout, see transfer_output_files.
2712 use_x509userproxy = <True | False>
2718 Set this command to Trueto indicate that the job requires an X.509
2720 user proxy. If x509userproxyis set, then that file is used for the
2721 proxy. Otherwise, the proxy is looked for in the standard locations.
2722 If x509userproxyis set or if the job is a grid universe job of grid
2723 type gt2, gt5, cream, or nordugrid, then the value of use_x509user‐
2724 proxyis forced to True. Defaults to False.
2725 x509userproxy = <full-pathname>
2731 Used to override the default path name for X.509 user certificates.
2733 The default location for X.509 proxies is the /tmpdirectory, which
2734 is generally a local file system. Setting this value would allow
2735 HTCondor to access the proxy in a shared file system (for example,
2736 AFS). HTCondor will use the proxy specified in the submit descrip‐
2737 tion file first. If nothing is specified in the submit description
2738 file, it will use the environment variable X509_USER_PROXY. If that
2739 variable is not present, it will search in the default location.
2740 Note that proxies are only valid for a limited time. Condor_submit
2741 will not submit a job with an expired proxy, it will return an
2742 error. Also, if the configuration parameter CRED_MIN_TIME_LEFT is
2743 set to some number of seconds, and if the proxy will expire before
2744 that many seconds, condor_submit will also refuse to submit the job.
2745 That is, if CRED_MIN_TIME_LEFT is set to 60, condor_submit will
2746 refuse to submit a job whose proxy will expire 60 seconds from the
2747 time of submission.
2748 x509userproxyis relevant when the universeis vanilla, or when the
2750 universeis gridand the type of grid system is one of gt2, gt5, con‐
2751 dor, cream, or nordugrid. Defining a value causes the proxy to be
2752 delegated to the execute machine. Further, VOMS attributes defined
2753 in the proxy will appear in the job ClassAd.
2754 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2760 hold_kill_sig = <signal-number>
2768 For the scheduler universe only, signal-numberis the signal deliv‐
2770 ered to the job when the job is put on hold with condor_hold. sig‐
2771 nal-numbermay be either the platform-specific name or value of the
2772 signal. If this command is not present, the value of kill_sigis
2773 used.
2774 jar_files = <file_list>
2780 Specifies a list of additional JAR files to include when using the
2782 Java universe. JAR files will be transferred along with the exe‐
2783 cutable and automatically added to the classpath.
2784 java_vm_args = <argument_list>
2790 Specifies a list of additional arguments to the Java VM itself, When
2792 HTCondor runs the Java program, these are the arguments that go
2793 before the class name. This can be used to set VM-specific arguments
2794 like stack size, garbage-collector arguments and initial property
2795 values.
2796 machine_count = <max>
2802 For the parallel universe, a single value (max) is required. It is
2804 neither a maximum or minimum, but the number of machines to be dedi‐
2805 cated toward running the job.
2806 remove_kill_sig = <signal-number>
2812 For the scheduler universe only, signal-numberis the signal deliv‐
2814 ered to the job when the job is removed with condor_rm. signal-num‐
2815 bermay be either the platform-specific name or value of the signal.
2816 This example shows it both ways for a Linux signal:
2817 remove_kill_sig = SIGUSR1
2819 remove_kill_sig = 10
2820 If this command is not present, the value of kill_sigis used.
2822 COMMANDS FOR THE VM UNIVERSE
2828 vm_disk = file1:device1:permission1, file2:device2:permission2:format2,
2836 ...
2837 A list of comma separated disk files. Each disk file is specified by
2839 4 colon separated fields. The first field is the path and file name
2840 of the disk file. The second field specifies the device. The third
2841 field specifies permissions, and the optional fourth field specifies
2842 the image format. If a disk file will be transferred by HTCondor,
2843 then the first field should just be the simple file name (no path
2844 information).
2845 An example that specifies two disk files:
2847 vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2849 vm_checkpoint = <True | False>
2855 A boolean value specifying whether or not to take checkpoints. If
2857 not specified, the default value is False. In the current implemen‐
2858 tation, setting both vm_checkpointand vm_networkingto Truedoes not
2859 yet work in all cases. Networking cannot be used if a vm universe
2860 job uses a checkpoint in order to continue execution after migration
2861 to another machine.
2862 vm_macaddr = <MACAddr>
2868 Defines that MAC address that the virtual machine's network inter‐
2870 face should have, in the standard format of six groups of two hexa‐
2871 decimal digits separated by colons.
2872 vm_memory = <MBytes-of-memory>
2878 The amount of memory in MBytes that a vm universe job requires.
2880 vm_networking = <True | False>
2886 Specifies whether to use networking or not. In the current implemen‐
2888 tation, setting both vm_checkpointand vm_networkingto Truedoes not
2889 yet work in all cases. Networking cannot be used if a vm universe
2890 job uses a checkpoint in order to continue execution after migration
2891 to another machine.
2892 vm_networking_type = <nat | bridge >
2898 When vm_networkingis True, this definition augments the job's
2900 requirements to match only machines with the specified networking.
2901 If not specified, then either networking type matches.
2902 vm_no_output_vm = <True | False>
2908 When True, prevents HTCondor from transferring output files back to
2910 the machine from which the vm universe job was submitted. If not
2911 specified, the default value is False.
2912 vm_type = <vmware | xen | kvm>
2918 Specifies the underlying virtual machine software that this job
2920 expects.
2921 vmware_dir = <pathname>
2927 The complete path and name of the directory where VMware-specific
2929 files and applications such as the VMDK (Virtual Machine Disk For‐
2930 mat) and VMX (Virtual Machine Configuration) reside. This command is
2931 optional; when not specified, all relevant VMware image files are to
2932 be listed using transfer_input_files.
2933 vmware_should_transfer_files = <True | False>
2939 Specifies whether HTCondor will transfer VMware-specific files
2941 located as specified by vmware_dirto the execute machine (True) or
2942 rely on access through a shared file system (False). Omission of
2943 this required command (for VMware vm universe jobs) results in an
2944 error message from condor_submit, and the job will not be submitted.
2945 vmware_snapshot_disk = <True | False>
2951 When True, causes HTCondor to utilize a VMware snapshot disk for new
2953 or modified files. If not specified, the default value is True.
2954 xen_initrd = <image-file>
2960 When xen_kernelgives a file name for the kernel image to use, this
2962 optional command may specify a path to a ramdisk (initrd) image
2963 file. If the image file will be transferred by HTCondor, then the
2964 value should just be the simple file name (no path information).
2965 xen_kernel = <included | path-to-kernel>
2971 A value of includedspecifies that the kernel is included in the disk
2973 file. If not one of these values, then the value is a path and file
2974 name of the kernel to be used. If a kernel file will be transferred
2975 by HTCondor, then the value should just be the simple file name (no
2976 path information).
2977 xen_kernel_params = <string>
2983 A string that is appended to the Xen kernel command line.
2985 xen_root = <string>
2991 A string that is appended to the Xen kernel command line to specify
2993 the root device. This string is required when xen_kernelgives a path
2994 to a kernel. Omission for this required case results in an error
2995 message during submission.
2996 COMMANDS FOR THE DOCKER UNIVERSE
3002 docker_image = < image-name >
3010 Defines the name of the Docker image that is the basis for the
3012 docker container.
3013 ADVANCED COMMANDS
3019 accounting_group = <accounting-group-name>
3027 Causes jobs to negotiate under the given accounting group. This
3029 value is advertised in the job ClassAd as AcctGroup. The HTCondor
3030 Administrator's manual contains more information about accounting
3031 groups.
3032 accounting_group_user = <accounting-group-user-name>
3038 Sets the user name associated with the accounting group name for
3040 resource usage accounting purposes. If not set, defaults to the
3041 value of the job ClassAd attribute Owner. This value is advertised
3042 in the job ClassAd as AcctGroupUser. If an accounting group has not
3043 been set with the accounting_groupcommand, this command is ignored.
3044 concurrency_limits = <string-list>
3050 A list of resources that this job needs. The resources are presumed
3052 to have concurrency limits placed upon them, thereby limiting the
3053 number of concurrent jobs in execution which need the named
3054 resource. Commas and space characters delimit the items in the list.
3055 Each item in the list is a string that identifies the limit, or it
3056 is a ClassAd expression that evaluates to a string, and it is evalu‐
3057 ated in the context of machine ClassAd being considered as a match.
3058 Each item in the list also may specify a numerical value identifying
3059 the integer number of resources required for the job. The syntax
3060 follows the resource name by a colon character ( : ) and the numeri‐
3061 cal value. Details on concurrency limits are in the HTCondor Admin‐
3062 istrator's manual.
3063 concurrency_limits_expr = <ClassAd String Expression>
3069 A ClassAd expression that represents the list of resources that this
3071 job needs after evaluation. The ClassAd expression may specify
3072 machine ClassAd attributes that are evaluated against a matched
3073 machine. After evaluation, the list sets concurrency_limits.
3074 copy_to_spool = <True | False>
3080 If copy_to_spoolis True, then condor_submitcopies the executable to
3082 the local spool directory before running it on a remote host. As
3083 copying can be quite time consuming and unnecessary, the default
3084 value is Falsefor all job universes other than the standard uni‐
3085 verse. When False, condor_submitdoes not copy the executable to a
3086 local spool directory. The default is Truein standard universe,
3087 because resuming execution from a checkpoint can only be guaranteed
3088 to work using precisely the same executable that created the check‐
3089 point.
3090 coresize = <size>
3096 Should the user's program abort and produce a core file, coresize‐
3098 specifies the maximum size in bytes of the core file which the user
3099 wishes to keep. If coresizeis not specified in the command file, the
3100 system's user resource limit coredumpsizeis used (note that core‐
3101 dumpsizeis not an HTCondor parameter – it is an operating sys‐
3102 tem parameter that can be viewed with the limitor ulimitcommand on
3103 Unix and in the Registry on Windows). A value of -1 results in no
3104 limits being applied to the core file size. If HTCondor is running
3105 as root, a coresizesetting greater than the system coredumpsizelimit
3106 will override the system setting; if HTCondor is notrunning as root,
3107 the system coredumpsizelimit will override coresize.
3108 cron_day_of_month = <Cron-evaluated Day>
3114 The set of days of the month for which a deferral time applies. The
3116 HTCondor User's manual section on Time Scheduling for Job Execution
3117 has further details.
3118 cron_day_of_week = <Cron-evaluated Day>
3124 The set of days of the week for which a deferral time applies. The
3126 HTCondor User's manual section on Time Scheduling for Job Execution
3127 has further details.
3128 cron_hour = <Cron-evaluated Hour>
3134 The set of hours of the day for which a deferral time applies. The
3136 HTCondor User's manual section on Time Scheduling for Job Execution
3137 has further details.
3138 cron_minute = <Cron-evaluated Minute>
3144 The set of minutes within an hour for which a deferral time applies.
3146 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3147 tion has further details.
3148 cron_month = <Cron-evaluated Month>
3154 The set of months within a year for which a deferral time applies.
3156 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3157 tion has further details.
3158 cron_prep_time = <ClassAd Integer Expression>
3164 Analogous to deferral_prep_time. The number of seconds prior to a
3166 job's deferral time that the job may be matched and sent to an exe‐
3167 cution machine.
3168 cron_window = <ClassAd Integer Expression>
3174 Analogous to the submit command deferral_window. It allows cron jobs
3176 that miss their deferral time to begin execution.
3177 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3179 tion has further details.
3180 dagman_log = <pathname>
3186 DAGMan inserts this command to specify an event log that it watches
3188 to maintain the state of the DAG. If the logcommand is not specified
3189 in the submit file, DAGMan uses the logcommand to specify the event
3190 log.
3191 deferral_prep_time = <ClassAd Integer Expression>
3197 The number of seconds prior to a job's deferral time that the job
3199 may be matched and sent to an execution machine.
3200 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3202 tion has further details.
3203 deferral_time = <ClassAd Integer Expression>
3209 Allows a job to specify the time at which its execution is to begin,
3211 instead of beginning execution as soon as it arrives at the execu‐
3212 tion machine. The deferral time is an expression that evaluates to a
3213 Unix Epoch timestamp (the number of seconds elapsed since 00:00:00
3214 on January 1, 1970, Coordinated Universal Time). Deferral time is
3215 evaluated with respect to the execution machine. This option delays
3216 the start of execution, but not the matching and claiming of a
3217 machine for the job. If the job is not available and ready to begin
3218 execution at the deferral time, it has missed its deferral time. A
3219 job that misses its deferral time will be put on hold in the queue.
3220 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3222 tion has further details.
3223 Due to implementation details, a deferral time may not be used for
3225 scheduler universe jobs.
3226 deferral_window = <ClassAd Integer Expression>
3232 The deferral window is used in conjunction with the defer‐
3234 ral_timecommand to allow jobs that miss their deferral time to begin
3235 execution.
3236 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3238 tion has further details.
3239 description = <string>
3245 A string that sets the value of the job ClassAd attribute JobDe‐
3247 scription. When set, tools which display the executable such as con‐
3248 dor_qwill instead use this string.
3249 email_attributes = <list-of-job-ad-attributes>
3255 A comma-separated list of attributes from the job ClassAd. These
3257 attributes and their values will be included in the e-mail notifica‐
3258 tion of job completion.
3259 image_size = <size>
3265 Advice to HTCondor specifying the maximum virtual image size to
3267 which the job will grow during its execution. HTCondor will then
3268 execute the job only on machines which have enough resources, (such
3269 as virtual memory), to support executing the job. If not specified,
3270 HTCondor will automatically make a (reasonably accurate) estimate
3271 about the job's size and adjust this estimate as the program runs.
3272 If specified and underestimated, the job may crash due to the
3273 inability to acquire more address space; for example, if mal‐
3274 loc()fails. If the image size is overestimated, HTCondor may have
3275 difficulty finding machines which have the required resources.
3276 sizeis specified in KiB. For example, for an image size of 8 MiB,
3277 sizeshould be 8000.
3278 initialdir = <directory-path>
3284 Used to give jobs a directory with respect to file input and output.
3286 Also provides a directory (on the machine from which the job is sub‐
3287 mitted) for the job event log, when a full path is not specified.
3288 For vanilla universe jobs where there is a shared file system, it is
3290 the current working directory on the machine where the job is exe‐
3291 cuted.
3292 For vanilla or grid universe jobs where file transfer mechanisms are
3294 utilized (there is nota shared file system), it is the directory on
3295 the machine from which the job is submitted where the input files
3296 come from, and where the job's output files go to.
3297 For standard universe jobs, it is the directory on the machine from
3299 which the job is submitted where the condor_shadowdaemon runs; the
3300 current working directory for file input and output accomplished
3301 through remote system calls.
3302 For scheduler universe jobs, it is the directory on the machine from
3304 which the job is submitted where the job runs; the current working
3305 directory for file input and output with respect to relative path
3306 names.
3307 Note that the path to the executable is notrelative to initialdir;
3309 if it is a relative path, it is relative to the directory in which
3310 the condor_submitcommand is run.
3311 job_ad_information_attrs = <attribute-list>
3317 A comma-separated list of job ClassAd attribute names. The named
3319 attributes and their values are written to the job event log when‐
3320 ever any event is being written to the log. This implements the same
3321 thing as the configuration variable EVENT_LOG_INFORMATION_ATTRS(see
3322 page ), but it applies to the job event log, instead of the system
3323 event log.
3324 JobBatchName = <batch_name>
3330 Set the batch name for this submit. The batch name is displayed by
3332 condor_q-batch. It is intended for use by users to give meaningful
3333 names to their jobs and to influence how condor_qgroups jobs for
3334 display. This value in a submit file can be overridden by specifying
3335 the -batch-nameargument on the condor_submitcommand line.
3336 job_lease_duration = <number-of-seconds>
3342 For vanilla, parallel, VM, and java universe jobs only, the duration
3344 in seconds of a job lease. The default value is 2,400, or forty min‐
3345 utes. If a job lease is not desired, the value can be explicitly set
3346 to 0 to disable the job lease semantics. The value can also be a
3347 ClassAd expression that evaluates to an integer. The HTCondor User's
3348 manual section on Special Environment Considerations has further
3349 details.
3350 job_machine_attrs = <attr1, attr2, ...>
3356 A comma and/or space separated list of machine attribute names that
3358 should be recorded in the job ClassAd in addition to the ones speci‐
3359 fied by the condor_schedddaemon's system configuration variable SYS‐
3360 TEM_JOB_MACHINE_ATTRS. When there are multiple run attempts, history
3361 of machine attributes from previous run attempts may be kept. The
3362 number of run attempts to store may be extended beyond the system-
3363 specified history length by using the submit file command
3364 job_machine_attrs_history_length. A machine attribute named Xwill be
3365 inserted into the job ClassAd as an attribute named MachineAttrX0.
3366 The previous value of this attribute will be named MachineAttrX1,
3367 the previous to that will be named MachineAttrX2, and so on, up to
3368 the specified history length. A history of length 1 means that only
3369 MachineAttrX0will be recorded. The value recorded in the job ClassAd
3370 is the evaluation of the machine attribute in the context of the job
3371 ClassAd when the condor_schedddaemon initiates the start up of the
3372 job. If the evaluation results in an Undefinedor Errorresult, the
3373 value recorded in the job ad will be Undefinedor Error, respec‐
3374 tively.
3375 want_graceful_removal = <boolean expression>
3381 When True, this causes a graceful shutdown of the job when the job
3383 is removed or put on hold, giving it time to clean up or save state.
3384 Otherwise, the job is abruptly killed. The default is false.
3385 kill_sig = <signal-number>
3391 When HTCondor needs to kick a job off of a machine, it will send the
3393 job the signal specified by signal-number. signal-numberneeds to be
3394 an integer which represents a valid signal on the execution machine.
3395 For jobs submitted to the standard universe, the default value is
3396 the number for SIGTSTP which tells the HTCondor libraries to initi‐
3397 ate a checkpoint of the process. For jobs submitted to other uni‐
3398 verses, the default value, when not defined, is SIGTERM , which is
3399 the standard way to terminate a program in Unix.
3400 kill_sig_timeout = <seconds>
3406 This submit command should no longer be used as of HTCondor version
3408 7.7.3; use job_max_vacate_timeinstead. If job_max_vacate_timeis not
3409 defined, this defines the number of seconds that HTCondor should
3410 wait following the sending of the kill signal defined by kill_sigand
3411 forcibly killing the job. The actual amount of time between sending
3412 the signal and forcibly killing the job is the smallest of this
3413 value and the configuration variable KILLING_TIMEOUT, as defined on
3414 the execute machine.
3415 load_profile = <True | False>
3421 When True, loads the account profile of the dedicated run account
3423 for Windows jobs. May not be used with run_as_owner.
3424 match_list_length = <integer value>
3430 Defaults to the value zero (0). When match_list_lengthis defined
3432 with an integer value greater than zero (0), attributes are inserted
3433 into the job ClassAd. The maximum number of attributes defined is
3434 given by the integer value. The job ClassAds introduced are given as
3435 LastMatchName0 = "most-recent-Name"
3437 LastMatchName1 = "next-most-recent-Name"
3438 The value for each introduced ClassAd is given by the value of the
3440 Nameattribute from the machine ClassAd of a previous execution
3441 (match). As a job is matched, the definitions for these attributes
3442 will roll, with LastMatchName1 becoming LastMatchName2 , Last‐
3443 MatchName0 becoming LastMatchName1 , and LastMatchName0 being set
3444 by the most recent value of the Nameattribute.
3445 An intended use of these job attributes is in the requirements
3447 expression. The requirements can allow a job to prefer a match with
3448 either the same or a different resource than a previous match.
3449 job_max_vacate_time = <integer expression>
3455 An integer-valued expression (in seconds) that may be used to adjust
3457 the time given to an evicted job for gracefully shutting down. If
3458 the job's setting is less than the machine's, the job's is used. If
3459 the job's setting is larger than the machine's, the result depends
3460 on whether the job has any excess retirement time. If the job has
3461 more retirement time left than the machine's max vacate time set‐
3462 ting, then retirement time will be converted into vacating time, up
3463 to the amount requested by the job.
3464 Setting this expression does not affect the job's resource require‐
3466 ments or preferences. For a job to only run on a machine with a min‐
3467 imum MachineMaxVacateTime, or to preferentially run on such
3468 machines, explicitly specify this in the requirements and/or rank
3469 expressions.
3470 max_job_retirement_time = <integer expression>
3476 An integer-valued expression (in seconds) that does nothing unless
3478 the machine that runs the job has been configured to provide retire‐
3479 ment time. Retirement time is a grace period given to a job to fin‐
3480 ish when a resource claim is about to be preempted. The default
3481 behavior in many cases is to take as much retirement time as the
3482 machine offers, so this command will rarely appear in a submit
3483 description file.
3484 When a resource claim is to be preempted, this expression in the
3486 submit file specifies the maximum run time of the job (in seconds,
3487 since the job started). This expression has no effect, if it is
3488 greater than the maximum retirement time provided by the machine
3489 policy. If the resource claim is notpreempted, this expression and
3490 the machine retirement policy are irrelevant. If the resource claim
3491 ispreempted the job will be allowed to run until the retirement time
3492 expires, at which point it is hard-killed. The job will be soft-
3493 killed when it is getting close to the end of retirement in order to
3494 give it time to gracefully shut down. The amount of lead-time for
3495 soft-killing is determined by the maximum vacating time granted to
3496 the job.
3497 Standard universe jobs and any jobs running with nice_userpriority
3499 have a default max_job_retirement_timeof 0, so no retirement time is
3500 utilized by default. In all other cases, no default value is pro‐
3501 vided, so the maximum amount of retirement time is utilized by
3502 default.
3503 Setting this expression does not affect the job's resource require‐
3505 ments or preferences. For a job to only run on a machine with a min‐
3506 imum MaxJobRetirementTime, or to preferentially run on such
3507 machines, explicitly specify this in the requirements and/or rank
3508 expressions.
3509 nice_user = <True | False>
3515 Normally, when a machine becomes available to HTCondor, HTCondor
3517 decides which job to run based upon user and job priorities. Setting
3518 nice_userequal to Truetells HTCondor not to use your regular user
3519 priority, but that this job should have last priority among all
3520 users and all jobs. So jobs submitted in this fashion run only on
3521 machines which no other non-nice_user job wants — a true bot‐
3522 tom-feeder job! This is very handy if a user has some jobs they wish
3523 to run, but do not wish to use resources that could instead be used
3524 to run other people's HTCondor jobs. Jobs submitted in this fashion
3525 have "nice-user."prepended to the owner name when viewed from con‐
3526 dor_qor condor_userprio. The default value is False.
3527 noop_job = <ClassAd Boolean Expression>
3533 When this boolean expression is True, the job is immediately removed
3535 from the queue, and HTCondor makes no attempt at running the job.
3536 The log file for the job will show a job submitted event and a job
3537 terminated event, along with an exit code of 0, unless the user
3538 specifies a different signal or exit code.
3539 noop_job_exit_code = <return value>
3545 When noop_jobis in the submit description file and evaluates to
3547 True, this command allows the job to specify the return value as
3548 shown in the job's log file job terminated event. If not specified,
3549 the job will show as having terminated with status 0. This overrides
3550 any value specified with noop_job_exit_signal.
3551 noop_job_exit_signal = <signal number>
3557 When noop_jobis in the submit description file and evaluates to
3559 True, this command allows the job to specify the signal number that
3560 the job's log event will show the job having terminated with.
3561 remote_initialdir = <directory-path>
3567 The path specifies the directory in which the job is to be executed
3569 on the remote machine. This is currently supported in all universes
3570 except for the standard universe.
3571 rendezvousdir = <directory-path>
3577 Used to specify the shared file system directory to be used for file
3579 system authentication when submitting to a remote scheduler. Should
3580 be a path to a preexisting directory.
3581 run_as_owner = <True | False>
3587 A boolean value that causes the job to be run under the login of the
3589 submitter, if supported by the joint configuration of the submit and
3590 execute machines. On Unix platforms, this defaults to True, and on
3591 Windows platforms, it defaults to False. May not be used with
3592 load_profile. See the HTCondor manual Platform-Specific Information
3593 chapter for administrative details on configuring Windows to support
3594 this option.
3595 stack_size = <size in bytes>
3601 This command applies only to Linux platform jobs that are not stan‐
3603 dard universe jobs. An integer number of bytes, representing the
3604 amount of stack space to be allocated for the job. This value
3605 replaces the default allocation of stack space, which is unlimited
3606 in size.
3607 submit_event_notes = <note>
3613 A string that is appended to the submit event in the job's log file.
3615 For DAGMan jobs, the string DAG Node:and the node's name is automat‐
3616 ically defined for submit_event_notes, causing the logged submit
3617 event to identify the DAG node job submitted.
3618 +<attribute> = <value>
3624 A line that begins with a '+' (plus) character instructs condor_sub‐
3626 mitto insert the given attributeinto the job ClassAd with the given
3627 value. Note that setting an attribute should not be used in place of
3628 one of the specific commands listed above. Often, the command name
3629 does not directly correspond to an attribute name; furthermore, many
3630 submit commands result in actions more complex than simply setting
3631 an attribute or attributes. See for a list of HTCondor job
3632 attributes.
3633 MACROS AND COMMENTS
3639 In addition to commands, the submit description file can contain macros
3641 and comments.
3642 Macros
3644 Parameterless macros in the form of $(macro_name:default initial
3646 value)may be used anywhere in HTCondor submit description files to
3647 provide textual substitution at submit time. Macros can be defined
3648 by lines in the form of
3649 <macro_name> = <string>
3651 Two pre-defined macros are supplied by the submit description file
3653 parser. The $(Cluster)or $(ClusterId)macro supplies the value of the
3654 ClusterIdjob ClassAd attribute, and the $(Process)or $(ProcId)macro
3655 supplies the value of the ProcIdjob ClassAd attribute. These macros
3656 are intended to aid in the specification of input/output files,
3657 arguments, etc., for clusters with lots of jobs, and/or could be
3658 used to supply an HTCondor process with its own cluster and process
3659 numbers on the command line.
3660 The $(Node)macro is defined for parallel universe jobs, and is espe‐
3662 cially relevant for MPI applications. It is a unique value assigned
3663 for the duration of the job that essentially identifies the machine
3664 (slot) on which a program is executing. Values assigned start at 0
3665 and increase monotonically. The values are assigned as the parallel
3666 job is about to start.
3667 Recursive definition of macros is permitted. An example of a con‐
3669 struction that works is the following:
3670 foo = bar
3672 foo = snap $(foo)
3673 As a result, foo = snap bar.
3675 Note that both left- and right- recursion works, so
3677 foo = bar
3679 foo = $(foo) snap
3680 has as its result foo = bar snap.
3682 The construction
3684 foo = $(foo) bar
3686 by itself will notwork, as it does not have an initial base case.
3688 Mutually recursive constructions such as:
3689 B = bar
3691 C = $(B)
3692 B = $(C) boo
3693 will notwork, and will fill memory with expansions.
3695 A default value may be specified, for use if the macro has no defi‐
3697 nition. Consider the example
3698 D = $(E:24)
3700 Where Eis not defined within the submit description file, the
3702 default value 24 is used, resulting in
3703 D = 24
3705 This is of limited value, as the scope of macro substitution is the
3707 submit description file. Thus, either the macro is or is not defined
3708 within the submit description file. If the macro is defined, then
3709 the default value is useless. If the macro is not defined, then
3710 there is no point in using it in a submit command.
3711 To use the dollar sign character ( $ ) as a literal, without macro
3713 expansion, use
3714 $(DOLLAR)
3716 In addition to the normal macro, there is also a special kind of
3718 macro called a substitution macrothat allows the substitution of a
3719 machine ClassAd attribute value defined on the resource machine
3720 itself (gotten after a match to the machine has been made) into spe‐
3721 cific commands within the submit description file. The substitution
3722 macro is of the form:
3723 $$(attribute)
3725 As this form of the substitution macro is only evaluated within the
3727 context of the machine ClassAd, use of a scope resolution prefix
3728 TARGET.or MY.is not allowed.
3729 A common use of this form of the substitution macro is for the het‐
3731 erogeneous submission of an executable:
3732 executable = povray.$$(OpSys).$$(Arch)
3734 Values for the OpSysand Archattributes are substituted at match time
3736 for any given resource. This example allows HTCondor to automati‐
3737 cally choose the correct executable for the matched machine.
3738 An extension to the syntax of the substitution macro provides an
3740 alternative string to use if the machine attribute within the sub‐
3741 stitution macro is undefined. The syntax appears as:
3742 $$(attribute:string_if_attribute_undefined)
3744 An example using this extended syntax provides a path name to a
3746 required input file. Since the file can be placed in different loca‐
3747 tions on different machines, the file's path name is given as an
3748 argument to the program.
3749 arguments = $$(input_file_path:/usr/foo)
3751 On the machine, if the attribute input_file_pathis not defined, then
3753 the path /usr/foois used instead.
3754 A further extension to the syntax of the substitution macro allows
3756 the evaluation of a ClassAd expression to define the value. In this
3757 form, the expression may refer to machine attributes by prefacing
3758 them with the TARGET.scope resolution prefix. To place a ClassAd
3759 expression into the substitution macro, square brackets are added to
3760 delimit the expression. The syntax appears as:
3761 $$([ClassAd expression])
3763 An example of a job that uses this syntax may be one that wants to
3765 know how much memory it can use. The application cannot detect this
3766 itself, as it would potentially use all of the memory on a multi-
3767 slot machine. So the job determines the memory per slot, reducing it
3768 by 10% to account for miscellaneous overhead, and passes this as a
3769 command line argument to the application. In the submit description
3770 file will be
3771 arguments = --memory $$([TARGET.Memory * 0.9])
3773 To insert two dollar sign characters ( $$ ) as literals into a Clas‐
3775 sAd string, use
3776 $$(DOLLARDOLLAR)
3778 The environment macro, $ENV, allows the evaluation of an environment
3780 variable to be used in setting a submit description file command.
3781 The syntax used is
3782 $ENV(variable)
3784 An example submit description file command that uses this function‐
3786 ality evaluates the submitter's home directory in order to set the
3787 path and file name of a log file:
3788 log = $ENV(HOME)/jobs/logfile
3790 The environment variable is evaluated when the submit description
3792 file is processed.
3793 The $RANDOM_CHOICE macro allows a random choice to be made from a
3795 given list of parameters at submission time. For an expression, if
3796 some randomness needs to be generated, the macro may appear as
3797 $RANDOM_CHOICE(0,1,2,3,4,5,6)
3799 When evaluated, one of the parameters values will be chosen.
3801 Comments
3807 Blank lines and lines beginning with a pound sign ('#') character
3809 are ignored by the submit description file parser.
3810
3816 While processing the queuecommand in a submit file or from the command
3817 line, condor_submitwill set the values of several automatic submit
3818 variables so that they can be referred to by statements in the submit
3819 file. With the exception of Cluster and Process, if these variables are
3820 set by the submit file, they will not be modified during queueprocess‐
3821 ing.
3822 ClusterId
3824 Set to the integer value that the ClusterIdattribute that the job
3826 ClassAd will have when the job is submitted. All jobs in a single
3827 submit will normally have the same value for the ClusterId. If the
3828 -dry-runargument is specified, The value will be 1.
3829 Cluster
3835 Alternate name for the ClusterId submit variable. Before HTCondor
3837 version 8.4 this was the only name.
3838 ProcId
3844 Set to the integer value that the ProcIdattribute of the job ClassAd
3846 will have when the job is submitted. The value will start at 0 and
3847 increment by 1 for each job submitted.
3848 Process
3854 Alternate name for the ProcId submit variable. Before HTCondor ver‐
3856 sion 8.4 this was the only name.
3857 Node
3863 For parallel universes, set to the value #pArAlLeLnOdE# or #MpInOdE#
3865 depending on the parallel universe type For other universes it is
3866 set to nothing.
3867 Step
3873 Set to the step value as it varies from 0 to N-1 where N is the num‐
3875 ber provided on the queueargument. This variable changes at the same
3876 rate as ProcId when it changes at all. For submit files that don't
3877 make use of the queue number option, Step will always be 0. For sub‐
3878 mit files that don't make use of any of the foreach options, Step
3879 and ProcId will always be the same.
3880 ItemIndex
3886 Set to the index within the item list being processed by the various
3888 queue foreach options. For submit files that don't make use of any
3889 queue foreach list, ItemIndex will always be 0 For submit files that
3890 make use of a slice to select only some items in a foreach list,
3891 ItemIndex will only be set to selected values.
3892 Row
3898 Alternate name for ItemIndex.
3900 Item
3906 when a queue foreach option is used and no variable list is sup‐
3908 plied, this variable will be set to the value of the current item.
3909 The automatic variables below are set before parsing the submit file,
3913 and will not vary during processing unless the submit file itself sets
3914 them.
3915 ARCH
3917 Set to the CPU architecture of the machine running condor_submit.
3919 The value will be the same as the automatic configuration variable
3920 of the same name.
3921 OPSYS
3927 Set to the name of the operating system on the machine running con‐
3929 dor_submit. The value will be the same as the automatic configura‐
3930 tion variable of the same name.
3931 OPSYSANDVER
3937 Set to the name and major version of the operating system on the
3939 machine running condor_submit. The value will be the same as the
3940 automatic configuration variable of the same name.
3941 OPSYSMAJORVER
3947 Set to the major version of the operating system on the machine run‐
3949 ning condor_submit. The value will be the same as the automatic con‐
3950 figuration variable of the same name.
3951 OPSYSVER
3957 Set to the version of the operating system on the machine running
3959 condor_submit. The value will be the same as the automatic configu‐
3960 ration variable of the same name.
3961 SPOOL
3967 Set to the full path of the HTCondor spool directory. The value will
3969 be the same as the automatic configuration variable of the same
3970 name.
3971 IsLinux
3977 Set to true if the operating system of the machine running con‐
3979 dor_submitis a Linux variant. Set to false otherwise.
3980 IsWindows
3986 Set to true if the operating system of the machine running con‐
3988 dor_submitis a Microsoft Windows variant. Set to false otherwise.
3989 SUBMIT_FILE
3995 Set to the full pathname of the submit file being processed by con‐
3997 dor_submit. If submit statements are read from standard input, it is
3998 set to nothing.
3999
4003 condor_submitwill exit with a status value of 0 (zero) upon success,
4004 and a non-zero value upon failure.
4005
4007 * Submit Description File Example 1: This example queues three jobs
4008 for execution by HTCondor. The first will be given command line
4009 arguments of 15and 2000, and it will write its standard output to
4010 foo.out1. The second will be given command line arguments of 30and
4011 2000, and it will write its standard output to foo.out2. Similarly
4012 the third will have arguments of 45and 6000, and it will use
4013 foo.out3for its standard output. Standard error output (if any) from
4014 all three programs will appear in foo.error.
4015 ####################
4019 #
4020 # submit description file
4021 # Example 1: queuing multiple jobs with differing
4022 # command line arguments and output files.
4023 #
4024 ####################
4025 Executable = foo
4027 Universe = vanilla
4028 Arguments = 15 2000
4030 Output = foo.out0
4031 Error = foo.err0
4032 Queue
4033 Arguments = 30 2000
4035 Output = foo.out1
4036 Error = foo.err1
4037 Queue
4038 Arguments = 45 6000
4040 Output = foo.out2
4041 Error = foo.err2
4042 Queue
4043 Or you can get the same results as the above submit file by using a
4045 list of arguments with the Queue statement
4046 ####################
4050 #
4051 # submit description file
4052 # Example 1b: queuing multiple jobs with differing
4053 # command line arguments and output files, alternate syntax
4054 #
4055 ####################
4056 Executable = foo
4058 Universe = vanilla
4059 # generate different output and error filenames for each process
4061 Output = foo.out$(Process)
4062 Error = foo.err$(Process)
4063 Queue Arguments From (
4065 15 2000
4066 30 2000
4067 45 6000
4068 )
4069 * Submit Description File Example 2: This submit description file
4073 example queues 150 runs of program foowhich must have been compiled
4074 and linked for an Intel x86 processor running RHEL 3. HTCondor will
4075 not attempt to run the processes on machines which have less than 32
4076 Megabytes of physical memory, and it will run them on machines which
4077 have at least 64 Megabytes, if such machines are available. Stdin,
4078 stdout, and stderr will refer to in.0, out.0, and err.0for the first
4079 run of this program (process 0). Stdin, stdout, and stderr will
4080 refer to in.1, out.1, and err.1for process 1, and so forth. A log
4081 file containing entries about where and when HTCondor runs, takes
4082 checkpoints, and migrates processes in this cluster will be written
4083 into file foo.log.
4084 ####################
4088 #
4089 # Example 2: Show off some fancy features including
4090 # use of pre-defined macros and logging.
4091 #
4092 ####################
4093 Executable = foo
4095 Universe = standard
4096 Requirements = OpSys == "LINUX" && Arch =="INTEL"
4097 Rank = Memory >= 64
4098 Request_Memory = 32 Mb
4099 Image_Size = 28 Mb
4100 Error = err.$(Process)
4102 Input = in.$(Process)
4103 Output = out.$(Process)
4104 Log = foo.log
4105 Queue 150
4106 * Submit Description File Example 3: This example targets the
4110 /bin/sleepprogram to run only on a platform running a RHEL 6 operat‐
4111 ing system. The example presumes that the pool contains machines
4112 running more than one version of Linux, and this job needs the par‐
4113 ticular operating system to run correctly.
4114 ####################
4118 #
4119 # Example 3: Run on a RedHat 6 machine
4120 #
4121 ####################
4122 Universe = vanilla
4123 Executable = /bin/sleep
4124 Arguments = 30
4125 Requirements = (OpSysAndVer == "RedHat6")
4126 Error = err.$(Process)
4128 Input = in.$(Process)
4129 Output = out.$(Process)
4130 Log = sleep.log
4131 Queue
4132 * Command Line example: The following command uses the -appendoption
4136 to add two commands before the job(s) is queued. A log file and an
4137 error log file are specified. The submit description file is
4138 unchanged.
4139 condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
4141 Note that each of the added commands is contained within quote marks
4142 because there are space characters within the command.
4143 * periodic_removeexample: A job should be removed from the queue, if
4147 the total suspension time of the job is more than half of the run
4148 time of the job.
4149 Including the command
4151 periodic_remove = CumulativeSuspensionTime >
4153 ((RemoteWallClockTime - CumulativeSuspensionTime)
4154 / 2.0) in the submit description file causes this to happen.
4155
4159 * For security reasons, HTCondor will refuse to run any jobs submit‐
4160 ted by user root (UID = 0) or by a user whose default group is group
4161 wheel (GID = 0). Jobs submitted by user root or a user with a
4162 default group of wheel will appear to sit forever in the queue in an
4163 idle state.
4164 * All path names specified in the submit description file must be
4168 less than 256 characters in length, and command line arguments must
4169 be less than 4096 characters in length; otherwise, condor_submit‐
4170 gives a warning message but the jobs will not execute properly.
4171 * Somewhat understandably, behavior gets bizarre if the user makes
4175 the mistake of requesting multiple HTCondor jobs to write to the
4176 same file, and/or if the user alters any files that need to be
4177 accessed by an HTCondor job which is still in the queue. For exam‐
4178 ple, the compressing of data or output files before an HTCondor job
4179 has completed is a common mistake.
4180 * To disable checkpointing for Standard Universe jobs, include the
4184 line:
4185 +WantCheckpoint = False
4187 in the submit description file before the queue command(s).
4189
4191 HTCondor User Manual
4192
4194 Center for High Throughput Computing, University of Wiscon‐
4195 sin–Madison
4196
4198 Copyright © 1990-2019 Center for High Throughput Computing, Computer
4199 Sciences Department, University of Wisconsin-Madison, Madison, WI. All
4200 Rights Reserved. Licensed under the Apache License, Version 2.0.
4201 date condor_submit(1)