1condor_submit(1) General Commands Manual condor_submit(1)
2
3
4
6 condor_submitQueue jobs for execution under HTCondor
7
9 condor_submit[-terse] [-verbose] [-unused] [-file submit_file] [-name
10 schedd_name] [-remote schedd_name] [-addr <ip:port>] [-pool pool_name]
11 [-disable] [-password passphrase] [-debug] [-appendcommand...] [-batch-
12 name batch_name] [-spool] [-dump filename] [-interactive] [-allow-crlf-
13 script] [-dry-run] [-maxjobs number-of-jobs] [-single-cluster] [-stm
14 method] [<submit-variable>=<value>] [submit description file] [-queue
15 queue_arguments]
16
18 condor_submitis the program for submitting jobs for execution under
19 HTCondor. condor_submitrequires one or more submit description commands
20 to direct the queuing of jobs. These commands may come from a file,
21 standard input, the command line, or from some combination of these.
22 One submit description may contain specifications for the queuing of
23 many HTCondor jobs at once. A single invocation of condor_submitmay
24 cause one or more clusters. A cluster is a set of jobs specified in the
25 submit description between queuecommands for which the executable is
26 not changed. It is advantageous to submit multiple jobs as a single
27 cluster because:
28
29 * Much less memory is used by the scheduler to hold the same number
30 of jobs.
31
32 * Only one copy of the checkpoint file is needed to represent all
33 jobs in a cluster until they begin execution.
34
35 * There is much less overhead involved for HTCondor to start the
36 next job in a cluster than for HTCondor to start a new cluster. This
37 can make a big difference when submitting lots of short jobs.
38
39 Multiple clusters may be specified within a single submit description.
40 Each cluster must specify a single executable.
41
42 The job ClassAd attribute ClusterIdidentifies a cluster.
43
44 The submit description fileargument is the path and file name of the
45 submit description file. If this optional argument is the dash charac‐
46 ter (-), then the commands are taken from standard input. If -is speci‐
47 fied for the submit description file, -verboseis implied; this can be
48 overridden by specifying -terse.
49
50 If no submit discription fileargument is given, and no -queueargument
51 is given, commands are taken automatically from standard input.
52
53 Note that submission of jobs from a Windows machine requires a stashed
54 password to allow HTCondor to impersonate the user submitting the job.
55 To stash a password, use the condor_store_credcommand. See the manual
56 page for details.
57
58 For lengthy lines within the submit description file, the backslash ( \
59 ) is a line continuation character. Placing the backslash at the end of
60 a line causes the current line's command to be continued with the next
61 line of the file. Submit description files may contain comments. A com‐
62 ment is any line beginning with a pound character ( # ).
63
65 -terse
66
67 Terse output - display JobId ranges only.
68
69
70
71
72
73 -verbose
74
75 Verbose output - display the created job ClassAd
76
77
78
79
80
81 -unused
82
83 As a default, causes no warnings to be issued about user-defined
84 macros not being used within the submit description file. The mean‐
85 ing reverses (toggles) when the configuration variable
86 WARN_ON_UNUSED_SUBMIT_FILE_MACROSis set to the non default value of
87 False. Printing the warnings can help identify spelling errors of
88 submit description file commands. The warnings are sent to stderr.
89
90
91
92
93
94 -file submit_file
95
96 Use submit_fileas the submit discription file. This is equivalent to
97 providing submit_fileas an argument without the preceeding -file.
98
99
100
101
102
103 -name schedd_name
104
105 Submit to the specified condor_schedd. Use this option to submit to
106 a condor_scheddother than the default local one. schedd_nameis the
107 value of the NameClassAd attribute on the machine where the con‐
108 dor_schedddaemon runs.
109
110
111
112
113
114 -remote schedd_name
115
116 Submit to the specified condor_schedd, spooling all required input
117 files over the network connection. schedd_nameis the value of the
118 NameClassAd attribute on the machine where the condor_schedddaemon
119 runs. This option is equivalent to using both -nameand -spool.
120
121
122
123
124
125 -addr <ip:port>
126
127 Submit to the condor_scheddat the IP address and port given by the
128 sinful stringargument <ip:port>.
129
130
131
132
133
134 -pool pool_name
135
136 Look in the specified pool for the condor_scheddto submit to. This
137 option is used with -nameor -remote.
138
139
140
141
142
143 -disable
144
145 Disable file permission checks when submitting a job for read per‐
146 missions on all input files, such as those defined by commands
147 inputand transfer_input_files, as well as write permission to output
148 files, such as a log file defined by logand output files defined
149 with outputor transfer_output_files.
150
151
152
153
154
155 -password passphrase
156
157 Specify a password to the MyProxyserver.
158
159
160
161
162
163 -debug
164
165 Cause debugging information to be sent to stderr, based on the value
166 of the configuration variable TOOL_DEBUG.
167
168
169
170
171
172 -append command
173
174 Augment the commands in the submit description file with the given
175 command. This command will be considered to immediately precede the
176 queuecommand within the submit description file, and come after all
177 other previous commands. If the commandspecifies a queuecommand, as
178 in the example
179
180 condor_submit mysubmitfile -append "queue input in A, B, C"
181
182 then the entire -appendcommand line option and its arguments are
183 converted to
184
185 condor_submit mysubmitfile -queue input in A, B, C
186
187 The submit description file is not modified. Multiple commands are
188 specified by using the -appendoption multiple times. Each new com‐
189 mand is given in a separate -appendoption. Commands with spaces in
190 them will need to be enclosed in double quote marks.
191
192
193
194
195
196 -batch-name batch_name
197
198 Set the batch name for this submit. The batch name is displayed by
199 condor_q-batch. It is intended for use by users to give meaningful
200 names to their jobs and to influence how condor_qgroups jobs for
201 display. Use of this argument takes precedence over a batch name
202 specified in the submit description file itself.
203
204
205
206
207
208 -spool
209
210 Spool all required input files, job event log, and proxy over the
211 connection to the condor_schedd. After submission, modify local
212 copies of the files without affecting your jobs. Any output files
213 for completed jobs need to be retrieved with condor_transfer_data.
214
215
216
217
218
219 -dump filename
220
221 Sends all ClassAds to the specified file, instead of to the con‐
222 dor_schedd.
223
224
225
226
227
228 -interactive
229
230 Indicates that the user wants to run an interactive shell on an exe‐
231 cute machine in the pool. This is equivalent to creating a submit
232 description file of a vanilla universe sleep job, and then running
233 condor_ssh_to_jobby hand. Without any additional arguments, con‐
234 dor_submitwith the -interactive flag creates a dummy vanilla uni‐
235 verse job that sleeps, submits it to the local scheduler, waits for
236 the job to run, and then launches condor_ssh_to_jobto run a shell.
237 If the user would like to run the shell on a machine that matches a
238 particular requirementsexpression, the submit description file is
239 specified, and it will contain the expression. Note that all policy
240 expressions specified in the submit description file are honored,
241 but any executableor universecommands are overwritten to be sleep
242 and vanilla. The job ClassAd attribute InteractiveJobis set to
243 Trueto identify interactive jobs for condor_startdpolicy usage.
244
245
246
247
248
249 -allow-crlf-script
250
251 Changes the check for an invalid line ending on the executable
252 script's #!line from an ERROR to a WARNING. The #!line will be
253 ignored by Windows, so it won't matter if it is invalid; but Unix
254 and Linux will not run a script that has a Windows/DOS line ending
255 on the first line of the script. So condor_submitwill not allow such
256 a script to be submitted as the job's executable unless this option
257 is supplied.
258
259
260
261
262
263 -dry-run file
264
265 Parse the submit description file, sending the resulting job ClassAd
266 to the file given by file, but do not submit the job(s). This per‐
267 mits observation of the job specification, and it facilitates debug‐
268 ging the submit description file contents. If fileis -, the output
269 is written to stdout.
270
271
272
273
274
275 -maxjobs number-of-jobs
276
277 If the total number of jobs specified by the submit description file
278 is more than the integer value given by number-of-jobs, then no jobs
279 are submitted for execution and an error message is generated. A 0
280 or negative value for the number-of-jobscauses no limit to be
281 imposed.
282
283
284
285
286
287 -single-cluster
288
289 If the jobs specified by the submit description file causes more
290 than a single cluster value to be assigned, then no jobs are submit‐
291 ted for execution and an error message is generated.
292
293
294
295
296
297 -stm method
298
299 Specify the method use to move a sandbox into HTCondor. methodis one
300 of stm_use_schedd_onlyor stm_use_transferd.
301
302
303
304
305
306 <submit-variable>=<value>
307
308 Defines a submit command or submit variable with a value, and parses
309 it as if it was placed at the beginning of the submit description
310 file. The submit description file is not changed. To correctly parse
311 the condor_submitcommand line, this option must be specified without
312 white space characters before and after the equals sign (=), or the
313 entire option must be surrounded by double quote marks.
314
315
316
317
318
319 -queue queue_arguments
320
321 A command line specification of how many jobs to queue, which is
322 only permitted if the submit description file does not have a
323 queuecommand. The queue_argumentsare the same as may be within a
324 submit description file. The parsing of the queue_argumentsfinishes
325 at the end of the line or when a dash character (-) is encountered.
326 Therefore, its best placement within the command line will be at the
327 end of the command line.
328
329 On a Unix command line, the shell expands file globs before parsing
330 occurs.
331
332
333
334
335
337 Note: more information on submitting HTCondor jobs can be found here: .
338
339 As of version 8.5.6, the condor_submitlanguage supports multi-line val‐
340 ues in commands. The syntax is the same as the configuration language
341 (see more details here: ).
342
343 Each submit description file describes one or more clusters of jobs to
344 be placed in the HTCondor execution pool. All jobs in a cluster must
345 share the same executable, but they may have different input and output
346 files, and different program arguments. The submit description file is
347 generally the last command-line argument to condor_submit. If the sub‐
348 mit description file argument is omitted, condor_submitwill read the
349 submit description from standard input.
350
351 The submit description file must contain at least one executablecommand
352 and at least one queuecommand. All of the other commands have default
353 actions.
354
355 Note that a submit file that contains more than one executable command
356 will produce multiple clusters when submitted. This is not generally
357 recommended, and is not allowed for submit files that are run as DAG
358 node jobs by condor_dagman.
359
360 The commands which can appear in the submit description file are numer‐
361 ous. They are listed here in alphabetical order by category.
362
363 BASIC COMMANDS
364
365
366
367
368
369
370
371 arguments = <argument_list>
372
373 List of arguments to be supplied to the executable as part of the
374 command line.
375
376 In the javauniverse, the first argument must be the name of the
377 class containing main.
378
379 There are two permissible formats for specifying arguments, identi‐
380 fied as the old syntax and the new syntax. The old syntax supports
381 white space characters within arguments only in special circum‐
382 stances; when used, the command line arguments are represented in
383 the job ClassAd attribute Args. The new syntax supports uniform
384 quoting of white space characters within arguments; when used, the
385 command line arguments are represented in the job ClassAd attribute
386 Arguments.
387
388 Old Syntax
389
390 In the old syntax, individual command line arguments are delimited
391 (separated) by space characters. To allow a double quote mark in an
392 argument, it is escaped with a backslash; that is, the two character
393 sequence \" becomes a single double quote mark within an argument.
394
395 Further interpretation of the argument string differs depending on
396 the operating system. On Windows, the entire argument string is
397 passed verbatim (other than the backslash in front of double quote
398 marks) to the Windows application. Most Windows applications will
399 allow spaces within an argument value by surrounding the argument
400 with double quotes marks. In all other cases, there is no further
401 interpretation of the arguments.
402
403 Example:
404
405
406
407 arguments = one \"two\" 'three'
408
409 Produces in Unix vanilla universe:
410
411
412
413 argument 1: one
414 argument 2: "two"
415 argument 3: 'three'
416
417 New Syntax
418
419 Here are the rules for using the new syntax:
420
421
422
423 1. The entire string representing the command line arguments is
424 surrounded by double quote marks. This permits the white space
425 characters of spaces and tabs to potentially be embedded within a
426 single argument. Putting the double quote mark within the argu‐
427 ments is accomplished by escaping it with another double quote
428 mark.
429
430
431
432 2. The white space characters of spaces or tabs delimit argu‐
433 ments.
434
435
436
437 3. To embed white space characters of spaces or tabs within a
438 single argument, surround the entire argument with single quote
439 marks.
440
441
442
443 4. To insert a literal single quote mark, escape it within an
444 argument already delimited by single quote marks by adding
445 another single quote mark.
446
447
448
449 Example:
450
451 arguments = "3 simple arguments"
452
453 Produces:
454
455 argument 1: 3
456 argument 2: simple
457 argument 3: arguments
458
459 Another example:
460
461 arguments = "one 'two with spaces' 3"
462
463 Produces:
464
465 argument 1: one
466 argument 2: two with spaces
467 argument 3: 3
468
469 And yet another example:
470
471 arguments = "one ""two"" 'spacey ”quoted” argument'"
472
473 Produces:
474
475 argument 1: one
476 argument 2: "two"
477 argument 3: spacey 'quoted' argument
478
479 Notice that in the new syntax, the backslash has no special meaning.
480 This is for the convenience of Windows users.
481
482
483
484
485
486 environment = <parameter_list>
487
488 List of environment variables.
489
490 There are two different formats for specifying the environment vari‐
491 ables: the old format and the new format. The old format is retained
492 for backward-compatibility. It suffers from a platform-dependent
493 syntax and the inability to insert some special characters into the
494 environment.
495
496 The new syntax for specifying environment values:
497
498
499
500 1. Put double quote marks around the entire argument string. This
501 distinguishes the new syntax from the old. The old syntax does
502 not have double quote marks around it. Any literal double quote
503 marks within the string must be escaped by repeating the double
504 quote mark.
505
506
507
508 2. Each environment entry has the form
509
510
511
512 <name>=<value>
513
514
515
516 3. Use white space (space or tab characters) to separate environ‐
517 ment entries.
518
519
520
521 4. To put any white space in an environment entry, surround the
522 space and as much of the surrounding entry as desired with single
523 quote marks.
524
525
526
527 5. To insert a literal single quote mark, repeat the single quote
528 mark anywhere inside of a section surrounded by single quote
529 marks.
530
531
532
533 Example:
534
535
536
537 environment = "one=1 two=""2"" three='spacey ”quoted”
538 value'"
539
540 Produces the following environment entries:
541
542
543
544 one=1
545 two="2"
546 three=spacey 'quoted' value
547
548 Under the old syntax, there are no double quote marks surrounding
549 the environment specification. Each environment entry remains of the
550 form
551
552 <name>=<value>
553
554 Under Unix, list multiple environment entries by separating them
555 with a semicolon (;). Under Windows, separate multiple entries with
556 a vertical bar (|). There is no way to insert a literal semicolon
557 under Unix or a literal vertical bar under Windows. Note that spaces
558 are accepted, but rarely desired, characters within parameter names
559 and values, because they are treated as literal characters, not sep‐
560 arators or ignored white space. Place spaces within the parameter
561 list only if required.
562
563 A Unix example:
564
565
566
567 environment = one=1;two=2;three="quotes have no 'special' meaning"
568
569 This produces the following:
570
571
572
573 one=1
574 two=2
575 three="quotes have no 'special' meaning"
576
577 If the environment is set with the environmentcommand andgetenvis
578 also set to true, values specified with environmentoverride values
579 in the submitter's environment (regardless of the order of the envi‐
580 ronmentand getenvcommands).
581
582
583
584
585
586 error = <pathname>
587
588 A path and file name used by HTCondor to capture any error messages
589 the program would normally write to the screen (that is, this file
590 becomes stderr). A path is given with respect to the file system of
591 the machine on which the job is submitted. The file is written (by
592 the job) in the remote scratch directory of the machine where the
593 job is executed. When the job exits, the resulting file is trans‐
594 ferred back to the machine where the job was submitted, and the path
595 is utilized for file placement. If not specified, the default value
596 of /dev/nullis used for submission to a Unix machine. If not speci‐
597 fied, error messages are ignored for submission to a Windows
598 machine. More than one job should not use the same error file, since
599 this will cause one job to overwrite the errors of another. If
600 HTCondor detects that the error and output files for a job are the
601 same, it will run the job such that the output and error data is
602 merged.
603
604
605
606
607
608 executable = <pathname>
609
610 An optional path and a required file name of the executable file for
611 this job cluster. Only one executablecommand within a submit
612 description file is guaranteed to work properly. More than one often
613 works.
614
615 If no path or a relative path is used, then the executable file is
616 presumed to be relative to the current working directory of the user
617 as the condor_submitcommand is issued.
618
619 If submitting into the standard universe, then the named executable
620 must have been re-linked with the HTCondor libraries (such as via
621 the condor_compilecommand). If submitting into the vanilla universe
622 (the default), then the named executable need not be re-linked and
623 can be any process which can run in the background (shell scripts
624 work fine as well). If submitting into the Java universe, then the
625 argument must be a compiled .classfile.
626
627
628
629
630
631 getenv = <True | False>
632
633 If getenvis set to True, then condor_submitwill copy all of the
634 user's current shell environment variables at the time of job sub‐
635 mission into the job ClassAd. The job will therefore execute with
636 the same set of environment variables that the user had at submit
637 time. Defaults to False.
638
639 If the environment is set with the environmentcommand andgetenvis
640 also set to true, values specified with environmentoverride values
641 in the submitter's environment (regardless of the order of the envi‐
642 ronmentand getenvcommands).
643
644
645
646
647
648 input = <pathname>
649
650 HTCondor assumes that its jobs are long-running, and that the user
651 will not wait at the terminal for their completion. Because of this,
652 the standard files which normally access the terminal, (stdin, std‐
653 out, and stderr), must refer to files. Thus, the file name specified
654 with inputshould contain any keyboard input the program requires
655 (that is, this file becomes stdin). A path is given with respect to
656 the file system of the machine on which the job is submitted. The
657 file is transferred before execution to the remote scratch directory
658 of the machine where the job is executed. If not specified, the
659 default value of /dev/nullis used for submission to a Unix machine.
660 If not specified, input is ignored for submission to a Windows
661 machine. For grid universe jobs, inputmay be a URL that the Globus
662 tool globus_url_copyunderstands.
663
664 Note that this command does notrefer to the command-line arguments
665 of the program. The command-line arguments are specified by the
666 argumentscommand.
667
668
669
670
671
672 log = <pathname>
673
674 Use logto specify a file name where HTCondor will write a log file
675 of what is happening with this job cluster, called a job event log.
676 For example, HTCondor will place a log entry into this file when and
677 where the job begins running, when the job produces a checkpoint, or
678 moves (migrates) to another machine, and when the job completes.
679 Most users find specifying a logfile to be handy; its use is recom‐
680 mended. If no logentry is specified, HTCondor does not create a log
681 for this cluster. If a relative path is specified, it is relative to
682 the current working directory as the job is submitted or the direc‐
683 tory specified by submit command initialdiron the submit machine.
684
685
686
687
688
689 log_xml = <True | False>
690
691 If log_xmlis True, then the job event log file will be written in
692 ClassAd XML. If not specified, XML is not used. Note that the file
693 is an XML fragment; it is missing the file header and footer. Do not
694 mix XML and non-XML within a single file. If multiple jobs write to
695 a single job event log file, ensure that all of the jobs specify
696 this option in the same way.
697
698
699
700
701
702 notification = <Always | Complete | Error | Never>
703
704 Owners of HTCondor jobs are notified by e-mail when certain events
705 occur. If defined by Always, the owner will be notified whenever the
706 job produces a checkpoint, as well as when the job completes. If
707 defined by Complete, the owner will be notified when the job termi‐
708 nates. If defined by Error, the owner will only be notified if the
709 job terminates abnormally, (as defined by JobSuccessExitCode, if
710 defined) or if the job is placed on hold because of a failure, and
711 not by user request. If defined by Never(the default), the owner
712 will not receive e-mail, regardless to what happens to the job. The
713 HTCondor User's manual documents statistics included in the e-mail.
714
715
716
717
718
719 notify_user = <email-address>
720
721 Used to specify the e-mail address to use when HTCondor sends e-mail
722 about a job. If not specified, HTCondor defaults to using the e-mail
723 address defined by
724
725 job-owner@UID_DOMAIN
726
727 where the configuration variable UID_DOMAINis specified by the
728 HTCondor site administrator. If UID_DOMAINhas not been specified,
729 HTCondor sends the e-mail to:
730
731 job-owner@submit-machine-name
732
733
734
735
736
737 output = <pathname>
738
739 The outputfile captures any information the program would ordinarily
740 write to the screen (that is, this file becomes stdout). A path is
741 given with respect to the file system of the machine on which the
742 job is submitted. The file is written (by the job) in the remote
743 scratch directory of the machine where the job is executed. When the
744 job exits, the resulting file is transferred back to the machine
745 where the job was submitted, and the path is utilized for file
746 placement. If not specified, the default value of /dev/nullis used
747 for submission to a Unix machine. If not specified, output is
748 ignored for submission to a Windows machine. Multiple jobs should
749 not use the same output file, since this will cause one job to over‐
750 write the output of another. If HTCondor detects that the error and
751 output files for a job are the same, it will run the job such that
752 the output and error data is merged.
753
754 Note that if a program explicitly opens and writes to a file, that
755 file should notbe specified as the outputfile.
756
757
758
759
760
761 priority = <integer>
762
763 An HTCondor job priority can be any integer, with 0 being the
764 default. Jobs with higher numerical priority will run before jobs
765 with lower numerical priority. Note that this priority is on a per
766 user basis. One user with many jobs may use this command to order
767 his/her own jobs, and this will have no effect on whether or not
768 these jobs will run ahead of another user's jobs.
769
770 Note that the priority setting in an HTCondor submit file will be
771 overridden by condor_dagmanif the submit file is used for a node in
772 a DAG, and the priority of the node within the DAG is non-zero (see
773 for more details).
774
775
776
777
778
779 queue [<int expr>]
780
781 Places zero or more copies of the job into the HTCondor queue.
782
783
784
785 queue
786
787 [<int expr>] [<varname>] in[slice] <list of items>Places zero or
788 more copies of the job in the queue based on items in a <list of
789 items>
790
791
792
793 queue
794
795 [<int expr>] [<varname>] matching[files | dirs] [slice] <list of
796 items with file globbing>] Places zero or more copies of the job in
797 the queue based on files that match a <list of items with file glob‐
798 bing>
799
800
801
802 queue
803
804 [<int expr>] [<list of varnames>] from[slice] <file name> | <list of
805 items>] Places zero or more copies of the job in the queue based on
806 lines from the submit file or from <file name>
807
808 The optional argument <int expr>specifies how many times to repeat
809 the job submission for a given set of arguments. It may be an inte‐
810 ger or an expression that evaluates to an integer, and it defaults
811 to 1. All but the first form of this command are various ways of
812 specifying a list of items. When these forms are used <int expr>jobs
813 will be queued for each item in the list. The in, matchingand
814 fromkeyword indicates how the list will be specified.
815
816 * inThe list of items is an explicit comma and/or space separated
817 <list of items>. If the <list of items>begins with an open paren,
818 and the close paren is not on the same line as the open, then the
819 list continues until a line that begins with a close paren is
820 read from the submit file.
821
822 * matchingEach item in the <list of items with file globbing>will
823 be matched against the names of files and directories relative to
824 the current directory, the set of matching names is the resulting
825 list of items.
826
827 * filesOnly filenames will matched.
828
829 * dirsOnly directory names will be matched.
830
831 * from<file name> | <list of items>Each line from <file name>or
832 <list of items>is a single item, this allows for multiple vari‐
833 ables to be set for each item. Lines from <file name>or <list of
834 items>will be split on comma and/or space until there are values
835 for each of the variables specified in <list of varnames>. The
836 last variable will contain the remainder of the line. When the
837 <list of items>form is used, the list continues until the first
838 line that begins with a close paren, and lines beginning with
839 pound sign ('#') will be skipped. When using the <file name>form,
840 if the <file name>ends with |, then it will be executed as a
841 script whatever the script writes to stdoutwill be the list of
842 items. The optional argument <varname>or <list of varnames>is the
843 name or names of of variables that will be set to the value of
844 the current item when queuing the job. If no <varname>is speci‐
845 fied the variable ITEM will be used. Leading and trailing white‐
846 space be trimmed. The optional argument <slice>is a python style
847 slice selecting only some of the items in the list of items. Neg‐
848 ative step values are not supported.
849
850 A submit file may contain more than one queuestatement, and if
851 desired, any commands may be placed between subsequent queuecom‐
852 mands, such as new input, output, error, initialdir, or argu‐
853 mentscommands. This is handy when submitting multiple runs into one
854 cluster with one submit description file.
855
856
857
858
859
860 universe = <vanilla | standard | scheduler | local | grid | java | vm |
861 parallel | docker>
862
863 Specifies which HTCondor universe to use when running this job. The
864 HTCondor universe specifies an HTCondor execution environment.
865
866 The vanillauniverse is the default (except where the configuration
867 variable DEFAULT_UNIVERSEdefines it otherwise), and is an execution
868 environment for jobs which do not use HTCondor's mechanisms for tak‐
869 ing checkpoints; these are ones that have not been linked with the
870 HTCondor libraries. Use the vanillauniverse to submit shell scripts
871 to HTCondor.
872
873 The standarduniverse tells HTCondor that this job has been re-linked
874 via condor_compilewith the HTCondor libraries and therefore supports
875 taking checkpoints and remote system calls.
876
877 The scheduleruniverse is for a job that is to run on the machine
878 where the job is submitted. This universe is intended for a job that
879 acts as a metascheduler and will not be preempted.
880
881 The localuniverse is for a job that is to run on the machine where
882 the job is submitted. This universe runs the job immediately and
883 will not preempt the job.
884
885 The griduniverse forwards the job to an external job management sys‐
886 tem. Further specification of the griduniverse is done with the
887 grid_resourcecommand.
888
889 The javauniverse is for programs written to the Java Virtual
890 Machine.
891
892 The vmuniverse facilitates the execution of a virtual machine.
893
894 The paralleluniverse is for parallel jobs (e.g. MPI) that require
895 multiple machines in order to run.
896
897 The dockeruniverse runs a docker container as an HTCondor job.
898
899
900
901
902
903 COMMANDS FOR MATCHMAKING
904
905
906
907
908
909
910
911 rank = <ClassAd Float Expression>
912
913 A ClassAd Floating-Point expression that states how to rank machines
914 which have already met the requirements expression. Essentially,
915 rank expresses preference. A higher numeric value equals better
916 rank. HTCondor will give the job the machine with the highest rank.
917 For example,
918
919 request_memory = max({60, Target.TotalSlotMemory})
920 rank = Memory
921
922 asks HTCondor to find all available machines with more than 60
923 megabytes of memory and give to the job the machine with the most
924 amount of memory. The HTCondor User's Manual contains complete
925 information on the syntax and available attributes that can be used
926 in the ClassAd expression.
927
928
929
930
931
932 request_cpus = <num-cpus>
933
934 A requested number of CPUs (cores). If not specified, the number
935 requested will be 1. If specified, the expression
936
937 && (RequestCpus <= Target.Cpus)
938
939 is appended to the requirementsexpression for the job.
940
941 For pools that enable dynamic condor_startdprovisioning, specifies
942 the minimum number of CPUs requested for this job, resulting in a
943 dynamic slot being created with this many cores.
944
945
946
947
948
949 request_disk = <quantity>
950
951 The requested amount of disk space in KiB requested for this job. If
952 not specified, it will be set to the job ClassAd attribute
953 DiskUsage. The expression
954
955 && (RequestDisk <= Target.Disk)
956
957 is appended to the requirementsexpression for the job.
958
959 For pools that enable dynamic condor_startdprovisioning, a dynamic
960 slot will be created with at least this much disk space.
961
962 Characters may be appended to a numerical value to indicate units.
963 Kor KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers
964 of bytes. Gor GBindicates GiB, numbers of bytes. Tor TBindicates
965 TiB, numbers of bytes.
966
967
968
969
970
971 request_memory = <quantity>
972
973 The required amount of memory in MiB that this job needs to avoid
974 excessive swapping. If not specified and the submit command vm_memo‐
975 ryis specified, then the value specified for vm_memorydefines
976 request_memory. If neither request_memorynor vm_memoryis specified,
977 the value is set by the configuration variable JOB_DEFAULT_REQUEST‐
978 MEMORY. The actual amount of memory used by a job is represented by
979 the job ClassAd attribute MemoryUsage.
980
981 For pools that enable dynamic condor_startdprovisioning, a dynamic
982 slot will be created with at least this much RAM.
983
984 The expression
985
986 && (RequestMemory <= Target.Memory)
987
988 is appended to the requirementsexpression for the job.
989
990 Characters may be appended to a numerical value to indicate units.
991 Kor KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers
992 of bytes. Gor GBindicates GiB, numbers of bytes. Tor TBindicates
993 TiB, numbers of bytes.
994
995
996
997
998
999 request_<name> = <quantity>
1000
1001 The required amount of the custom machine resource identified by
1002 <name>that this job needs. The custom machine resource is defined in
1003 the machine's configuration. Machines that have available GPUs will
1004 define <name>to be GPUs.
1005
1006
1007
1008
1009
1010 requirements = <ClassAd Boolean Expression>
1011
1012 The requirements command is a boolean ClassAd expression which uses
1013 C-like operators. In order for any job in this cluster to run on a
1014 given machine, this requirements expression must evaluate to true on
1015 the given machine.
1016
1017 For scheduler and local universe jobs, the requirements expression
1018 is evaluated against the SchedulerClassAd which represents the the
1019 condor_schedddaemon running on the submit machine, rather than a
1020 remote machine. Like all commands in the submit description file, if
1021 multiple requirements commands are present, all but the last one are
1022 ignored. By default, condor_submitappends the following clauses to
1023 the requirements expression:
1024
1025 1. Arch and OpSys are set equal to the Arch and OpSys of the sub‐
1026 mit machine. In other words: unless you request otherwise, HTCon‐
1027 dor will give your job machines with the same architecture and
1028 operating system version as the machine running condor_submit.
1029
1030 2. Cpus >= RequestCpus, if the job ClassAd attribute RequestCpu‐
1031 sis defined.
1032
1033 3. Disk >= RequestDisk, if the job ClassAd attribute Request‐
1034 Diskis defined. Otherwise, Disk >= DiskUsage is appended to the
1035 requirements. The DiskUsageattribute is initialized to the size
1036 of the executable plus the size of any files specified in a
1037 transfer_input_filescommand. It exists to ensure there is enough
1038 disk space on the target machine for HTCondor to copy over both
1039 the executable and needed input files. The DiskUsageattribute
1040 represents the maximum amount of total disk space required by the
1041 job in kilobytes. HTCondor automatically updates the DiskUsageat‐
1042 tribute approximately every 20 minutes while the job runs with
1043 the amount of space being used by the job on the execute machine.
1044
1045 4. Memory >= RequestMemory, if the job ClassAd attribute Request‐
1046 Memoryis defined.
1047
1048 5. If Universe is set to Vanilla, FileSystemDomain is set equal
1049 to the submit machine's FileSystemDomain. View the requirements
1050 of a job which has already been submitted (along with everything
1051 else about the job ClassAd) with the command condor_q -l; see the
1052 command reference for condor_qon page . Also, see the HTCondor
1053 Users Manual for complete information on the syntax and available
1054 attributes that can be used in the ClassAd expression.
1055
1056
1057
1058
1059
1060 FILE TRANSFER COMMANDS
1061
1062
1063
1064
1065
1066
1067
1068 dont_encrypt_input_files = < file1,file2,file... >
1069
1070 A comma and/or space separated list of input files that are notto be
1071 network encrypted when transferred with the file transfer mechanism.
1072 Specification of files in this manner overrides configuration that
1073 would use encryption. Each input file must also be in the list given
1074 by transfer_input_files. When a path to an input file or directory
1075 is specified, this specifies the path to the file on the submit
1076 side. A single wild card character (*) may be used in each file
1077 name.
1078
1079
1080
1081
1082
1083 dont_encrypt_output_files = < file1,file2,file... >
1084
1085 A comma and/or space separated list of output files that are notto
1086 be network encrypted when transferred back with the file transfer
1087 mechanism. Specification of files in this manner overrides configu‐
1088 ration that would use encryption. The output file(s) must also
1089 either be in the list given by transfer_output_filesor be discovered
1090 and to be transferred back with the file transfer mechanism. When a
1091 path to an output file or directory is specified, this specifies the
1092 path to the file on the execute side. A single wild card character
1093 (*) may be used in each file name.
1094
1095
1096
1097
1098
1099 encrypt_execute_directory = <True | False>
1100
1101 Defaults to False. If set to True, HTCondor will encrypt the con‐
1102 tents of the remote scratch directory of the machine where the job
1103 is executed. This encryption is transparent to the job itself, but
1104 ensures that files left behind on the local disk of the execute
1105 machine, perhaps due to a system crash, will remain private. In
1106 addition, condor_submitwill append to the job's requirementsexpres‐
1107 sion
1108
1109 && (TARGET.HasEncryptExecuteDirectory)
1110
1111 to ensure the job is matched to a machine that is capable of
1112 encrypting the contents of the execute directory. This support is
1113 limited to Windows platforms that use the NTFS file system and Linux
1114 platforms with the ecryptfs-utilspackage installed.
1115
1116
1117
1118
1119
1120 encrypt_input_files = < file1,file2,file... >
1121
1122 A comma and/or space separated list of input files that are to be
1123 network encrypted when transferred with the file transfer mechanism.
1124 Specification of files in this manner overrides configuration that
1125 would notuse encryption. Each input file must also be in the list
1126 given by transfer_input_files. When a path to an input file or
1127 directory is specified, this specifies the path to the file on the
1128 submit side. A single wild card character (*) may be used in each
1129 file name. The method of encryption utilized will be as agreed upon
1130 in security negotiation; if that negotiation failed, then the file
1131 transfer mechanism must also fail for files to be network encrypted.
1132
1133
1134
1135
1136
1137 encrypt_output_files = < file1,file2,file... >
1138
1139 A comma and/or space separated list of output files that are to be
1140 network encrypted when transferred back with the file transfer mech‐
1141 anism. Specification of files in this manner overrides configuration
1142 that would notuse encryption. The output file(s) must also either be
1143 in the list given by transfer_output_filesor be discovered and to be
1144 transferred back with the file transfer mechanism. When a path to an
1145 output file or directory is specified, this specifies the path to
1146 the file on the execute side. A single wild card character (*) may
1147 be used in each file name. The method of encryption utilized will be
1148 as agreed upon in security negotiation; if that negotiation failed,
1149 then the file transfer mechanism must also fail for files to be net‐
1150 work encrypted.
1151
1152
1153
1154
1155
1156 max_transfer_input_mb = <ClassAd Integer Expression>
1157
1158 This integer expression specifies the maximum allowed total size in
1159 MiB of the input files that are transferred for a job. This expres‐
1160 sion does notapply to grid universe, standard universe, or files
1161 transferred via file transfer plug-ins. The expression may refer to
1162 attributes of the job. The special value -1 indicates no limit. If
1163 not defined, the value set by configuration variable MAX_TRANS‐
1164 FER_INPUT_MBis used. If the observed size of all input files at sub‐
1165 mit time is larger than the limit, the job will be immediately
1166 placed on hold with a HoldReasonCodevalue of 32. If the job passes
1167 this initial test, but the size of the input files increases or the
1168 limit decreases so that the limit is violated, the job will be
1169 placed on hold at the time when the file transfer is attempted.
1170
1171
1172
1173
1174
1175 max_transfer_output_mb = <ClassAd Integer Expression>
1176
1177 This integer expression specifies the maximum allowed total size in
1178 MiB of the output files that are transferred for a job. This expres‐
1179 sion does notapply to grid universe, standard universe, or files
1180 transferred via file transfer plug-ins. The expression may refer to
1181 attributes of the job. The special value -1 indicates no limit. If
1182 not set, the value set by configuration variable MAX_TRANSFER_OUT‐
1183 PUT_MBis used. If the total size of the job's output files to be
1184 transferred is larger than the limit, the job will be placed on hold
1185 with a HoldReasonCodevalue of 33. The output will be transferred up
1186 to the point when the limit is hit, so some files may be fully
1187 transferred, some partially, and some not at all.
1188
1189
1190
1191
1192
1193 output_destination = <destination-URL>
1194
1195 When present, defines a URL that specifies both a plug-in and a des‐
1196 tination for the transfer of the entire output sandbox or a subset
1197 of output files as specified by the submit command transfer_out‐
1198 put_files. The plug-in does the transfer of files, and no files are
1199 sent back to the submit machine. The HTCondor Administrator's manual
1200 has full details.
1201
1202
1203
1204
1205
1206 should_transfer_files = <YES | NO | IF_NEEDED >
1207
1208 The should_transfer_filessetting is used to define if HTCondor
1209 should transfer files to and from the remote machine where the job
1210 runs. The file transfer mechanism is used to run jobs which are not
1211 in the standard universe (and can therefore use remote system calls
1212 for file access) on machines which do not have a shared file system
1213 with the submit machine. should_transfer_filesequal to YESwill cause
1214 HTCondor to always transfer files for the job. NOdisables HTCondor's
1215 file transfer mechanism. IF_NEEDEDwill not transfer files for the
1216 job if it is matched with a resource in the same FileSystemDomainas
1217 the submit machine (and therefore, on a machine with the same shared
1218 file system). If the job is matched with a remote resource in a dif‐
1219 ferent FileSystemDomain, HTCondor will transfer the necessary files.
1220
1221 For more information about this and other settings related to trans‐
1222 ferring files, see the HTCondor User's manual section on the file
1223 transfer mechanism.
1224
1225 Note that should_transfer_filesis not supported for jobs submitted
1226 to the grid universe.
1227
1228
1229
1230
1231
1232 skip_filechecks = <True | False>
1233
1234 When True, file permission checks for the submitted job are dis‐
1235 abled. When False, file permissions are checked; this is the behav‐
1236 ior when this command is not present in the submit description file.
1237 File permissions are checked for read permissions on all input
1238 files, such as those defined by commands inputand trans‐
1239 fer_input_files, and for write permission to output files, such as a
1240 log file defined by logand output files defined with outputor trans‐
1241 fer_output_files.
1242
1243
1244
1245
1246
1247 stream_error = <True | False>
1248
1249 If True, then stderris streamed back to the machine from which the
1250 job was submitted. If False, stderris stored locally and transferred
1251 back when the job completes. This command is ignored if the job
1252 ClassAd attribute TransferErris False. The default value is False.
1253 This command must be used in conjunction with error, otherwise
1254 stderrwill sent to /dev/nullon Unix machines and ignored on Windows
1255 machines.
1256
1257
1258
1259
1260
1261 stream_input = <True | False>
1262
1263 If True, then stdinis streamed from the machine on which the job was
1264 submitted. The default value is False. The command is only relevant
1265 for jobs submitted to the vanilla or java universes, and it is
1266 ignored by the grid universe. This command must be used in conjunc‐
1267 tion with input, otherwise stdinwill be /dev/nullon Unix machines
1268 and ignored on Windows machines.
1269
1270
1271
1272
1273
1274 stream_output = <True | False>
1275
1276 If True, then stdoutis streamed back to the machine from which the
1277 job was submitted. If False, stdoutis stored locally and transferred
1278 back when the job completes. This command is ignored if the job
1279 ClassAd attribute TransferOutis False. The default value is False.
1280 This command must be used in conjunction with output, otherwise std‐
1281 outwill sent to /dev/nullon Unix machines and ignored on Windows
1282 machines.
1283
1284
1285
1286
1287
1288 transfer_executable = <True | False>
1289
1290 This command is applicable to jobs submitted to the grid and vanilla
1291 universes. If transfer_executableis set to False, then HTCondor
1292 looks for the executable on the remote machine, and does not trans‐
1293 fer the executable over. This is useful for an already pre-staged
1294 executable; HTCondor behaves more like rsh. The default value is
1295 True.
1296
1297
1298
1299
1300
1301 transfer_input_files = < file1,file2,file... >
1302
1303 A comma-delimited list of all the files and directories to be trans‐
1304 ferred into the working directory for the job, before the job is
1305 started. By default, the file specified in the executablecommand and
1306 any file specified in the inputcommand (for example, stdin) are
1307 transferred.
1308
1309 When a path to an input file or directory is specified, this speci‐
1310 fies the path to the file on the submit side. The file is placed in
1311 the job's temporary scratch directory on the execute side, and it is
1312 named using the base name of the original path. For example,
1313 /path/to/input_filebecomes input_filein the job's scratch directory.
1314
1315 A directory may be specified by appending the forward slash charac‐
1316 ter (/) as a trailing path separator. This syntax is used for both
1317 Windows and Linux submit hosts. A directory example using a trailing
1318 path separator is input_data/. When a directory is specified with
1319 the trailing path separator, the contents of the directory are
1320 transferred, but the directory itself is not transferred. It is as
1321 if each of the items within the directory were listed in the trans‐
1322 fer list. When there is no trailing path separator, the directory is
1323 transferred, its contents are transferred, and these contents are
1324 placed inside the transferred directory.
1325
1326 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1327 tories is not currently supported.
1328
1329 Symbolic links to files are transferred as the files they point to.
1330 Transfer of symbolic links to directories is not currently sup‐
1331 ported.
1332
1333 For vanilla and vm universe jobs only, a file may be specified by
1334 giving a URL, instead of a file name. The implementation for URL
1335 transfers requires both configuration and available plug-in.
1336
1337
1338
1339
1340
1341 transfer_output_files = < file1,file2,file... >
1342
1343 This command forms an explicit list of output files and directories
1344 to be transferred back from the temporary working directory on the
1345 execute machine to the submit machine. If there are multiple files,
1346 they must be delimited with commas. Setting transfer_output_filesto
1347 the empty string ( "" ) means that no files are to be transferred.
1348
1349 For HTCondor-C jobs and all other non-grid universe jobs, if trans‐
1350 fer_output_filesis not specified, HTCondor will automatically trans‐
1351 fer back all files in the job's temporary working directory which
1352 have been modified or created by the job. Subdirectories are not
1353 scanned for output, so if output from subdirectories is desired, the
1354 output list must be explicitly specified. For grid universe jobs
1355 other than HTCondor-C, desired output files must also be explicitly
1356 listed. Another reason to explicitly list output files is for a job
1357 that creates many files, and the user wants only a subset trans‐
1358 ferred back.
1359
1360 For grid universe jobs other than with grid type condor, to have
1361 files other than standard output and standard error transferred from
1362 the execute machine back to the submit machine, do use transfer_out‐
1363 put_files, listing all files to be transferred. These files are
1364 found on the execute machine in the working directory of the job.
1365
1366 When a path to an output file or directory is specified, it speci‐
1367 fies the path to the file on the execute side. As a destination on
1368 the submit side, the file is placed in the job's initial working
1369 directory, and it is named using the base name of the original path.
1370 For example, path/to/output_filebecomes output_filein the job's ini‐
1371 tial working directory. The name and path of the file that is writ‐
1372 ten on the submit side may be modified by using transfer_out‐
1373 put_remaps. Note that this remap function only works with files but
1374 not with directories.
1375
1376 A directory may be specified using a trailing path separator. An
1377 example of a trailing path separator is the slash character on Unix
1378 platforms; a directory example using a trailing path separator is
1379 input_data/. When a directory is specified with a trailing path sep‐
1380 arator, the contents of the directory are transferred, but the
1381 directory itself is not transferred. It is as if each of the items
1382 within the directory were listed in the transfer list. When there is
1383 no trailing path separator, the directory is transferred, its con‐
1384 tents are transferred, and these contents are placed inside the
1385 transferred directory.
1386
1387 For grid universe jobs other than HTCondor-C, the transfer of direc‐
1388 tories is not currently supported.
1389
1390 Symbolic links to files are transferred as the files they point to.
1391 Transfer of symbolic links to directories is not currently sup‐
1392 ported.
1393
1394
1395
1396
1397
1398 transfer_output_remaps = < “ name = newname ; name2 = newname2
1399 ... ”>
1400
1401 This specifies the name (and optionally path) to use when download‐
1402 ing output files from the completed job. Normally, output files are
1403 transferred back to the initial working directory with the same name
1404 they had in the execution directory. This gives you the option to
1405 save them with a different path or name. If you specify a relative
1406 path, the final path will be relative to the job's initial working
1407 directory.
1408
1409 namedescribes an output file name produced by your job, and new‐
1410 namedescribes the file name it should be downloaded to. Multiple
1411 remaps can be specified by separating each with a semicolon. If you
1412 wish to remap file names that contain equals signs or semicolons,
1413 these special characters may be escaped with a backslash. You cannot
1414 specify directories to be remapped.
1415
1416
1417
1418
1419
1420 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT >
1421
1422
1423
1424 Setting when_to_transfer_outputequal to ON_EXITwill cause HTCondor
1425 to transfer the job's output files back to the submitting machine
1426 only when the job completes (exits on its own).
1427
1428 The ON_EXIT_OR_EVICToption is intended for fault tolerant jobs which
1429 periodically save their own state and can restart where they left
1430 off. In this case, files are spooled to the submit machine any time
1431 the job leaves a remote site, either because it exited on its own,
1432 or was evicted by the HTCondor system for any reason prior to job
1433 completion. The files spooled back are placed in a directory defined
1434 by the value of the SPOOLconfiguration variable. Any output files
1435 transferred back to the submit machine are automatically sent back
1436 out again as input files if the job restarts.
1437
1438
1439
1440
1441
1442 POLICY COMMANDS
1443
1444
1445
1446
1447
1448
1449
1450 max_retries = <integer>
1451
1452 The maximum number of retries allowed for this job (must be non-neg‐
1453 ative). If the job fails (does not exit with the success_exit_code‐
1454 exit code) it will be retried up to max_retriestimes (unless retries
1455 are ceased because of the retry_untilcommand). If max_retriesis not
1456 defined, and either retry_untilor success_exit_codeis, the value of
1457 DEFAULT_JOB_MAX_RETRIESwill be used for the maximum number of
1458 retries.
1459
1460 The combination of the max_retries, retry_until, and suc‐
1461 cess_exit_codecommands causes an appropriate OnExitRemoveexpression
1462 to be automatically generated. If retry command(s) and
1463 on_exit_removeare both defined, the OnExitRemoveexpression will be
1464 generated by OR'ing the expression specified in OnExitRemoveand the
1465 expression generated by the retry commands.
1466
1467
1468
1469
1470
1471 retry_until <Integer | ClassAd Boolean Expression>
1472
1473 An integer value or boolean expression that prevents further retries
1474 from taking place, even if max_retrieshave not been exhausted. If
1475 retry_untilis an integer, the job exiting with that exit code will
1476 cause retries to cease. If retry_untilis a ClassAd expression, the
1477 expression evaluating to Truewill cause retries to cease.
1478
1479
1480
1481
1482
1483 success_exit_code = <integer>
1484
1485 The exit code that is considered successful for this job. Defaults
1486 to 0 if not defined.
1487
1488 Note: non-zero values of success_exit_codeshould generally not be
1489 used for DAG node jobs.At the present time, condor_dagmandoes not
1490 take into account the value of success_exit_code. This means that,
1491 if success_exit_codeis set to a non-zero value, condor_dagmanwill
1492 consider the job failed when it actually succeeds. For single-proc
1493 DAG node jobs, this can be overcome by using a POST script that
1494 takes into account the value of success_exit_code(although this is
1495 not recommended). For multi-proc DAG node jobs, there is currently
1496 no way to overcome this limitation.
1497
1498
1499
1500
1501
1502 hold = <True | False>
1503
1504 If holdis set to True, then the submitted job will be placed into
1505 the Hold state. Jobs in the Hold state will not run until released
1506 by condor_release. Defaults to False.
1507
1508
1509
1510
1511
1512 keep_claim_idle = <integer>
1513
1514 An integer number of seconds that a job requests the condor_scheddto
1515 wait before releasing its claim after the job exits or after the job
1516 is removed.
1517
1518 The process by which the condor_scheddclaims a condor_startdis some‐
1519 what time-consuming. To amortize this cost, the condor_scheddtries
1520 to reuse claims to run subsequent jobs, after a job using a claim is
1521 done. However, it can only do this if there is an idle job in the
1522 queue at the moment the previous job completes. Sometimes, and espe‐
1523 cially for the node jobs when using DAGMan, there is a subsequent
1524 job about to be submitted, but it has not yet arrived in the queue
1525 when the previous job completes. As a result, the condor_sched‐
1526 dreleases the claim, and the next job must wait an entire negotia‐
1527 tion cycle to start. When this submit command is defined with a non-
1528 negative integer, when the job exits, the condor_scheddtries as
1529 usual to reuse the claim. If it cannot, instead of releasing the
1530 claim, the condor_scheddkeeps the claim until either the number of
1531 seconds given as a parameter, or a new job which matches that claim
1532 arrives, whichever comes first. The condor_startdin question will
1533 remain in the Claimed/Idle state, and the original job will be
1534 "charged" (in terms of priority) for the time in this state.
1535
1536
1537
1538
1539
1540 leave_in_queue = <ClassAd Boolean Expression>
1541
1542 When the ClassAd Expression evaluates to True, the job is not
1543 removed from the queue upon completion. This allows the user of a
1544 remotely spooled job to retrieve output files in cases where HTCon‐
1545 dor would have removed them as part of the cleanup associated with
1546 completion. The job will only exit the queue once it has been marked
1547 for removal (via condor_rm, for example) and the leave_in_queueex‐
1548 pression has become False. leave_in_queuedefaults to False.
1549
1550 As an example, if the job is to be removed once the output is
1551 retrieved with condor_transfer_data, then use
1552
1553 leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED)
1554 ||\
1555 (StageOutFinish == 0))
1556
1557
1558
1559
1560
1561 next_job_start_delay = <ClassAd Boolean Expression>
1562
1563 This expression specifies the number of seconds to delay after
1564 starting up this job before the next job is started. The maximum
1565 allowed delay is specified by the HTCondor configuration variable
1566 MAX_NEXT_JOB_START_DELAY, which defaults to 10 minutes. This command
1567 does not apply to scheduleror localuniverse jobs.
1568
1569 This command has been historically used to implement a form of job
1570 start throttling from the job submitter's perspective. It was effec‐
1571 tive for the case of multiple job submission where the transfer of
1572 extremely large input data sets to the execute machine caused
1573 machine performance to suffer. This command is no longer useful, as
1574 throttling should be accomplished through configuration of the con‐
1575 dor_schedddaemon.
1576
1577
1578
1579
1580
1581 on_exit_hold = <ClassAd Boolean Expression>
1582
1583 The ClassAd expression is checked when the job exits, and if True,
1584 places the job into the Hold state. If False(the default value when
1585 not defined), then nothing happens and the on_exit_removeexpression
1586 is checked to determine if that needs to be applied.
1587
1588 For example: Suppose a job is known to run for a minimum of an hour.
1589 If the job exits after less than an hour, the job should be placed
1590 on hold and an e-mail notification sent, instead of being allowed to
1591 leave the queue.
1592
1593
1594
1595 on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1596
1597 This expression places the job on hold if it exits for any reason
1598 before running for an hour. An e-mail will be sent to the user
1599 explaining that the job was placed on hold because this expression
1600 became True.
1601
1602 periodic_*expressions take precedence over on_exit_*expressions, and
1603 *_holdexpressions take precedence over a *_removeexpressions.
1604
1605 Only job ClassAd attributes will be defined for use by this ClassAd
1606 expression. This expression is available for the vanilla, java, par‐
1607 allel, grid, local and scheduler universes. It is additionally
1608 available, when submitted from a Unix machine, for the standard uni‐
1609 verse.
1610
1611
1612
1613
1614
1615 on_exit_hold_reason = <ClassAd String Expression>
1616
1617 When the job is placed on hold due to the on_exit_holdexpression
1618 becoming True, this expression is evaluated to set the value of Hol‐
1619 dReasonin the job ClassAd. If this expression is UNDEFINEDor pro‐
1620 duces an empty or invalid string, a default description is used.
1621
1622
1623
1624
1625
1626 on_exit_hold_subcode = <ClassAd Integer Expression>
1627
1628 When the job is placed on hold due to the on_exit_holdexpression
1629 becoming True, this expression is evaluated to set the value of Hol‐
1630 dReasonSubCodein the job ClassAd. The default subcode is 0. The Hol‐
1631 dReasonCodewill be set to 3, which indicates that the job went on
1632 hold due to a job policy expression.
1633
1634
1635
1636
1637
1638 on_exit_remove = <ClassAd Boolean Expression>
1639
1640 The ClassAd expression is checked when the job exits, and if
1641 True(the default value when undefined), then it allows the job to
1642 leave the queue normally. If False, then the job is placed back into
1643 the Idle state. If the user job runs under the vanilla universe,
1644 then the job restarts from the beginning. If the user job runs under
1645 the standard universe, then it continues from where it left off,
1646 using the last checkpoint.
1647
1648 For example, suppose a job occasionally segfaults, but chances are
1649 that the job will finish successfully if the job is run again with
1650 the same data. The on_exit_removeexpression can cause the job to run
1651 again with the following command. Assume that the signal identifier
1652 for the segmentation fault is 11 on the platform where the job will
1653 be running.
1654
1655 on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11) This
1656 expression lets the job leave the queue if the job was not killed by a
1657 signal or if it was killed by a signal other than 11, representing seg‐
1658 mentation fault in this example. So, if the exited due to signal 11, it
1659 will stay in the job queue. In any other case of the job exiting, the
1660 job will leave the queue as it normally would have done.
1661
1662 As another example, if the job should only leave the queue if it
1663 exited on its own with status 0, this on_exit_removeexpression works
1664 well:
1665
1666
1667
1668 on_exit_remove = (ExitBySignal == False) && (ExitCode == 0) If the
1669 job was killed by a signal or exited with a non-zero exit status,
1670 HTCondor would leave the job in the queue to run again.
1671
1672 periodic_*expressions take precedence over on_exit_*expressions, and
1673 *_holdexpressions take precedence over a *_removeexpressions.
1674
1675 Only job ClassAd attributes will be defined for use by this ClassAd
1676 expression.
1677
1678
1679
1680
1681
1682 periodic_hold = <ClassAd Boolean Expression>
1683
1684 This expression is checked periodically when the job is not in the
1685 Held state. If it becomes True, the job will be placed on hold. If
1686 unspecified, the default value is False.
1687
1688 periodic_*expressions take precedence over on_exit_*expressions, and
1689 *_holdexpressions take precedence over a *_removeexpressions.
1690
1691 Only job ClassAd attributes will be defined for use by this ClassAd
1692 expression. Note that, by default, this expression is only checked
1693 once every 60 seconds. The period of these evaluations can be
1694 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1695 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1696
1697
1698
1699
1700
1701 periodic_hold_reason = <ClassAd String Expression>
1702
1703 When the job is placed on hold due to the periodic_holdexpression
1704 becoming True, this expression is evaluated to set the value of Hol‐
1705 dReasonin the job ClassAd. If this expression is UNDEFINEDor pro‐
1706 duces an empty or invalid string, a default description is used.
1707
1708
1709
1710
1711
1712 periodic_hold_subcode = <ClassAd Integer Expression>
1713
1714 When the job is placed on hold due to the periodic_holdexpression
1715 becoming true, this expression is evaluated to set the value of Hol‐
1716 dReasonSubCodein the job ClassAd. The default subcode is 0. The Hol‐
1717 dReasonCodewill be set to 3, which indicates that the job went on
1718 hold due to a job policy expression.
1719
1720
1721
1722
1723
1724 periodic_release = <ClassAd Boolean Expression>
1725
1726 This expression is checked periodically when the job is in the Held
1727 state. If the expression becomes True, the job will be released.
1728
1729 Only job ClassAd attributes will be defined for use by this ClassAd
1730 expression. Note that, by default, this expression is only checked
1731 once every 60 seconds. The period of these evaluations can be
1732 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1733 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1734
1735
1736
1737
1738
1739 periodic_remove = <ClassAd Boolean Expression>
1740
1741 This expression is checked periodically. If it becomes True, the job
1742 is removed from the queue. If unspecified, the default value is
1743 False.
1744
1745 See the Examples section of this manual page for an example of a
1746 periodic_removeexpression.
1747
1748 periodic_*expressions take precedence over on_exit_*expressions, and
1749 *_holdexpressions take precedence over a *_removeexpressions. So,
1750 the periodic_removeexpression takes precedent over the
1751 on_exit_removeexpression, if the two describe conflicting actions.
1752
1753 Only job ClassAd attributes will be defined for use by this ClassAd
1754 expression. Note that, by default, this expression is only checked
1755 once every 60 seconds. The period of these evaluations can be
1756 adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERI‐
1757 ODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICEconfiguration macros.
1758
1759
1760
1761
1762
1763 COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
1764
1765
1766
1767
1768
1769
1770
1771 allow_startup_script = <True | False>
1772
1773 If True, a standard universe job will execute a script instead of
1774 submitting the job, and the consistency check to see if the exe‐
1775 cutable has been linked using condor_compileis omitted. The exe‐
1776 cutablecommand within the submit description file specifies the name
1777 of the script. The script is used to do preprocessing before the job
1778 is submitted. The shell script ends with an execof the job exe‐
1779 cutable, such that the process id of the executable is the same as
1780 that of the shell script. Here is an example script that gets a copy
1781 of a machine-specific executable before the exec.
1782
1783 #! /bin/sh
1784
1785 # get the host name of the machine
1786 $host=`uname -n`
1787
1788 # grab a standard universe executable designed specifically
1789 # for this host
1790 scp elsewhere@cs.wisc.edu:${host} executable
1791
1792 # The PID MUST stay the same, so exec the new standard universe
1793 process.
1794 exec executable ${1+"$@"} If this command is not present (defined),
1795 then the value defaults to false.
1796
1797
1798
1799
1800
1801 append_files = file1, file2, ...
1802
1803
1804
1805 If your job attempts to access a file mentioned in this list, HTCon‐
1806 dor will force all writes to that file to be appended to the end.
1807 Furthermore, condor_submit will not truncate it. This list uses the
1808 same syntax as compress_files, shown above.
1809
1810 This option may yield some surprising results. If several jobs
1811 attempt to write to the same file, their output may be intermixed.
1812 If a job is evicted from one or more machines during the course of
1813 its lifetime, such an output file might contain several copies of
1814 the results. This option should be only be used when you wish a cer‐
1815 tain file to be treated as a running log instead of a precise
1816 result.
1817
1818 This option only applies to standard-universe jobs.
1819
1820
1821
1822
1823
1824 buffer_files = < “ name = (size,block-size) ; name2 =
1825 (size,block-size) ... ” >
1826
1827
1828
1829
1830
1831 buffer_size = <bytes-in-buffer>
1832
1833
1834
1835
1836
1837 buffer_block_size = <bytes-in-block>
1838
1839 HTCondor keeps a buffer of recently-used data for each file a job
1840 accesses. This buffer is used both to cache commonly-used data and
1841 to consolidate small reads and writes into larger operations that
1842 get better throughput. The default settings should produce reason‐
1843 able results for most programs.
1844
1845 These options only apply to standard-universe jobs.
1846
1847 If needed, you may set the buffer controls individually for each
1848 file using the buffer_files option. For example, to set the buffer
1849 size to 1 MiB and the block size to 256 KiB for the file input.data,
1850 use this command:
1851
1852
1853
1854 buffer_files = "input.data=(1000000,256000)"
1855
1856 Alternatively, you may use these two options to set the default
1857 sizes for all files used by your job:
1858
1859
1860
1861 buffer_size = 1000000
1862 buffer_block_size = 256000
1863
1864 If you do not set these, HTCondor will use the values given by these
1865 two configuration file macros:
1866
1867
1868
1869 DEFAULT_IO_BUFFER_SIZE = 1000000
1870 DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
1871
1872 Finally, if no other settings are present, HTCondor will use a buf‐
1873 fer of 512 KiB and a block size of 32 KiB.
1874
1875
1876
1877
1878
1879 compress_files = file1, file2, ...
1880
1881
1882
1883 If your job attempts to access any of the files mentioned in this
1884 list, HTCondor will automatically compress them (if writing) or
1885 decompress them (if reading). The compress format is the same as
1886 used by GNU gzip.
1887
1888 The files given in this list may be simple file names or complete
1889 paths and may include as a wild card. For example, this list causes
1890 the file /tmp/data.gz, any file named event.gz, and any file ending
1891 in .gzip to be automatically compressed or decompressed as needed:
1892
1893
1894
1895 compress_files = /tmp/data.gz, event.gz, *.gzip
1896
1897 Due to the nature of the compression format, compressed files must
1898 only be accessed sequentially. Random access reading is allowed but
1899 is very slow, while random access writing is simply not possible.
1900 This restriction may be avoided by using both compress_files and
1901 fetch_files at the same time. When this is done, a file is kept in
1902 the decompressed state at the execution machine, but is compressed
1903 for transfer to its original location.
1904
1905 This option only applies to standard universe jobs.
1906
1907
1908
1909
1910
1911 fetch_files = file1, file2, ...
1912
1913 If your job attempts to access a file mentioned in this list, HTCon‐
1914 dor will automatically copy the whole file to the executing machine,
1915 where it can be accessed quickly. When your job closes the file, it
1916 will be copied back to its original location. This list uses the
1917 same syntax as compress_files, shown above.
1918
1919 This option only applies to standard universe jobs.
1920
1921
1922
1923
1924
1925 file_remaps = < “ name = newname ; name2 = newname2 ... ”>
1926
1927
1928
1929 Directs HTCondor to use a new file name in place of an old one.
1930 namedescribes a file name that your job may attempt to open, and
1931 newnamedescribes the file name it should be replaced with. new‐
1932 namemay include an optional leading access specifier, local: or
1933 remote: . If left unspecified, the default access specifier is
1934 remote: . Multiple remaps can be specified by separating each with a
1935 semicolon.
1936
1937 This option only applies to standard universe jobs.
1938
1939 If you wish to remap file names that contain equals signs or semi‐
1940 colons, these special characters may be escaped with a backslash.
1941
1942
1943
1944 Example One:
1945
1946 Suppose that your job reads a file named dataset.1. To instruct
1947 HTCondor to force your job to read other.datasetinstead, add this
1948 to the submit file:
1949
1950 file_remaps = "dataset.1=other.dataset"
1951
1952
1953
1954 Example Two:
1955
1956 Suppose that your run many jobs which all read in the same large
1957 file, called very.big. If this file can be found in the same
1958 place on a local disk in every machine in the pool, (say
1959 /bigdisk/bigfile,) you can instruct HTCondor of this fact by
1960 remapping very.bigto /bigdisk/bigfileand specifying that the file
1961 is to be read locally, which will be much faster than reading
1962 over the network.
1963
1964 file_remaps = "very.big = local:/bigdisk/bigfile"
1965
1966
1967
1968 Example Three:
1969
1970 Several remaps can be applied at once by separating each with a
1971 semicolon.
1972
1973 file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 =
1974 other.dataset"
1975
1976
1977
1978
1979
1980
1981
1982 local_files = file1, file2, ...
1983
1984
1985
1986 If your job attempts to access a file mentioned in this list, HTCon‐
1987 dor will cause it to be read or written at the execution machine.
1988 This is most useful for temporary files not used for input or out‐
1989 put. This list uses the same syntax as compress_files, shown above.
1990
1991
1992
1993 local_files = /tmp/*
1994
1995 This option only applies to standard universe jobs.
1996
1997
1998
1999
2000
2001 want_remote_io = <True | False>
2002
2003 This option controls how a file is opened and manipulated in a stan‐
2004 dard universe job. If this option is true, which is the default,
2005 then the condor_shadowmakes all decisions about how each and every
2006 file should be opened by the executing job. This entails a network
2007 round trip (or more) from the job to the condor_shadowand back again
2008 for every single open()in addition to other needed information about
2009 the file. If set to false, then when the job queries the con‐
2010 dor_shadowfor the first time about how to open a file, the con‐
2011 dor_shadowwill inform the job to automatically perform all of its
2012 file manipulation on the local file system on the execute machine
2013 and any file remapping will be ignored. This means that there mustbe
2014 a shared file system (such as NFS or AFS) between the execute
2015 machine and the submit machine and that ALLpaths that the job could
2016 open on the execute machine must be valid. The ability of the stan‐
2017 dard universe job to checkpoint, possibly to a checkpoint server, is
2018 not affected by this attribute. However, when the job resumes it
2019 will be expecting the same file system conditions that were present
2020 when the job checkpointed.
2021
2022
2023
2024
2025
2026 COMMANDS FOR THE GRID
2027
2028
2029
2030
2031
2032
2033
2034 azure_admin_key = <pathname>
2035
2036 For grid type azurejobs, specifies the path and file name of a file
2037 that contains an SSH public key. This key can be used to log into
2038 the administrator account of the instance via SSH.
2039
2040
2041
2042
2043
2044 azure_admin_username = <account name>
2045
2046 For grid type azurejobs, specifies the name of an administrator
2047 account to be created in the instance. This account can be logged
2048 into via SSH.
2049
2050
2051
2052
2053
2054 azure_auth_file = <pathname>
2055
2056 For grid type azurejobs, specifies a path and file name of the
2057 authorization file that grants permission for HTCondor to use the
2058 Azure account. If it's not defined, then HTCondor will attempt to
2059 use the default credentials of the Azure CLI tools.
2060
2061
2062
2063
2064
2065 azure_image = <image id>
2066
2067 For grid type azurejobs, identifies the disk image to be used for
2068 the boot disk of the instance. This image must already be registered
2069 within Azure.
2070
2071
2072
2073
2074
2075 azure_location = <image id>
2076
2077 For grid type azurejobs, identifies the location within Azure where
2078 the instance should be run. As an example, one current location is
2079 centralus.
2080
2081
2082
2083
2084
2085 azure_size = <machine type>
2086
2087 For grid type azurejobs, the hardware configuration that the virtual
2088 machine instance is to run on.
2089
2090
2091
2092
2093
2094 batch_queue = <queuename>
2095
2096 Used for pbs, lsf, and sgegrid universe jobs. Specifies the name of
2097 the PBS/LSF/SGE job queue into which the job should be submitted. If
2098 not specified, the default queue is used.
2099
2100
2101
2102
2103
2104 boinc_authenticator_file = <pathname>
2105
2106 For grid type boincjobs, specifies a path and file name of the
2107 authorization file that grants permission for HTCondor to use the
2108 BOINC service. There is no default value when not specified.
2109
2110
2111
2112
2113
2114 cream_attributes = <name=value;...;name=value>
2115
2116 Provides a list of attribute/value pairs to be set in a CREAM job
2117 description of a grid universe job destined for the CREAM grid sys‐
2118 tem. The pairs are separated by semicolons, and written in New Clas‐
2119 sAd syntax.
2120
2121
2122
2123
2124
2125 delegate_job_GSI_credentials_lifetime = <seconds>
2126
2127 Specifies the maximum number of seconds for which delegated proxies
2128 should be valid. The default behavior when this command is not spec‐
2129 ified is determined by the configuration variable DELE‐
2130 GATE_JOB_GSI_CREDENTIALS_LIFETIME, which defaults to one day. A
2131 value of 0 indicates that the delegated proxy should be valid for as
2132 long as allowed by the credential used to create the proxy. This
2133 setting currently only applies to proxies delegated for non-grid
2134 jobs and for HTCondor-C jobs. It does not currently apply to globus
2135 grid jobs, which always behave as though this setting were 0. This
2136 variable has no effect if the configuration variable DELE‐
2137 GATE_JOB_GSI_CREDENTIALSis False, because in that case the job proxy
2138 is copied rather than delegated.
2139
2140
2141
2142
2143
2144 ec2_access_key_id = <pathname>
2145
2146 For grid type ec2jobs, identifies the file containing the access
2147 key.
2148
2149
2150
2151
2152
2153 ec2_ami_id = <EC2 xMI ID>
2154
2155 For grid type ec2jobs, identifies the machine image. Services com‐
2156 patible with the EC2 Query API may refer to these with abbreviations
2157 other than AMI, for example EMIis valid for Eucalyptus.
2158
2159
2160
2161
2162
2163 ec2_availability_zone = <zone name>
2164
2165 For grid type ec2jobs, specifies the Availability Zone that the
2166 instance should be run in. This command is optional, unless
2167 ec2_ebs_volumesis set. As an example, one current zone is us-
2168 east-1b.
2169
2170
2171
2172
2173
2174 ec2_block_device_mapping = <block-device>:<kernel-device>,<block-
2175 device>:<kernel-device>, ...
2176
2177 For grid type ec2jobs, specifies the block device to kernel device
2178 mapping. This command is optional.
2179
2180
2181
2182
2183
2184 ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device name>,...
2185
2186 For grid type ec2jobs, optionally specifies a list of Elastic Block
2187 Store (EBS) volumes to be made available to the instance and the
2188 device names they should have in the instance.
2189
2190
2191
2192
2193
2194 ec2_elastic_ip = <elastic IP address>
2195
2196 For grid type ec2jobs, and optional specification of an Elastic IP
2197 address that should be assigned to this instance.
2198
2199
2200
2201
2202
2203 ec2_iam_profile_arn = <IAM profile ARN>
2204
2205 For grid type ec2jobs, an Amazon Resource Name (ARN) identifying
2206 which Identity and Access Management (IAM) (instance) profile to as‐
2207 sociate with the instance.
2208
2209
2210
2211
2212
2213 ec2_iam_profile_name= <IAM profile name>
2214
2215 For grid type ec2jobs, a name identifying which Identity and Access
2216 Management (IAM) (instance) profile to associate with the instance.
2217
2218
2219
2220
2221
2222 ec2_instance_type = <instance type>
2223
2224 For grid type ec2jobs, identifies the instance type. Different ser‐
2225 vices may offer different instance types, so no default value is
2226 set.
2227
2228
2229
2230
2231
2232 ec2_keypair = <ssh key-pair name>
2233
2234 For grid type ec2jobs, specifies the name of an SSH key-pair that is
2235 already registered with the EC2 service. The associated private key
2236 can be used to sshinto the virtual machine once it is running.
2237
2238
2239
2240
2241
2242 ec2_keypair_file = <pathname>
2243
2244 For grid type ec2jobs, specifies the complete path and file name of
2245 a file into which HTCondor will write an SSH key for use with ec2
2246 jobs. The key can be used to sshinto the virtual machine once it is
2247 running. If ec2_keypairis specified for a job, ec2_keypair_fileis
2248 ignored.
2249
2250
2251
2252
2253
2254 ec2_parameter_names = ParameterName1, ParameterName2, ...
2255
2256 For grid type ec2jobs, a space or comma separated list of the names
2257 of additional parameters to pass when instantiating an instance.
2258
2259
2260
2261
2262
2263 ec2_parameter_<name> = <value>
2264
2265 For grid type ec2jobs, specifies the value for the correspondingly
2266 named (instance instantiation) parameter. <name>is the parameter
2267 name specified in the submit command ec2_parameter_names, but with
2268 any periods replaced by underscores.
2269
2270
2271
2272
2273
2274 ec2_secret_access_key = <pathname>
2275
2276 For grid type ec2jobs, specifies the path and file name containing
2277 the secret access key.
2278
2279
2280
2281
2282
2283 ec2_security_groups = group1, group2, ...
2284
2285 For grid type ec2jobs, defines the list of EC2 security groups which
2286 should be associated with the job.
2287
2288
2289
2290
2291
2292 ec2_security_ids = id1, id2, ...
2293
2294 For grid type ec2jobs, defines the list of EC2 security group IDs
2295 which should be associated with the job.
2296
2297
2298
2299
2300
2301 ec2_spot_price = <bid>
2302
2303 For grid type ec2jobs, specifies the spot instance bid, which is the
2304 most that the job submitter is willing to pay per hour to run this
2305 job.
2306
2307
2308
2309
2310
2311 ec2_tag_names = <name0,name1,name...>
2312
2313 For grid type ec2jobs, specifies the case of tag names that will be
2314 associated with the running instance. This is only necessary if a
2315 tag name case matters. By default the list will be automatically
2316 generated.
2317
2318
2319
2320
2321
2322 ec2_tag_<name> = <value>
2323
2324 For grid type ec2jobs, specifies a tag to be associated with the
2325 running instance. The tag name will be lower-cased, use
2326 ec2_tag_namesto change the case.
2327
2328
2329
2330
2331
2332 WantNameTag = <True | False>
2333
2334 For grid type ec2jobs, a job may request that its 'name' tag be
2335 (not) set by HTCondor. If the job does not otherwise specify any
2336 tags, not setting its name tag will eliminate a call by the EC2
2337 GAHP, improving performance.
2338
2339
2340
2341
2342
2343 ec2_user_data = <data>
2344
2345 For grid type ec2jobs, provides a block of data that can be accessed
2346 by the virtual machine. If both ec2_user_dataand
2347 ec2_user_data_fileare specified for a job, the two blocks of data
2348 are concatenated, with the data from this ec2_user_datasubmit com‐
2349 mand occurring first.
2350
2351
2352
2353
2354
2355 ec2_user_data_file = <pathname>
2356
2357 For grid type ec2jobs, specifies a path and file name whose contents
2358 can be accessed by the virtual machine. If both ec2_user_dataand
2359 ec2_user_data_fileare specified for a job, the two blocks of data
2360 are concatenated, with the data from that ec2_user_datasubmit com‐
2361 mand occurring first.
2362
2363
2364
2365
2366
2367 ec2_vpc_ip = <a.b.c.d>
2368
2369 For grid type ec2jobs, that are part of a Virtual Private Cloud
2370 (VPC), an optional specification of the IP address that this
2371 instance should have within the VPC.
2372
2373
2374
2375
2376
2377 ec2_vpc_subnet = <subnet specification string>
2378
2379 For grid type ec2jobs, an optional specification of the Virtual Pri‐
2380 vate Cloud (VPC) that this instance should be a part of.
2381
2382
2383
2384
2385
2386 gce_account = <account name>
2387
2388 For grid type gcejobs, specifies the Google cloud services account
2389 to use. If this submit command isn't specified, then a random
2390 account from the authorization file given by gce_auth_filewill be
2391 used.
2392
2393
2394
2395
2396
2397 gce_auth_file = <pathname>
2398
2399 For grid type gcejobs, specifies a path and file name of the autho‐
2400 rization file that grants permission for HTCondor to use the Google
2401 account. If this command is not specified, then the default file of
2402 the Google command-line tools will be used.
2403
2404
2405
2406
2407
2408 gce_image = <image id>
2409
2410 For grid type gcejobs, the identifier of the virtual machine image
2411 representing the HTCondor job to be run. This virtual machine image
2412 must already be register with GCE and reside in Google's Cloud Stor‐
2413 age service.
2414
2415
2416
2417
2418
2419 gce_json_file = <pathname>
2420
2421 For grid type gcejobs, specifies the path and file name of a file
2422 that contains JSON elements that should be added to the instance
2423 description submitted to the GCE service.
2424
2425
2426
2427
2428
2429 gce_machine_type = <machine type>
2430
2431 For grid type gcejobs, the long form of the URL that describes the
2432 machine configuration that the virtual machine instance is to run
2433 on.
2434
2435
2436
2437
2438
2439 gce_metadata = <name=value,...,name=value>
2440
2441 For grid type gcejobs, a comma separated list of name and value
2442 pairs that define metadata for a virtual machine instance that is an
2443 HTCondor job.
2444
2445
2446
2447
2448
2449 gce_metadata_file = <pathname>
2450
2451 For grid type gcejobs, specifies a path and file name of the file
2452 that contains metadata for a virtual machine instance that is an
2453 HTCondor job. Within the file, each name and value pair is on its
2454 own line; so, the pairs are separated by the newline character.
2455
2456
2457
2458
2459
2460 gce_preemptible = <True | False>
2461
2462 For grid type gcejobs, specifies whether the virtual machine
2463 instance should be preemptible. The default is for the instance to
2464 not be preemptible.
2465
2466
2467
2468
2469
2470 globus_rematch = <ClassAd Boolean Expression>
2471
2472 This expression is evaluated by the condor_gridmanagerwhenever:
2473
2474 1. the globus_resubmitexpression evaluates to True
2475
2476 2. the condor_gridmanagerdecides it needs to retry a submission
2477 (as when a previous submission failed to commit) If
2478 globus_rematchevaluates to True, then beforethe job is submitted
2479 again to globus, the condor_gridmanagerwill request that the con‐
2480 dor_schedddaemon renegotiate with the matchmaker (the con‐
2481 dor_negotiator). The result is this job will be matched again.
2482
2483
2484
2485
2486
2487 globus_resubmit = <ClassAd Boolean Expression>
2488
2489 The expression is evaluated by the condor_gridmanagereach time the
2490 condor_gridmanagergets a job ad to manage. Therefore, the expression
2491 is evaluated:
2492
2493 1. when a grid universe job is first submitted to HTCondor-G
2494
2495 2. when a grid universe job is released from the hold state
2496
2497 3. when HTCondor-G is restarted (specifically, whenever the con‐
2498 dor_gridmanageris restarted) If the expression evaluates to True,
2499 then any previous submission to the grid universe will be forgot‐
2500 ten and this job will be submitted again as a fresh submission to
2501 the grid universe. This may be useful if there is a desire to
2502 give up on a previous submission and try again. Note that this
2503 may result in the same job running more than once. Do not treat
2504 this operation lightly.
2505
2506
2507
2508
2509
2510 globus_rsl = <RSL-string>
2511
2512 Used to provide any additional Globus RSL string attributes which
2513 are not covered by other submit description file commands or job
2514 attributes. Used for griduniversejobs, where the grid resource has a
2515 grid-type-stringof gt2.
2516
2517
2518
2519
2520
2521 grid_resource = <grid-type-string> <grid-specific-parameter-list>
2522
2523 For each grid-type-stringvalue, there are further type-specific val‐
2524 ues that must specified. This submit description file command allows
2525 each to be given in a space-separated list. Allowable grid-type-
2526 stringvalues are batch, condor, cream, ec2, gt2, gt5, lsf, nor‐
2527 dugrid, pbs, sge, and unicore. The HTCondor manual chapter on Grid
2528 Computing details the variety of grid types.
2529
2530 For a grid-type-stringof batch, the single parameter is the name of
2531 the local batch system, and will be one of pbs, lsf, or sge.
2532
2533 For a grid-type-stringof condor, the first parameter is the name of
2534 the remote condor_schedddaemon. The second parameter is the name of
2535 the pool to which the remote condor_schedddaemon belongs.
2536
2537 For a grid-type-stringof cream, there are three parameters. The
2538 first parameter is the web services address of the CREAM server. The
2539 second parameter is the name of the batch system that sits behind
2540 the CREAM server. The third parameter identifies a site-specific
2541 queue within the batch system.
2542
2543 For a grid-type-stringof ec2, one additional parameter specifies the
2544 EC2 URL.
2545
2546 For a grid-type-stringof gt2, the single parameter is the name of
2547 the pre-WS GRAM resource to be used.
2548
2549 For a grid-type-stringof gt5, the single parameter is the name of
2550 the pre-WS GRAM resource to be used, which is the same as for the
2551 grid-type-stringof gt2.
2552
2553 For a grid-type-stringof lsf, no additional parameters are used.
2554
2555 For a grid-type-stringof nordugrid, the single parameter is the name
2556 of the NorduGrid resource to be used.
2557
2558 For a grid-type-stringof pbs, no additional parameters are used.
2559
2560 For a grid-type-stringof sge, no additional parameters are used.
2561
2562 For a grid-type-stringof unicore, the first parameter is the name of
2563 the Unicore Usite to be used. The second parameter is the name of
2564 the Unicore Vsite to be used.
2565
2566
2567
2568
2569
2570 keystore_alias = <name>
2571
2572 A string to locate the certificate in a Java keystore file, as used
2573 for a unicorejob.
2574
2575
2576
2577
2578
2579 keystore_file = <pathname>
2580
2581 The complete path and file name of the Java keystore file containing
2582 the certificate to be used for a unicorejob.
2583
2584
2585
2586
2587
2588 keystore_passphrase_file = <pathname>
2589
2590 The complete path and file name to the file containing the
2591 passphrase protecting a Java keystore file containing the certifi‐
2592 cate. Relevant for a unicorejob.
2593
2594
2595
2596
2597
2598 MyProxyCredentialName = <symbolic name>
2599
2600 The symbolic name that identifies a credential to the MyProxyserver.
2601 This symbolic name is set as the credential is initially stored on
2602 the server (using myproxy-init).
2603
2604
2605
2606
2607
2608 MyProxyHost = <host>:<port>
2609
2610 The Internet address of the host that is the MyProxyserver. The
2611 hostmay be specified by either a host name (as in head.example.com)
2612 or an IP address (of the form 123.456.7.8). The portnumber is an
2613 integer.
2614
2615
2616
2617
2618
2619 MyProxyNewProxyLifetime = <number-of-minutes>
2620
2621 The new lifetime (in minutes) of the proxy after it is refreshed.
2622
2623
2624
2625
2626
2627 MyProxyPassword = <password>
2628
2629 The password needed to refresh a credential on the MyProxyserver.
2630 This password is set when the user initially stores credentials on
2631 the server (using myproxy-init). As an alternative to using MyProxy‐
2632 Passwordin the submit description file, the password may be speci‐
2633 fied as a command line argument to condor_submitwith the -passwor‐
2634 dargument.
2635
2636
2637
2638
2639
2640 MyProxyRefreshThreshold = <number-of-seconds>
2641
2642 The time (in seconds) before the expiration of a proxy that the
2643 proxy should be refreshed. For example, if MyProxyRefreshThresholdis
2644 set to the value 600, the proxy will be refreshed 10 minutes before
2645 it expires.
2646
2647
2648
2649
2650
2651 MyProxyServerDN = <credential subject>
2652
2653 A string that specifies the expected Distinguished Name (credential
2654 subject, abbreviated DN) of the MyProxyserver. It must be specified
2655 when the MyProxyserver DN does not follow the conventional naming
2656 scheme of a host credential. This occurs, for example, when the
2657 MyProxyserver DN begins with a user credential.
2658
2659
2660
2661
2662
2663 nordugrid_rsl = <RSL-string>
2664
2665 Used to provide any additional RSL string attributes which are not
2666 covered by regular submit description file parameters. Used when the
2667 universeis grid, and the type of grid system is nordugrid.
2668
2669
2670
2671
2672
2673 transfer_error = <True | False>
2674
2675 For jobs submitted to the grid universe only. If True, then the
2676 error output (from stderr) from the job is transferred from the
2677 remote machine back to the submit machine. The name of the file
2678 after transfer is given by the errorcommand. If False, no transfer
2679 takes place (from the remote machine to submit machine), and the
2680 name of the file is given by the errorcommand. The default value is
2681 True.
2682
2683
2684
2685
2686
2687 transfer_input = <True | False>
2688
2689 For jobs submitted to the grid universe only. If True, then the job
2690 input (stdin) is transferred from the machine where the job was sub‐
2691 mitted to the remote machine. The name of the file that is trans‐
2692 ferred is given by the inputcommand. If False, then the job's input
2693 is taken from a pre-staged file on the remote machine, and the name
2694 of the file is given by the inputcommand. The default value is True.
2695
2696 For transferring files other than stdin, see transfer_input_files.
2697
2698
2699
2700
2701
2702 transfer_output = <True | False>
2703
2704 For jobs submitted to the grid universe only. If True, then the out‐
2705 put (from stdout) from the job is transferred from the remote
2706 machine back to the submit machine. The name of the file after
2707 transfer is given by the outputcommand. If False, no transfer takes
2708 place (from the remote machine to submit machine), and the name of
2709 the file is given by the outputcommand. The default value is True.
2710
2711 For transferring files other than stdout, see transfer_output_files.
2712
2713
2714
2715
2716
2717 use_x509userproxy = <True | False>
2718
2719 Set this command to Trueto indicate that the job requires an X.509
2720 user proxy. If x509userproxyis set, then that file is used for the
2721 proxy. Otherwise, the proxy is looked for in the standard locations.
2722 If x509userproxyis set or if the job is a grid universe job of grid
2723 type gt2, gt5, cream, or nordugrid, then the value of use_x509user‐
2724 proxyis forced to True. Defaults to False.
2725
2726
2727
2728
2729
2730 x509userproxy = <full-pathname>
2731
2732 Used to override the default path name for X.509 user certificates.
2733 The default location for X.509 proxies is the /tmpdirectory, which
2734 is generally a local file system. Setting this value would allow
2735 HTCondor to access the proxy in a shared file system (for example,
2736 AFS). HTCondor will use the proxy specified in the submit descrip‐
2737 tion file first. If nothing is specified in the submit description
2738 file, it will use the environment variable X509_USER_PROXY. If that
2739 variable is not present, it will search in the default location.
2740 Note that proxies are only valid for a limited time. Condor_submit
2741 will not submit a job with an expired proxy, it will return an
2742 error. Also, if the configuration parameter CRED_MIN_TIME_LEFT is
2743 set to some number of seconds, and if the proxy will expire before
2744 that many seconds, condor_submit will also refuse to submit the job.
2745 That is, if CRED_MIN_TIME_LEFT is set to 60, condor_submit will
2746 refuse to submit a job whose proxy will expire 60 seconds from the
2747 time of submission.
2748
2749 x509userproxyis relevant when the universeis vanilla, or when the
2750 universeis gridand the type of grid system is one of gt2, gt5, con‐
2751 dor, cream, or nordugrid. Defining a value causes the proxy to be
2752 delegated to the execute machine. Further, VOMS attributes defined
2753 in the proxy will appear in the job ClassAd.
2754
2755
2756
2757
2758
2759 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2760
2761
2762
2763
2764
2765
2766
2767 hold_kill_sig = <signal-number>
2768
2769 For the scheduler universe only, signal-numberis the signal deliv‐
2770 ered to the job when the job is put on hold with condor_hold. sig‐
2771 nal-numbermay be either the platform-specific name or value of the
2772 signal. If this command is not present, the value of kill_sigis
2773 used.
2774
2775
2776
2777
2778
2779 jar_files = <file_list>
2780
2781 Specifies a list of additional JAR files to include when using the
2782 Java universe. JAR files will be transferred along with the exe‐
2783 cutable and automatically added to the classpath.
2784
2785
2786
2787
2788
2789 java_vm_args = <argument_list>
2790
2791 Specifies a list of additional arguments to the Java VM itself, When
2792 HTCondor runs the Java program, these are the arguments that go
2793 before the class name. This can be used to set VM-specific arguments
2794 like stack size, garbage-collector arguments and initial property
2795 values.
2796
2797
2798
2799
2800
2801 machine_count = <max>
2802
2803 For the parallel universe, a single value (max) is required. It is
2804 neither a maximum or minimum, but the number of machines to be dedi‐
2805 cated toward running the job.
2806
2807
2808
2809
2810
2811 remove_kill_sig = <signal-number>
2812
2813 For the scheduler universe only, signal-numberis the signal deliv‐
2814 ered to the job when the job is removed with condor_rm. signal-num‐
2815 bermay be either the platform-specific name or value of the signal.
2816 This example shows it both ways for a Linux signal:
2817
2818 remove_kill_sig = SIGUSR1
2819 remove_kill_sig = 10
2820
2821 If this command is not present, the value of kill_sigis used.
2822
2823
2824
2825
2826
2827 COMMANDS FOR THE VM UNIVERSE
2828
2829
2830
2831
2832
2833
2834
2835 vm_disk = file1:device1:permission1, file2:device2:permission2:format2,
2836 ...
2837
2838 A list of comma separated disk files. Each disk file is specified by
2839 4 colon separated fields. The first field is the path and file name
2840 of the disk file. The second field specifies the device. The third
2841 field specifies permissions, and the optional fourth field specifies
2842 the image format. If a disk file will be transferred by HTCondor,
2843 then the first field should just be the simple file name (no path
2844 information).
2845
2846 An example that specifies two disk files:
2847
2848 vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2849
2850
2851
2852
2853
2854 vm_checkpoint = <True | False>
2855
2856 A boolean value specifying whether or not to take checkpoints. If
2857 not specified, the default value is False. In the current implemen‐
2858 tation, setting both vm_checkpointand vm_networkingto Truedoes not
2859 yet work in all cases. Networking cannot be used if a vm universe
2860 job uses a checkpoint in order to continue execution after migration
2861 to another machine.
2862
2863
2864
2865
2866
2867 vm_macaddr = <MACAddr>
2868
2869 Defines that MAC address that the virtual machine's network inter‐
2870 face should have, in the standard format of six groups of two hexa‐
2871 decimal digits separated by colons.
2872
2873
2874
2875
2876
2877 vm_memory = <MBytes-of-memory>
2878
2879 The amount of memory in MBytes that a vm universe job requires.
2880
2881
2882
2883
2884
2885 vm_networking = <True | False>
2886
2887 Specifies whether to use networking or not. In the current implemen‐
2888 tation, setting both vm_checkpointand vm_networkingto Truedoes not
2889 yet work in all cases. Networking cannot be used if a vm universe
2890 job uses a checkpoint in order to continue execution after migration
2891 to another machine.
2892
2893
2894
2895
2896
2897 vm_networking_type = <nat | bridge >
2898
2899 When vm_networkingis True, this definition augments the job's
2900 requirements to match only machines with the specified networking.
2901 If not specified, then either networking type matches.
2902
2903
2904
2905
2906
2907 vm_no_output_vm = <True | False>
2908
2909 When True, prevents HTCondor from transferring output files back to
2910 the machine from which the vm universe job was submitted. If not
2911 specified, the default value is False.
2912
2913
2914
2915
2916
2917 vm_type = <vmware | xen | kvm>
2918
2919 Specifies the underlying virtual machine software that this job
2920 expects.
2921
2922
2923
2924
2925
2926 vmware_dir = <pathname>
2927
2928 The complete path and name of the directory where VMware-specific
2929 files and applications such as the VMDK (Virtual Machine Disk For‐
2930 mat) and VMX (Virtual Machine Configuration) reside. This command is
2931 optional; when not specified, all relevant VMware image files are to
2932 be listed using transfer_input_files.
2933
2934
2935
2936
2937
2938 vmware_should_transfer_files = <True | False>
2939
2940 Specifies whether HTCondor will transfer VMware-specific files
2941 located as specified by vmware_dirto the execute machine (True) or
2942 rely on access through a shared file system (False). Omission of
2943 this required command (for VMware vm universe jobs) results in an
2944 error message from condor_submit, and the job will not be submitted.
2945
2946
2947
2948
2949
2950 vmware_snapshot_disk = <True | False>
2951
2952 When True, causes HTCondor to utilize a VMware snapshot disk for new
2953 or modified files. If not specified, the default value is True.
2954
2955
2956
2957
2958
2959 xen_initrd = <image-file>
2960
2961 When xen_kernelgives a file name for the kernel image to use, this
2962 optional command may specify a path to a ramdisk (initrd) image
2963 file. If the image file will be transferred by HTCondor, then the
2964 value should just be the simple file name (no path information).
2965
2966
2967
2968
2969
2970 xen_kernel = <included | path-to-kernel>
2971
2972 A value of includedspecifies that the kernel is included in the disk
2973 file. If not one of these values, then the value is a path and file
2974 name of the kernel to be used. If a kernel file will be transferred
2975 by HTCondor, then the value should just be the simple file name (no
2976 path information).
2977
2978
2979
2980
2981
2982 xen_kernel_params = <string>
2983
2984 A string that is appended to the Xen kernel command line.
2985
2986
2987
2988
2989
2990 xen_root = <string>
2991
2992 A string that is appended to the Xen kernel command line to specify
2993 the root device. This string is required when xen_kernelgives a path
2994 to a kernel. Omission for this required case results in an error
2995 message during submission.
2996
2997
2998
2999
3000
3001 COMMANDS FOR THE DOCKER UNIVERSE
3002
3003
3004
3005
3006
3007
3008
3009 docker_image = < image-name >
3010
3011 Defines the name of the Docker image that is the basis for the
3012 docker container.
3013
3014
3015
3016
3017
3018 ADVANCED COMMANDS
3019
3020
3021
3022
3023
3024
3025
3026 accounting_group = <accounting-group-name>
3027
3028 Causes jobs to negotiate under the given accounting group. This
3029 value is advertised in the job ClassAd as AcctGroup. The HTCondor
3030 Administrator's manual contains more information about accounting
3031 groups.
3032
3033
3034
3035
3036
3037 accounting_group_user = <accounting-group-user-name>
3038
3039 Sets the user name associated with the accounting group name for
3040 resource usage accounting purposes. If not set, defaults to the
3041 value of the job ClassAd attribute Owner. This value is advertised
3042 in the job ClassAd as AcctGroupUser. If an accounting group has not
3043 been set with the accounting_groupcommand, this command is ignored.
3044
3045
3046
3047
3048
3049 concurrency_limits = <string-list>
3050
3051 A list of resources that this job needs. The resources are presumed
3052 to have concurrency limits placed upon them, thereby limiting the
3053 number of concurrent jobs in execution which need the named
3054 resource. Commas and space characters delimit the items in the list.
3055 Each item in the list is a string that identifies the limit, or it
3056 is a ClassAd expression that evaluates to a string, and it is evalu‐
3057 ated in the context of machine ClassAd being considered as a match.
3058 Each item in the list also may specify a numerical value identifying
3059 the integer number of resources required for the job. The syntax
3060 follows the resource name by a colon character ( : ) and the numeri‐
3061 cal value. Details on concurrency limits are in the HTCondor Admin‐
3062 istrator's manual.
3063
3064
3065
3066
3067
3068 concurrency_limits_expr = <ClassAd String Expression>
3069
3070 A ClassAd expression that represents the list of resources that this
3071 job needs after evaluation. The ClassAd expression may specify
3072 machine ClassAd attributes that are evaluated against a matched
3073 machine. After evaluation, the list sets concurrency_limits.
3074
3075
3076
3077
3078
3079 copy_to_spool = <True | False>
3080
3081 If copy_to_spoolis True, then condor_submitcopies the executable to
3082 the local spool directory before running it on a remote host. As
3083 copying can be quite time consuming and unnecessary, the default
3084 value is Falsefor all job universes other than the standard uni‐
3085 verse. When False, condor_submitdoes not copy the executable to a
3086 local spool directory. The default is Truein standard universe,
3087 because resuming execution from a checkpoint can only be guaranteed
3088 to work using precisely the same executable that created the check‐
3089 point.
3090
3091
3092
3093
3094
3095 coresize = <size>
3096
3097 Should the user's program abort and produce a core file, coresize‐
3098 specifies the maximum size in bytes of the core file which the user
3099 wishes to keep. If coresizeis not specified in the command file, the
3100 system's user resource limit coredumpsizeis used (note that core‐
3101 dumpsizeis not an HTCondor parameter – it is an operating sys‐
3102 tem parameter that can be viewed with the limitor ulimitcommand on
3103 Unix and in the Registry on Windows). A value of -1 results in no
3104 limits being applied to the core file size. If HTCondor is running
3105 as root, a coresizesetting greater than the system coredumpsizelimit
3106 will override the system setting; if HTCondor is notrunning as root,
3107 the system coredumpsizelimit will override coresize.
3108
3109
3110
3111
3112
3113 cron_day_of_month = <Cron-evaluated Day>
3114
3115 The set of days of the month for which a deferral time applies. The
3116 HTCondor User's manual section on Time Scheduling for Job Execution
3117 has further details.
3118
3119
3120
3121
3122
3123 cron_day_of_week = <Cron-evaluated Day>
3124
3125 The set of days of the week for which a deferral time applies. The
3126 HTCondor User's manual section on Time Scheduling for Job Execution
3127 has further details.
3128
3129
3130
3131
3132
3133 cron_hour = <Cron-evaluated Hour>
3134
3135 The set of hours of the day for which a deferral time applies. The
3136 HTCondor User's manual section on Time Scheduling for Job Execution
3137 has further details.
3138
3139
3140
3141
3142
3143 cron_minute = <Cron-evaluated Minute>
3144
3145 The set of minutes within an hour for which a deferral time applies.
3146 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3147 tion has further details.
3148
3149
3150
3151
3152
3153 cron_month = <Cron-evaluated Month>
3154
3155 The set of months within a year for which a deferral time applies.
3156 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3157 tion has further details.
3158
3159
3160
3161
3162
3163 cron_prep_time = <ClassAd Integer Expression>
3164
3165 Analogous to deferral_prep_time. The number of seconds prior to a
3166 job's deferral time that the job may be matched and sent to an exe‐
3167 cution machine.
3168
3169
3170
3171
3172
3173 cron_window = <ClassAd Integer Expression>
3174
3175 Analogous to the submit command deferral_window. It allows cron jobs
3176 that miss their deferral time to begin execution.
3177
3178 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3179 tion has further details.
3180
3181
3182
3183
3184
3185 dagman_log = <pathname>
3186
3187 DAGMan inserts this command to specify an event log that it watches
3188 to maintain the state of the DAG. If the logcommand is not specified
3189 in the submit file, DAGMan uses the logcommand to specify the event
3190 log.
3191
3192
3193
3194
3195
3196 deferral_prep_time = <ClassAd Integer Expression>
3197
3198 The number of seconds prior to a job's deferral time that the job
3199 may be matched and sent to an execution machine.
3200
3201 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3202 tion has further details.
3203
3204
3205
3206
3207
3208 deferral_time = <ClassAd Integer Expression>
3209
3210 Allows a job to specify the time at which its execution is to begin,
3211 instead of beginning execution as soon as it arrives at the execu‐
3212 tion machine. The deferral time is an expression that evaluates to a
3213 Unix Epoch timestamp (the number of seconds elapsed since 00:00:00
3214 on January 1, 1970, Coordinated Universal Time). Deferral time is
3215 evaluated with respect to the execution machine. This option delays
3216 the start of execution, but not the matching and claiming of a
3217 machine for the job. If the job is not available and ready to begin
3218 execution at the deferral time, it has missed its deferral time. A
3219 job that misses its deferral time will be put on hold in the queue.
3220
3221 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3222 tion has further details.
3223
3224 Due to implementation details, a deferral time may not be used for
3225 scheduler universe jobs.
3226
3227
3228
3229
3230
3231 deferral_window = <ClassAd Integer Expression>
3232
3233 The deferral window is used in conjunction with the defer‐
3234 ral_timecommand to allow jobs that miss their deferral time to begin
3235 execution.
3236
3237 The HTCondor User's manual section on Time Scheduling for Job Execu‐
3238 tion has further details.
3239
3240
3241
3242
3243
3244 description = <string>
3245
3246 A string that sets the value of the job ClassAd attribute JobDe‐
3247 scription. When set, tools which display the executable such as con‐
3248 dor_qwill instead use this string.
3249
3250
3251
3252
3253
3254 email_attributes = <list-of-job-ad-attributes>
3255
3256 A comma-separated list of attributes from the job ClassAd. These
3257 attributes and their values will be included in the e-mail notifica‐
3258 tion of job completion.
3259
3260
3261
3262
3263
3264 image_size = <size>
3265
3266 Advice to HTCondor specifying the maximum virtual image size to
3267 which the job will grow during its execution. HTCondor will then
3268 execute the job only on machines which have enough resources, (such
3269 as virtual memory), to support executing the job. If not specified,
3270 HTCondor will automatically make a (reasonably accurate) estimate
3271 about the job's size and adjust this estimate as the program runs.
3272 If specified and underestimated, the job may crash due to the
3273 inability to acquire more address space; for example, if mal‐
3274 loc()fails. If the image size is overestimated, HTCondor may have
3275 difficulty finding machines which have the required resources.
3276 sizeis specified in KiB. For example, for an image size of 8 MiB,
3277 sizeshould be 8000.
3278
3279
3280
3281
3282
3283 initialdir = <directory-path>
3284
3285 Used to give jobs a directory with respect to file input and output.
3286 Also provides a directory (on the machine from which the job is sub‐
3287 mitted) for the job event log, when a full path is not specified.
3288
3289 For vanilla universe jobs where there is a shared file system, it is
3290 the current working directory on the machine where the job is exe‐
3291 cuted.
3292
3293 For vanilla or grid universe jobs where file transfer mechanisms are
3294 utilized (there is nota shared file system), it is the directory on
3295 the machine from which the job is submitted where the input files
3296 come from, and where the job's output files go to.
3297
3298 For standard universe jobs, it is the directory on the machine from
3299 which the job is submitted where the condor_shadowdaemon runs; the
3300 current working directory for file input and output accomplished
3301 through remote system calls.
3302
3303 For scheduler universe jobs, it is the directory on the machine from
3304 which the job is submitted where the job runs; the current working
3305 directory for file input and output with respect to relative path
3306 names.
3307
3308 Note that the path to the executable is notrelative to initialdir;
3309 if it is a relative path, it is relative to the directory in which
3310 the condor_submitcommand is run.
3311
3312
3313
3314
3315
3316 job_ad_information_attrs = <attribute-list>
3317
3318 A comma-separated list of job ClassAd attribute names. The named
3319 attributes and their values are written to the job event log when‐
3320 ever any event is being written to the log. This implements the same
3321 thing as the configuration variable EVENT_LOG_INFORMATION_ATTRS(see
3322 page ), but it applies to the job event log, instead of the system
3323 event log.
3324
3325
3326
3327
3328
3329 JobBatchName = <batch_name>
3330
3331 Set the batch name for this submit. The batch name is displayed by
3332 condor_q-batch. It is intended for use by users to give meaningful
3333 names to their jobs and to influence how condor_qgroups jobs for
3334 display. This value in a submit file can be overridden by specifying
3335 the -batch-nameargument on the condor_submitcommand line.
3336
3337
3338
3339
3340
3341 job_lease_duration = <number-of-seconds>
3342
3343 For vanilla, parallel, VM, and java universe jobs only, the duration
3344 in seconds of a job lease. The default value is 2,400, or forty min‐
3345 utes. If a job lease is not desired, the value can be explicitly set
3346 to 0 to disable the job lease semantics. The value can also be a
3347 ClassAd expression that evaluates to an integer. The HTCondor User's
3348 manual section on Special Environment Considerations has further
3349 details.
3350
3351
3352
3353
3354
3355 job_machine_attrs = <attr1, attr2, ...>
3356
3357 A comma and/or space separated list of machine attribute names that
3358 should be recorded in the job ClassAd in addition to the ones speci‐
3359 fied by the condor_schedddaemon's system configuration variable SYS‐
3360 TEM_JOB_MACHINE_ATTRS. When there are multiple run attempts, history
3361 of machine attributes from previous run attempts may be kept. The
3362 number of run attempts to store may be extended beyond the system-
3363 specified history length by using the submit file command
3364 job_machine_attrs_history_length. A machine attribute named Xwill be
3365 inserted into the job ClassAd as an attribute named MachineAttrX0.
3366 The previous value of this attribute will be named MachineAttrX1,
3367 the previous to that will be named MachineAttrX2, and so on, up to
3368 the specified history length. A history of length 1 means that only
3369 MachineAttrX0will be recorded. The value recorded in the job ClassAd
3370 is the evaluation of the machine attribute in the context of the job
3371 ClassAd when the condor_schedddaemon initiates the start up of the
3372 job. If the evaluation results in an Undefinedor Errorresult, the
3373 value recorded in the job ad will be Undefinedor Error, respec‐
3374 tively.
3375
3376
3377
3378
3379
3380 want_graceful_removal = <boolean expression>
3381
3382 When True, this causes a graceful shutdown of the job when the job
3383 is removed or put on hold, giving it time to clean up or save state.
3384 Otherwise, the job is abruptly killed. The default is false.
3385
3386
3387
3388
3389
3390 kill_sig = <signal-number>
3391
3392 When HTCondor needs to kick a job off of a machine, it will send the
3393 job the signal specified by signal-number. signal-numberneeds to be
3394 an integer which represents a valid signal on the execution machine.
3395 For jobs submitted to the standard universe, the default value is
3396 the number for SIGTSTP which tells the HTCondor libraries to initi‐
3397 ate a checkpoint of the process. For jobs submitted to other uni‐
3398 verses, the default value, when not defined, is SIGTERM , which is
3399 the standard way to terminate a program in Unix.
3400
3401
3402
3403
3404
3405 kill_sig_timeout = <seconds>
3406
3407 This submit command should no longer be used as of HTCondor version
3408 7.7.3; use job_max_vacate_timeinstead. If job_max_vacate_timeis not
3409 defined, this defines the number of seconds that HTCondor should
3410 wait following the sending of the kill signal defined by kill_sigand
3411 forcibly killing the job. The actual amount of time between sending
3412 the signal and forcibly killing the job is the smallest of this
3413 value and the configuration variable KILLING_TIMEOUT, as defined on
3414 the execute machine.
3415
3416
3417
3418
3419
3420 load_profile = <True | False>
3421
3422 When True, loads the account profile of the dedicated run account
3423 for Windows jobs. May not be used with run_as_owner.
3424
3425
3426
3427
3428
3429 match_list_length = <integer value>
3430
3431 Defaults to the value zero (0). When match_list_lengthis defined
3432 with an integer value greater than zero (0), attributes are inserted
3433 into the job ClassAd. The maximum number of attributes defined is
3434 given by the integer value. The job ClassAds introduced are given as
3435
3436 LastMatchName0 = "most-recent-Name"
3437 LastMatchName1 = "next-most-recent-Name"
3438
3439 The value for each introduced ClassAd is given by the value of the
3440 Nameattribute from the machine ClassAd of a previous execution
3441 (match). As a job is matched, the definitions for these attributes
3442 will roll, with LastMatchName1 becoming LastMatchName2 , Last‐
3443 MatchName0 becoming LastMatchName1 , and LastMatchName0 being set
3444 by the most recent value of the Nameattribute.
3445
3446 An intended use of these job attributes is in the requirements
3447 expression. The requirements can allow a job to prefer a match with
3448 either the same or a different resource than a previous match.
3449
3450
3451
3452
3453
3454 job_max_vacate_time = <integer expression>
3455
3456 An integer-valued expression (in seconds) that may be used to adjust
3457 the time given to an evicted job for gracefully shutting down. If
3458 the job's setting is less than the machine's, the job's is used. If
3459 the job's setting is larger than the machine's, the result depends
3460 on whether the job has any excess retirement time. If the job has
3461 more retirement time left than the machine's max vacate time set‐
3462 ting, then retirement time will be converted into vacating time, up
3463 to the amount requested by the job.
3464
3465 Setting this expression does not affect the job's resource require‐
3466 ments or preferences. For a job to only run on a machine with a min‐
3467 imum MachineMaxVacateTime, or to preferentially run on such
3468 machines, explicitly specify this in the requirements and/or rank
3469 expressions.
3470
3471
3472
3473
3474
3475 max_job_retirement_time = <integer expression>
3476
3477 An integer-valued expression (in seconds) that does nothing unless
3478 the machine that runs the job has been configured to provide retire‐
3479 ment time. Retirement time is a grace period given to a job to fin‐
3480 ish when a resource claim is about to be preempted. The default
3481 behavior in many cases is to take as much retirement time as the
3482 machine offers, so this command will rarely appear in a submit
3483 description file.
3484
3485 When a resource claim is to be preempted, this expression in the
3486 submit file specifies the maximum run time of the job (in seconds,
3487 since the job started). This expression has no effect, if it is
3488 greater than the maximum retirement time provided by the machine
3489 policy. If the resource claim is notpreempted, this expression and
3490 the machine retirement policy are irrelevant. If the resource claim
3491 ispreempted the job will be allowed to run until the retirement time
3492 expires, at which point it is hard-killed. The job will be soft-
3493 killed when it is getting close to the end of retirement in order to
3494 give it time to gracefully shut down. The amount of lead-time for
3495 soft-killing is determined by the maximum vacating time granted to
3496 the job.
3497
3498 Standard universe jobs and any jobs running with nice_userpriority
3499 have a default max_job_retirement_timeof 0, so no retirement time is
3500 utilized by default. In all other cases, no default value is pro‐
3501 vided, so the maximum amount of retirement time is utilized by
3502 default.
3503
3504 Setting this expression does not affect the job's resource require‐
3505 ments or preferences. For a job to only run on a machine with a min‐
3506 imum MaxJobRetirementTime, or to preferentially run on such
3507 machines, explicitly specify this in the requirements and/or rank
3508 expressions.
3509
3510
3511
3512
3513
3514 nice_user = <True | False>
3515
3516 Normally, when a machine becomes available to HTCondor, HTCondor
3517 decides which job to run based upon user and job priorities. Setting
3518 nice_userequal to Truetells HTCondor not to use your regular user
3519 priority, but that this job should have last priority among all
3520 users and all jobs. So jobs submitted in this fashion run only on
3521 machines which no other non-nice_user job wants — a true bot‐
3522 tom-feeder job! This is very handy if a user has some jobs they wish
3523 to run, but do not wish to use resources that could instead be used
3524 to run other people's HTCondor jobs. Jobs submitted in this fashion
3525 have "nice-user."prepended to the owner name when viewed from con‐
3526 dor_qor condor_userprio. The default value is False.
3527
3528
3529
3530
3531
3532 noop_job = <ClassAd Boolean Expression>
3533
3534 When this boolean expression is True, the job is immediately removed
3535 from the queue, and HTCondor makes no attempt at running the job.
3536 The log file for the job will show a job submitted event and a job
3537 terminated event, along with an exit code of 0, unless the user
3538 specifies a different signal or exit code.
3539
3540
3541
3542
3543
3544 noop_job_exit_code = <return value>
3545
3546 When noop_jobis in the submit description file and evaluates to
3547 True, this command allows the job to specify the return value as
3548 shown in the job's log file job terminated event. If not specified,
3549 the job will show as having terminated with status 0. This overrides
3550 any value specified with noop_job_exit_signal.
3551
3552
3553
3554
3555
3556 noop_job_exit_signal = <signal number>
3557
3558 When noop_jobis in the submit description file and evaluates to
3559 True, this command allows the job to specify the signal number that
3560 the job's log event will show the job having terminated with.
3561
3562
3563
3564
3565
3566 remote_initialdir = <directory-path>
3567
3568 The path specifies the directory in which the job is to be executed
3569 on the remote machine. This is currently supported in all universes
3570 except for the standard universe.
3571
3572
3573
3574
3575
3576 rendezvousdir = <directory-path>
3577
3578 Used to specify the shared file system directory to be used for file
3579 system authentication when submitting to a remote scheduler. Should
3580 be a path to a preexisting directory.
3581
3582
3583
3584
3585
3586 run_as_owner = <True | False>
3587
3588 A boolean value that causes the job to be run under the login of the
3589 submitter, if supported by the joint configuration of the submit and
3590 execute machines. On Unix platforms, this defaults to True, and on
3591 Windows platforms, it defaults to False. May not be used with
3592 load_profile. See the HTCondor manual Platform-Specific Information
3593 chapter for administrative details on configuring Windows to support
3594 this option.
3595
3596
3597
3598
3599
3600 stack_size = <size in bytes>
3601
3602 This command applies only to Linux platform jobs that are not stan‐
3603 dard universe jobs. An integer number of bytes, representing the
3604 amount of stack space to be allocated for the job. This value
3605 replaces the default allocation of stack space, which is unlimited
3606 in size.
3607
3608
3609
3610
3611
3612 submit_event_notes = <note>
3613
3614 A string that is appended to the submit event in the job's log file.
3615 For DAGMan jobs, the string DAG Node:and the node's name is automat‐
3616 ically defined for submit_event_notes, causing the logged submit
3617 event to identify the DAG node job submitted.
3618
3619
3620
3621
3622
3623 +<attribute> = <value>
3624
3625 A line that begins with a '+' (plus) character instructs condor_sub‐
3626 mitto insert the given attributeinto the job ClassAd with the given
3627 value. Note that setting an attribute should not be used in place of
3628 one of the specific commands listed above. Often, the command name
3629 does not directly correspond to an attribute name; furthermore, many
3630 submit commands result in actions more complex than simply setting
3631 an attribute or attributes. See for a list of HTCondor job
3632 attributes.
3633
3634
3635
3636
3637
3638 MACROS AND COMMENTS
3639
3640 In addition to commands, the submit description file can contain macros
3641 and comments.
3642
3643 Macros
3644
3645 Parameterless macros in the form of $(macro_name:default initial
3646 value)may be used anywhere in HTCondor submit description files to
3647 provide textual substitution at submit time. Macros can be defined
3648 by lines in the form of
3649
3650 <macro_name> = <string>
3651
3652 Two pre-defined macros are supplied by the submit description file
3653 parser. The $(Cluster)or $(ClusterId)macro supplies the value of the
3654 ClusterIdjob ClassAd attribute, and the $(Process)or $(ProcId)macro
3655 supplies the value of the ProcIdjob ClassAd attribute. These macros
3656 are intended to aid in the specification of input/output files,
3657 arguments, etc., for clusters with lots of jobs, and/or could be
3658 used to supply an HTCondor process with its own cluster and process
3659 numbers on the command line.
3660
3661 The $(Node)macro is defined for parallel universe jobs, and is espe‐
3662 cially relevant for MPI applications. It is a unique value assigned
3663 for the duration of the job that essentially identifies the machine
3664 (slot) on which a program is executing. Values assigned start at 0
3665 and increase monotonically. The values are assigned as the parallel
3666 job is about to start.
3667
3668 Recursive definition of macros is permitted. An example of a con‐
3669 struction that works is the following:
3670
3671 foo = bar
3672 foo = snap $(foo)
3673
3674 As a result, foo = snap bar.
3675
3676 Note that both left- and right- recursion works, so
3677
3678 foo = bar
3679 foo = $(foo) snap
3680
3681 has as its result foo = bar snap.
3682
3683 The construction
3684
3685 foo = $(foo) bar
3686
3687 by itself will notwork, as it does not have an initial base case.
3688 Mutually recursive constructions such as:
3689
3690 B = bar
3691 C = $(B)
3692 B = $(C) boo
3693
3694 will notwork, and will fill memory with expansions.
3695
3696 A default value may be specified, for use if the macro has no defi‐
3697 nition. Consider the example
3698
3699 D = $(E:24)
3700
3701 Where Eis not defined within the submit description file, the
3702 default value 24 is used, resulting in
3703
3704 D = 24
3705
3706 This is of limited value, as the scope of macro substitution is the
3707 submit description file. Thus, either the macro is or is not defined
3708 within the submit description file. If the macro is defined, then
3709 the default value is useless. If the macro is not defined, then
3710 there is no point in using it in a submit command.
3711
3712 To use the dollar sign character ( $ ) as a literal, without macro
3713 expansion, use
3714
3715 $(DOLLAR)
3716
3717 In addition to the normal macro, there is also a special kind of
3718 macro called a substitution macrothat allows the substitution of a
3719 machine ClassAd attribute value defined on the resource machine
3720 itself (gotten after a match to the machine has been made) into spe‐
3721 cific commands within the submit description file. The substitution
3722 macro is of the form:
3723
3724 $$(attribute)
3725
3726 As this form of the substitution macro is only evaluated within the
3727 context of the machine ClassAd, use of a scope resolution prefix
3728 TARGET.or MY.is not allowed.
3729
3730 A common use of this form of the substitution macro is for the het‐
3731 erogeneous submission of an executable:
3732
3733 executable = povray.$$(OpSys).$$(Arch)
3734
3735 Values for the OpSysand Archattributes are substituted at match time
3736 for any given resource. This example allows HTCondor to automati‐
3737 cally choose the correct executable for the matched machine.
3738
3739 An extension to the syntax of the substitution macro provides an
3740 alternative string to use if the machine attribute within the sub‐
3741 stitution macro is undefined. The syntax appears as:
3742
3743 $$(attribute:string_if_attribute_undefined)
3744
3745 An example using this extended syntax provides a path name to a
3746 required input file. Since the file can be placed in different loca‐
3747 tions on different machines, the file's path name is given as an
3748 argument to the program.
3749
3750 arguments = $$(input_file_path:/usr/foo)
3751
3752 On the machine, if the attribute input_file_pathis not defined, then
3753 the path /usr/foois used instead.
3754
3755 A further extension to the syntax of the substitution macro allows
3756 the evaluation of a ClassAd expression to define the value. In this
3757 form, the expression may refer to machine attributes by prefacing
3758 them with the TARGET.scope resolution prefix. To place a ClassAd
3759 expression into the substitution macro, square brackets are added to
3760 delimit the expression. The syntax appears as:
3761
3762 $$([ClassAd expression])
3763
3764 An example of a job that uses this syntax may be one that wants to
3765 know how much memory it can use. The application cannot detect this
3766 itself, as it would potentially use all of the memory on a multi-
3767 slot machine. So the job determines the memory per slot, reducing it
3768 by 10% to account for miscellaneous overhead, and passes this as a
3769 command line argument to the application. In the submit description
3770 file will be
3771
3772 arguments = --memory $$([TARGET.Memory * 0.9])
3773
3774 To insert two dollar sign characters ( $$ ) as literals into a Clas‐
3775 sAd string, use
3776
3777 $$(DOLLARDOLLAR)
3778
3779 The environment macro, $ENV, allows the evaluation of an environment
3780 variable to be used in setting a submit description file command.
3781 The syntax used is
3782
3783 $ENV(variable)
3784
3785 An example submit description file command that uses this function‐
3786 ality evaluates the submitter's home directory in order to set the
3787 path and file name of a log file:
3788
3789 log = $ENV(HOME)/jobs/logfile
3790
3791 The environment variable is evaluated when the submit description
3792 file is processed.
3793
3794 The $RANDOM_CHOICE macro allows a random choice to be made from a
3795 given list of parameters at submission time. For an expression, if
3796 some randomness needs to be generated, the macro may appear as
3797
3798 $RANDOM_CHOICE(0,1,2,3,4,5,6)
3799
3800 When evaluated, one of the parameters values will be chosen.
3801
3802
3803
3804
3805
3806 Comments
3807
3808 Blank lines and lines beginning with a pound sign ('#') character
3809 are ignored by the submit description file parser.
3810
3811
3812
3813
3814
3816 While processing the queuecommand in a submit file or from the command
3817 line, condor_submitwill set the values of several automatic submit
3818 variables so that they can be referred to by statements in the submit
3819 file. With the exception of Cluster and Process, if these variables are
3820 set by the submit file, they will not be modified during queueprocess‐
3821 ing.
3822
3823 ClusterId
3824
3825 Set to the integer value that the ClusterIdattribute that the job
3826 ClassAd will have when the job is submitted. All jobs in a single
3827 submit will normally have the same value for the ClusterId. If the
3828 -dry-runargument is specified, The value will be 1.
3829
3830
3831
3832
3833
3834 Cluster
3835
3836 Alternate name for the ClusterId submit variable. Before HTCondor
3837 version 8.4 this was the only name.
3838
3839
3840
3841
3842
3843 ProcId
3844
3845 Set to the integer value that the ProcIdattribute of the job ClassAd
3846 will have when the job is submitted. The value will start at 0 and
3847 increment by 1 for each job submitted.
3848
3849
3850
3851
3852
3853 Process
3854
3855 Alternate name for the ProcId submit variable. Before HTCondor ver‐
3856 sion 8.4 this was the only name.
3857
3858
3859
3860
3861
3862 Node
3863
3864 For parallel universes, set to the value #pArAlLeLnOdE# or #MpInOdE#
3865 depending on the parallel universe type For other universes it is
3866 set to nothing.
3867
3868
3869
3870
3871
3872 Step
3873
3874 Set to the step value as it varies from 0 to N-1 where N is the num‐
3875 ber provided on the queueargument. This variable changes at the same
3876 rate as ProcId when it changes at all. For submit files that don't
3877 make use of the queue number option, Step will always be 0. For sub‐
3878 mit files that don't make use of any of the foreach options, Step
3879 and ProcId will always be the same.
3880
3881
3882
3883
3884
3885 ItemIndex
3886
3887 Set to the index within the item list being processed by the various
3888 queue foreach options. For submit files that don't make use of any
3889 queue foreach list, ItemIndex will always be 0 For submit files that
3890 make use of a slice to select only some items in a foreach list,
3891 ItemIndex will only be set to selected values.
3892
3893
3894
3895
3896
3897 Row
3898
3899 Alternate name for ItemIndex.
3900
3901
3902
3903
3904
3905 Item
3906
3907 when a queue foreach option is used and no variable list is sup‐
3908 plied, this variable will be set to the value of the current item.
3909
3910
3911
3912 The automatic variables below are set before parsing the submit file,
3913 and will not vary during processing unless the submit file itself sets
3914 them.
3915
3916 ARCH
3917
3918 Set to the CPU architecture of the machine running condor_submit.
3919 The value will be the same as the automatic configuration variable
3920 of the same name.
3921
3922
3923
3924
3925
3926 OPSYS
3927
3928 Set to the name of the operating system on the machine running con‐
3929 dor_submit. The value will be the same as the automatic configura‐
3930 tion variable of the same name.
3931
3932
3933
3934
3935
3936 OPSYSANDVER
3937
3938 Set to the name and major version of the operating system on the
3939 machine running condor_submit. The value will be the same as the
3940 automatic configuration variable of the same name.
3941
3942
3943
3944
3945
3946 OPSYSMAJORVER
3947
3948 Set to the major version of the operating system on the machine run‐
3949 ning condor_submit. The value will be the same as the automatic con‐
3950 figuration variable of the same name.
3951
3952
3953
3954
3955
3956 OPSYSVER
3957
3958 Set to the version of the operating system on the machine running
3959 condor_submit. The value will be the same as the automatic configu‐
3960 ration variable of the same name.
3961
3962
3963
3964
3965
3966 SPOOL
3967
3968 Set to the full path of the HTCondor spool directory. The value will
3969 be the same as the automatic configuration variable of the same
3970 name.
3971
3972
3973
3974
3975
3976 IsLinux
3977
3978 Set to true if the operating system of the machine running con‐
3979 dor_submitis a Linux variant. Set to false otherwise.
3980
3981
3982
3983
3984
3985 IsWindows
3986
3987 Set to true if the operating system of the machine running con‐
3988 dor_submitis a Microsoft Windows variant. Set to false otherwise.
3989
3990
3991
3992
3993
3994 SUBMIT_FILE
3995
3996 Set to the full pathname of the submit file being processed by con‐
3997 dor_submit. If submit statements are read from standard input, it is
3998 set to nothing.
3999
4000
4001
4003 condor_submitwill exit with a status value of 0 (zero) upon success,
4004 and a non-zero value upon failure.
4005
4007 * Submit Description File Example 1: This example queues three jobs
4008 for execution by HTCondor. The first will be given command line
4009 arguments of 15and 2000, and it will write its standard output to
4010 foo.out1. The second will be given command line arguments of 30and
4011 2000, and it will write its standard output to foo.out2. Similarly
4012 the third will have arguments of 45and 6000, and it will use
4013 foo.out3for its standard output. Standard error output (if any) from
4014 all three programs will appear in foo.error.
4015
4016
4017
4018 ####################
4019 #
4020 # submit description file
4021 # Example 1: queuing multiple jobs with differing
4022 # command line arguments and output files.
4023 #
4024 ####################
4025
4026 Executable = foo
4027 Universe = vanilla
4028
4029 Arguments = 15 2000
4030 Output = foo.out0
4031 Error = foo.err0
4032 Queue
4033
4034 Arguments = 30 2000
4035 Output = foo.out1
4036 Error = foo.err1
4037 Queue
4038
4039 Arguments = 45 6000
4040 Output = foo.out2
4041 Error = foo.err2
4042 Queue
4043
4044 Or you can get the same results as the above submit file by using a
4045 list of arguments with the Queue statement
4046
4047
4048
4049 ####################
4050 #
4051 # submit description file
4052 # Example 1b: queuing multiple jobs with differing
4053 # command line arguments and output files, alternate syntax
4054 #
4055 ####################
4056
4057 Executable = foo
4058 Universe = vanilla
4059
4060 # generate different output and error filenames for each process
4061 Output = foo.out$(Process)
4062 Error = foo.err$(Process)
4063
4064 Queue Arguments From (
4065 15 2000
4066 30 2000
4067 45 6000
4068 )
4069
4070
4071
4072 * Submit Description File Example 2: This submit description file
4073 example queues 150 runs of program foowhich must have been compiled
4074 and linked for an Intel x86 processor running RHEL 3. HTCondor will
4075 not attempt to run the processes on machines which have less than 32
4076 Megabytes of physical memory, and it will run them on machines which
4077 have at least 64 Megabytes, if such machines are available. Stdin,
4078 stdout, and stderr will refer to in.0, out.0, and err.0for the first
4079 run of this program (process 0). Stdin, stdout, and stderr will
4080 refer to in.1, out.1, and err.1for process 1, and so forth. A log
4081 file containing entries about where and when HTCondor runs, takes
4082 checkpoints, and migrates processes in this cluster will be written
4083 into file foo.log.
4084
4085
4086
4087 ####################
4088 #
4089 # Example 2: Show off some fancy features including
4090 # use of pre-defined macros and logging.
4091 #
4092 ####################
4093
4094 Executable = foo
4095 Universe = standard
4096 Requirements = OpSys == "LINUX" && Arch =="INTEL"
4097 Rank = Memory >= 64
4098 Request_Memory = 32 Mb
4099 Image_Size = 28 Mb
4100
4101 Error = err.$(Process)
4102 Input = in.$(Process)
4103 Output = out.$(Process)
4104 Log = foo.log
4105 Queue 150
4106
4107
4108
4109 * Submit Description File Example 3: This example targets the
4110 /bin/sleepprogram to run only on a platform running a RHEL 6 operat‐
4111 ing system. The example presumes that the pool contains machines
4112 running more than one version of Linux, and this job needs the par‐
4113 ticular operating system to run correctly.
4114
4115
4116
4117 ####################
4118 #
4119 # Example 3: Run on a RedHat 6 machine
4120 #
4121 ####################
4122 Universe = vanilla
4123 Executable = /bin/sleep
4124 Arguments = 30
4125 Requirements = (OpSysAndVer == "RedHat6")
4126
4127 Error = err.$(Process)
4128 Input = in.$(Process)
4129 Output = out.$(Process)
4130 Log = sleep.log
4131 Queue
4132
4133
4134
4135 * Command Line example: The following command uses the -appendoption
4136 to add two commands before the job(s) is queued. A log file and an
4137 error log file are specified. The submit description file is
4138 unchanged.
4139
4140 condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
4141 Note that each of the added commands is contained within quote marks
4142 because there are space characters within the command.
4143
4144
4145
4146 * periodic_removeexample: A job should be removed from the queue, if
4147 the total suspension time of the job is more than half of the run
4148 time of the job.
4149
4150 Including the command
4151
4152 periodic_remove = CumulativeSuspensionTime >
4153 ((RemoteWallClockTime - CumulativeSuspensionTime)
4154 / 2.0) in the submit description file causes this to happen.
4155
4156
4157
4159 * For security reasons, HTCondor will refuse to run any jobs submit‐
4160 ted by user root (UID = 0) or by a user whose default group is group
4161 wheel (GID = 0). Jobs submitted by user root or a user with a
4162 default group of wheel will appear to sit forever in the queue in an
4163 idle state.
4164
4165
4166
4167 * All path names specified in the submit description file must be
4168 less than 256 characters in length, and command line arguments must
4169 be less than 4096 characters in length; otherwise, condor_submit‐
4170 gives a warning message but the jobs will not execute properly.
4171
4172
4173
4174 * Somewhat understandably, behavior gets bizarre if the user makes
4175 the mistake of requesting multiple HTCondor jobs to write to the
4176 same file, and/or if the user alters any files that need to be
4177 accessed by an HTCondor job which is still in the queue. For exam‐
4178 ple, the compressing of data or output files before an HTCondor job
4179 has completed is a common mistake.
4180
4181
4182
4183 * To disable checkpointing for Standard Universe jobs, include the
4184 line:
4185
4186 +WantCheckpoint = False
4187
4188 in the submit description file before the queue command(s).
4189
4191 HTCondor User Manual
4192
4194 Center for High Throughput Computing, University of Wiscon‐
4195 sin–Madison
4196
4198 Copyright © 1990-2019 Center for High Throughput Computing, Computer
4199 Sciences Department, University of Wisconsin-Madison, Madison, WI. All
4200 Rights Reserved. Licensed under the Apache License, Version 2.0.
4201
4202 date condor_submit(1)