1CONDOR_SUBMIT(1)                HTCondor Manual               CONDOR_SUBMIT(1)
2
3
4

NAME

6       condor_submit - HTCondor Manual
7
8       Queue jobs for execution under HTCondor
9
10

SYNOPSIS

12       condor_submit  [-terse  ]  [-verbose  ]  [-unused ] [-file submit_file]
13       [-name schedd_name]  [-remote  schedd_name]  [-addr  <ip:port>]  [-pool
14       pool_name]  [-disable  ] [-password passphrase] [-debug ] [-append com‐
15       mand ...][-batch-name batch_name] [-spool ] [-dump filename] [-interac‐
16       tive  ]  [-allow-crlf-script  ]  [-dry-run  ] [-maxjobs number-of-jobs]
17       [-single-cluster ] [-stm method] [<submit-variable>=<value>  ]  [submit
18       description file ] [-queue queue_arguments]
19

DESCRIPTION

21       condor_submit  is  the  program for submitting jobs for execution under
22       HTCondor. condor_submit requires one or more  submit  description  com‐
23       mands  to  direct  the  queuing of jobs. These commands may come from a
24       file, standard input, the command line, or  from  some  combination  of
25       these.  One submit description may contain specifications for the queu‐
26       ing of many HTCondor jobs at once. A single invocation of condor_submit
27       may cause one or more clusters. A cluster is a set of jobs specified in
28       the submit description between queue    commands  for  which  the  exe‐
29       cutable is not changed. It is advantageous to submit multiple jobs as a
30       single cluster because:
31
32       • Much less memory is used by the scheduler to hold the same number  of
33         jobs.
34
35       • Only  one copy of the checkpoint file is needed to represent all jobs
36         in a cluster until they begin execution.
37
38       • There is much less overhead involved for HTCondor to start  the  next
39         job  in  a cluster than for HTCondor to start a new cluster. This can
40         make a big difference when submitting lots of short jobs.
41
42       Multiple clusters may be specified within a single submit  description.
43       Each cluster must specify a single executable.
44
45       The job ClassAd attribute ClusterId identifies a cluster.
46
47       The  submit  description file argument is the path and file name of the
48       submit description file. If this optional argument is the dash  charac‐
49       ter (-), then the commands are taken from standard input. If - is spec‐
50       ified for the submit description file, -verbose is implied; this can be
51       overridden by specifying -terse.
52
53       If no submit discription file argument is given, and no -queue argument
54       is given, commands are taken automatically from standard input.
55
56       Note that submission of jobs from a Windows machine requires a  stashed
57       password  to allow HTCondor to impersonate the user submitting the job.
58       To stash a password, use the condor_store_cred command. See the  manual
59       page for details.
60
61       For lengthy lines within the submit description file, the backslash (\)
62       is a line continuation character. Placing the backslash at the end of a
63       line  causes  the  current line's command to be continued with the next
64       line of the file. Submit description files may contain comments. A com‐
65       ment is any line beginning with a pound character (#).
66

OPTIONS

68          -terse Terse output - display JobId ranges only.
69
70          -verbose
71                 Verbose output - display the created job ClassAd
72
73          -unused
74                 As  a default, causes no warnings to be issued about user-de‐
75                 fined macros not being used  within  the  submit  description
76                 file.  The  meaning reverses (toggles) when the configuration
77                 variable WARN_ON_UNUSED_SUBMIT_FILE_MACROS
78                   is set to the non default  value  of  False.  Printing  the
79                 warnings can help identify spelling errors of submit descrip‐
80                 tion file commands. The warnings are sent to stderr.
81
82          -file submit_file
83                 Use submit_file as  the  submit  discription  file.  This  is
84                 equivalent  to  providing  submit_file as an argument without
85                 the preceeding -file.
86
87          -name schedd_name
88                 Submit to the specified condor_schedd.  Use  this  option  to
89                 submit  to  a condor_schedd other than the default local one.
90                 schedd_name is the value of the Name ClassAd attribute on the
91                 machine where the condor_schedd daemon runs.
92
93          -remote schedd_name
94                 Submit  to the specified condor_schedd, spooling all required
95                 input files over the network connection. schedd_name  is  the
96                 value  of the Name ClassAd attribute on the machine where the
97                 condor_schedd daemon runs. This option is equivalent to using
98                 both -name and -spool.
99
100          -addr <ip:port>
101                 Submit  to the condor_schedd at the IP address and port given
102                 by the sinful string argument <ip:port>.
103
104          -pool pool_name
105                 Look in the specified pool for the  condor_schedd  to  submit
106                 to.  This option is used with -name or -remote.
107
108          -disable
109                 Disable file permission checks when submitting a job for read
110                 permissions on all input files, such as those defined by com‐
111                 mands  input    and  transfer_input_files  , as well as write
112                 permission to output files, such as a log file defined by log
113                 and  output  files  defined  with  output    or transfer_out‐
114                 put_files  .
115
116          -password passphrase
117                 Specify a password to the MyProxy server.
118
119          -debug Cause debugging information to be sent to  stderr,  based  on
120                 the value of the configuration variable TOOL_DEBUG.
121
122          -append command
123                 Augment  the commands in the submit description file with the
124                 given command. This command will be considered to immediately
125                 precede the queue command within the submit description file,
126                 and come after all other previous commands.  If  the  command
127                 specifies a queue command, as in the example
128
129                 condor_submit mysubmitfile -append "queue input in A, B, C"
130
131                 then the entire -append command line option and its arguments
132                 are converted to
133
134                 condor_submit mysubmitfile -queue input in A, B, C
135
136                 The submit description file is not  modified.  Multiple  com‐
137                 mands  are  specified  by  using  the -append option multiple
138                 times. Each new command is given in a  separate  -append  op‐
139                 tion.  Commands  with spaces in them will need to be enclosed
140                 in double quote marks.
141
142          -batch-name batch_name
143                 Set the batch name for this submit. The batch  name  is  dis‐
144                 played by condor_q -batch. It is intended for use by users to
145                 give meaningful names to their jobs and to influence how con‐
146                 dor_q  groups  jobs  for  display. Use of this argument takes
147                 precedence over a batch name specified in the submit descrip‐
148                 tion file itself.
149
150          -spool Spool all required input files, job event log, and proxy over
151                 the connection to the condor_schedd. After submission, modify
152                 local  copies  of  the files without affecting your jobs. Any
153                 output files for completed jobs need  to  be  retrieved  with
154                 condor_transfer_data.
155
156          -dump filename
157                 Sends  all  ClassAds to the specified file, instead of to the
158                 condor_schedd.
159
160          -interactive
161                 Indicates that the user wants to run an interactive shell  on
162                 an  execute machine in the pool. This is equivalent to creat‐
163                 ing a submit description file of  a  vanilla  universe  sleep
164                 job,  and then running condor_ssh_to_job by hand. Without any
165                 additional arguments,  condor_submit  with  the  -interactive
166                 flag  creates  a dummy vanilla universe job that sleeps, sub‐
167                 mits it to the local scheduler, waits for the job to run, and
168                 then  launches  condor_ssh_to_job to run a shell. If the user
169                 would like to run the shell on a machine that matches a  par‐
170                 ticular  requirements expression, the submit description file
171                 is specified, and it will contain the expression.  Note  that
172                 all  policy  expressions  specified in the submit description
173                 file are honored, but any executable   or universe   commands
174                 are  overwritten to be sleep and vanilla. The job ClassAd at‐
175                 tribute InteractiveJob is set to True to identify interactive
176                 jobs for condor_startd policy usage.
177
178          -allow-crlf-script
179                 Changes  the  check  for  an  invalid line ending on the exe‐
180                 cutable script's #! line from an ERROR to a WARNING.  The  #!
181                 line  will be ignored by Windows, so it won't matter if it is
182                 invalid; but Unix and Linux will not run a script that has  a
183                 Windows/DOS  line  ending on the first line of the script. So
184                 condor_submit will not allow such a script to be submitted as
185                 the job's executable unless this option is supplied.
186
187          -dry-run file
188                 Parse  the submit description file, sending the resulting job
189                 ClassAd to the file given by file,  but  do  not  submit  the
190                 job(s).  This  permits  observation of the job specification,
191                 and it facilitates debugging the submit description file con‐
192                 tents. If file is -, the output is written to stdout.
193
194          -maxjobs number-of-jobs
195                 If  the total number of jobs specified by the submit descrip‐
196                 tion file is more  than  the  integer  value  given  by  num‐
197                 ber-of-jobs,  then no jobs are submitted for execution and an
198                 error message is generated.  A 0 or negative  value  for  the
199                 number-of-jobs causes no limit to be imposed.
200
201          -single-cluster
202                 If  the  jobs specified by the submit description file causes
203                 more than a single cluster value to be assigned, then no jobs
204                 are  submitted  for  execution and an error message is gener‐
205                 ated.
206
207          -stm method
208                 Specify the method use  to  move  a  sandbox  into  HTCondor.
209                 method is one of stm_use_schedd_only or stm_use_transferd.
210
211          <submit-variable>=<value>
212                 Defines a submit command or submit variable with a value, and
213                 parses it as if it was placed at the beginning of the  submit
214                 description file. The submit description file is not changed.
215                 To correctly parse the condor_submit command line,  this  op‐
216                 tion  must be specified without white space characters before
217                 and after the equals sign (=), or the entire option  must  be
218                 surrounded by double quote marks.
219
220          -queue queue_arguments
221                 A command line specification of how many jobs to queue, which
222                 is only permitted if the submit  description  file  does  not
223                 have a queue command. The queue_arguments are the same as may
224                 be within a submit  description  file.  The  parsing  of  the
225                 queue_arguments  finishes  at  the  end of the line or when a
226                 dash character (-) is encountered. Therefore, its best place‐
227                 ment  within  the command line will be at the end of the com‐
228                 mand line.
229
230                 On a Unix command line, the shell expands file  globs  before
231                 parsing occurs.
232

SUBMIT DESCRIPTION FILE COMMANDS

234       Note:  more  information on submitting HTCondor jobs can be found here:
235       /users-manual/submitting-a-job.
236
237       As of version 8.5.6, the  condor_submit  language  supports  multi-line
238       values  in  commands.  The syntax is the same as the configuration lan‐
239       guage (see more details  here:  admin-manual/introduction-to-configura‐
240       tion:multi-line values).
241
242       Each  submit description file describes one or more clusters of jobs to
243       be placed in the HTCondor execution pool. All jobs in  a  cluster  must
244       share the same executable, but they may have different input and output
245       files, and different program arguments. The submit description file  is
246       generally  the last command-line argument to condor_submit. If the sub‐
247       mit description file argument is omitted, condor_submit will  read  the
248       submit description from standard input.
249
250       The  submit  description file must contain at least one executable com‐
251       mand and at least one queue command. All of the other commands have de‐
252       fault actions.
253
254       Note  that a submit file that contains more than one executable command
255       will produce multiple clusters when submitted. This  is  not  generally
256       recommended,  and  is  not allowed for submit files that are run as DAG
257       node jobs by condor_dagman.
258
259       The commands which can appear in the submit description file are numer‐
260       ous. They are listed here in alphabetical order by category.
261
262       BASIC COMMANDS
263
264          arguments = <argument_list>
265                 List of arguments to be supplied to the executable as part of
266                 the command line.
267
268                 In the java universe, the first argument must be the name  of
269                 the class containing main.
270
271                 There  are  two permissible formats for specifying arguments,
272                 identified as the old syntax and the new syntax. The old syn‐
273                 tax  supports white space characters within arguments only in
274                 special circumstances; when used, the command line  arguments
275                 are  represented  in  the job ClassAd attribute Args. The new
276                 syntax supports uniform quoting  of  white  space  characters
277                 within  arguments;  when used, the command line arguments are
278                 represented in the job ClassAd attribute Arguments.
279
280                 Old Syntax
281
282                 In the old syntax, individual command line arguments are  de‐
283                 limited  (separated)  by  space characters. To allow a double
284                 quote mark in an argument, it is escaped  with  a  backslash;
285                 that  is, the two character sequence \" becomes a single dou‐
286                 ble quote mark within an argument.
287
288                 Further interpretation of the argument string differs depend‐
289                 ing  on the operating system. On Windows, the entire argument
290                 string is passed verbatim (other than the backslash in  front
291                 of  double quote marks) to the Windows application. Most Win‐
292                 dows applications will allow spaces within an argument  value
293                 by  surrounding the argument with double quotes marks. In all
294                 other cases, there is no further interpretation of the  argu‐
295                 ments.
296
297                 Example:
298
299                     arguments = one \"two\" 'three'
300
301                 Produces in Unix vanilla universe:
302
303                     argument 1: one
304                     argument 2: "two"
305                     argument 3: 'three'
306
307                 New Syntax
308
309                 Here are the rules for using the new syntax:
310
311                 1. The  entire string representing the command line arguments
312                    is surrounded by double  quote  marks.  This  permits  the
313                    white  space  characters of spaces and tabs to potentially
314                    be embedded within a single argument. Putting  the  double
315                    quote  mark within the arguments is accomplished by escap‐
316                    ing it with another double quote mark.
317
318                 2. The white space characters of spaces or tabs delimit argu‐
319                    ments.
320
321                 3. To embed white space characters of spaces or tabs within a
322                    single argument, surround the entire argument with  single
323                    quote marks.
324
325                 4. To insert a literal single quote mark, escape it within an
326                    argument already delimited by single quote marks by adding
327                    another single quote mark.
328
329                 Example:
330
331                     arguments = "3 simple arguments"
332
333                 Produces:
334
335                     argument 1: 3
336                     argument 2: simple
337                     argument 3: arguments
338
339                 Another example:
340
341                     arguments = "one 'two with spaces' 3"
342
343                 Produces:
344
345                     argument 1: one
346                     argument 2: two with spaces
347                     argument 3: 3
348
349                 And yet another example:
350
351                     arguments = "one ""two"" 'spacey ''quoted'' argument'"
352
353                 Produces:
354
355                     argument 1: one
356                     argument 2: "two"
357                     argument 3: spacey 'quoted' argument
358
359                 Notice  that  in the new syntax, the backslash has no special
360                 meaning.  This is for the convenience of Windows users.
361
362
363          environment = <parameter_list>
364                 List of environment
365                  variables.
366
367                 There are two different formats for specifying  the  environ‐
368                 ment  variables:  the  old format and the new format. The old
369                 format is retained  for  backward-compatibility.  It  suffers
370                 from  a platform-dependent syntax and the inability to insert
371                 some special characters into the environment.
372
373                 The new syntax for specifying environment values:
374
375                 1. Put double quote marks around the entire argument  string.
376                    This  distinguishes  the  new syntax from the old. The old
377                    syntax does not have double quote  marks  around  it.  Any
378                    literal  double  quote marks within the string must be es‐
379                    caped by repeating the double quote mark.
380
381                 2. Each environment entry has the form
382
383                        <name>=<value>
384
385                 3. Use white space (space or tab characters) to separate  en‐
386                    vironment entries.
387
388                 4. To  put  any white space in an environment entry, surround
389                    the space and as much of the surrounding entry as  desired
390                    with single quote marks.
391
392                 5. To  insert  a literal single quote mark, repeat the single
393                    quote mark anywhere inside of a section surrounded by sin‐
394                    gle quote marks.
395
396                 Example:
397
398                     environment = "one=1 two=""2"" three='spacey ''quoted'' value'"
399
400                 Produces the following environment entries:
401
402                     one=1
403                     two="2"
404                     three=spacey 'quoted' value
405
406                 Under  the  old  syntax, there are no double quote marks sur‐
407                 rounding the environment specification. Each environment  en‐
408                 try remains of the form
409
410                     <name>=<value>
411
412                 Under  Unix,  list multiple environment entries by separating
413                 them with a semicolon (;). Under Windows,  separate  multiple
414                 entries  with a vertical bar (|). There is no way to insert a
415                 literal semicolon under Unix or a literal vertical bar  under
416                 Windows.  Note  that spaces are accepted, but rarely desired,
417                 characters within parameter names and  values,  because  they
418                 are  treated as literal characters, not separators or ignored
419                 white space. Place spaces within the parameter list  only  if
420                 required.
421
422                 A Unix example:
423
424                     environment = one=1;two=2;three="quotes have no 'special' meaning"
425
426                 This produces the following:
427
428                     one=1
429                     two=2
430                     three="quotes have no 'special' meaning"
431
432                 If  the  environment  is set with the environment command and
433                 getenv   is also set to true, values specified with  environ‐
434                 ment  override values in the submitter's environment (regard‐
435                 less of the order of the environment and getenv commands).
436
437
438          error = <pathname>
439                 A path and file name used by HTCondor to  capture  any  error
440                 messages the program would normally write to the screen (that
441                 is, this file becomes stderr). A path is given  with  respect
442                 to the file system of the machine on which the job is submit‐
443                 ted. The file is written (by the job) in the  remote  scratch
444                 directory  of the machine where the job is executed. When the
445                 job exits, the resulting file is transferred back to the  ma‐
446                 chine  where  the job was submitted, and the path is utilized
447                 for file placement. If not specified, the  default  value  of
448                 /dev/null  is  used  for submission to a Unix machine. If not
449                 specified, error messages are ignored  for  submission  to  a
450                 Windows  machine.  More  than one job should not use the same
451                 error file, since this will cause one job  to  overwrite  the
452                 errors  of  another.  If  HTCondor detects that the error and
453                 output files for a job are the same, it will run the job such
454                 that the output and error data is merged.
455
456          executable = <pathname>
457                 An  optional  path and a required file name of the executable
458                 file for this job  cluster.  Only  one  executable    command
459                 within  a submit description file is guaranteed to work prop‐
460                 erly.  More than one often works.
461
462                 If no path or a relative path is used,  then  the  executable
463                 file is presumed to be relative to the current working direc‐
464                 tory of the user as the condor_submit command is issued.
465
466                 If submitting into the standard universe, then the named exe‐
467                 cutable  must have been re-linked with the HTCondor libraries
468                 (such as via the condor_compile command). If submitting  into
469                 the vanilla universe (the default), then the named executable
470                 need not be re-linked and can be any process which can run in
471                 the  background (shell scripts work fine as well). If submit‐
472                 ting into the Java universe, then the argument must be a com‐
473                 piled .class file.
474
475
476          getenv = <True | False>
477                 If getenv is set to
478                  True, then condor_submit will copy all of the user's current
479                 shell environment variables at the  time  of  job  submission
480                 into the job ClassAd. The job will therefore execute with the
481                 same set of environment variables that the user had at submit
482                 time. Defaults to False.
483
484                 If  the  environment  is set with the environment command and
485                 getenv is also set to true, values specified with environment
486                 override values in the submitter's environment (regardless of
487                 the order of the environment and getenv commands).
488
489          input = <pathname>
490                 HTCondor assumes that its jobs are long-running, and that the
491                 user  will not wait at the terminal for their completion. Be‐
492                 cause of this, the standard files which normally  access  the
493                 terminal,  (stdin,  stdout, and stderr), must refer to files.
494                 Thus, the file name specified with input   should contain any
495                 keyboard  input  the program requires (that is, this file be‐
496                 comes stdin). A path is given with respect to the file system
497                 of  the  machine  on  which the job is submitted. The file is
498                 transferred before execution to the remote scratch  directory
499                 of  the  machine where the job is executed. If not specified,
500                 the default value of /dev/null is used for  submission  to  a
501                 Unix  machine. If not specified, input is ignored for submis‐
502                 sion to a Windows machine.  For  grid  universe  jobs,  input
503                 may  be  a  URL  that  the Globus tool globus_url_copy under‐
504                 stands.
505
506                 Note that this command does not refer to the command-line ar‐
507                 guments of the program. The command-line arguments are speci‐
508                 fied by the arguments   command.
509
510
511          log = <pathname>
512                 Use log   to specify a file name where HTCondor will write  a
513                 log file of what is happening with this job cluster, called a
514                 job event log. For example, HTCondor will place a  log  entry
515                 into  this  file  when and where the job begins running, when
516                 the job produces a checkpoint, or moves (migrates) to another
517                 machine, and when the job completes. Most users find specify‐
518                 ing a log file to be handy; its use is recommended. If no log
519                 entry  is  specified, HTCondor does not create a log for this
520                 cluster. If a relative path is specified, it is  relative  to
521                 the  current working directory as the job is submitted or the
522                 directory specified by submit command initialdir on the  sub‐
523                 mit machine.
524
525
526          log_xml = <True | False>
527                 If  log_xml    is  True,  then the job event log file will be
528                 written in ClassAd XML. If not specified, XML  is  not  used.
529                 Note that the file is an XML fragment; it is missing the file
530                 header and footer. Do not mix XML and non-XML within a single
531                 file.  If multiple jobs write to a single job event log file,
532                 ensure that all of the jobs specify this option in  the  same
533                 way.
534
535
536
537
538          notification = <Always | Complete | Error | Never>
539                 Owners  of  HTCondor jobs are notified by e-mail when certain
540                 events occur. If defined by Always, the owner will  be  noti‐
541                 fied  whenever the job produces a checkpoint, as well as when
542                 the job completes. If defined by Complete, the owner will  be
543                 notified  when  the  job terminates. If defined by Error, the
544                 owner will only be notified if the job terminates abnormally,
545                 (as  defined by JobSuccessExitCode, if defined) or if the job
546                 is placed on hold because of a failure, and not by  user  re‐
547                 quest.  If defined by Never (the default), the owner will not
548                 receive e-mail, regardless to what happens to  the  job.  The
549                 HTCondor  User's  manual documents statistics included in the
550                 e-mail.
551
552          notify_user = <email-address>
553                 Used to specify the e-mail address to use when HTCondor sends
554                 e-mail  about  a  job. If not specified, HTCondor defaults to
555                 using the e-mail address defined by
556
557                     job-owner@UID_DOMAIN
558
559                 where the configuration variable UID_DOMAIN
560                   is specified by the HTCondor site administrator. If UID_DO‐
561                 MAIN   has not been specified, HTCondor sends the e-mail to:
562
563                     job-owner@submit-machine-name
564
565
566
567          output = <pathname>
568                 The  output   file captures any information the program would
569                 ordinarily write to the screen (that is,  this  file  becomes
570                 stdout).  A  path is given with respect to the file system of
571                 the machine on which the job is submitted. The file is  writ‐
572                 ten  (by  the job) in the remote scratch directory of the ma‐
573                 chine where the job is executed. When the job exits, the  re‐
574                 sulting file is transferred back to the machine where the job
575                 was submitted, and the path is utilized for  file  placement.
576                 If  not specified, the default value of /dev/null is used for
577                 submission to a Unix machine. If not specified, output is ig‐
578                 nored  for  submission  to  a  Windows machine. Multiple jobs
579                 should not use the same output file, since  this  will  cause
580                 one  job  to overwrite the output of another. If HTCondor de‐
581                 tects that the error and output files for a job are the same,
582                 it  will  run  the job such that the output and error data is
583                 merged.
584
585                 Note that if a program explicitly opens and writes to a file,
586                 that file should not be specified as the output   file.
587
588
589          priority = <integer>
590                 An HTCondor job priority can be any integer, with 0 being the
591                 default. Jobs with higher numerical priority will run  before
592                 jobs  with  lower numerical priority. Note that this priority
593                 is on a per user basis. One user with many jobs may use  this
594                 command  to order his/her own jobs, and this will have no ef‐
595                 fect on whether or not these jobs will run ahead  of  another
596                 user's jobs.
597
598                 Note  that  the  priority  setting in an HTCondor submit file
599                 will be overridden by condor_dagman if  the  submit  file  is
600                 used for a node in a DAG, and the priority of the node within
601                 the  DAG  is  non-zero   (see    users-manual/dagman-applica‐
602                 tions:advanced features of dagman for more details).
603
604          queue [<int expr> ]
605                 Places  zero  or  more  copies  of  the job into the HTCondor
606                 queue.
607
608          queue  [<int expr> ] [<varname> ] in [slice ] <list of items> Places
609                 zero or more copies of the job in the queue based on items in
610                 a <list of items>
611
612          queue  [<int expr> ] [<varname> ] matching [files | dirs ] [slice  ]
613                 <list  of  items  with  file  globbing>]  Places zero or more
614                 copies of the job in the queue based on files  that  match  a
615                 <list of items with file globbing>
616
617          queue  [<int expr> ] [<list of varnames> ] from [slice ] <file name>
618                 | <list of items>] Places zero or more copies of the  job  in
619                 the  queue  based on lines from the submit file or from <file
620                 name>
621
622                 The optional argument <int expr> specifies how many times  to
623                 repeat  the  job  submission for a given set of arguments. It
624                 may be an integer or an expression that evaluates to an inte‐
625                 ger,  and  it  defaults  to 1. All but the first form of this
626                 command are various ways of specifying a list of items.  When
627                 these  forms are used <int expr> jobs will be queued for each
628                 item in the list. The in, matching and from keyword indicates
629                 how the list will be specified.
630
631in The list of items is an explicit comma and/or space sep‐
632                   arated <list of items>. If the <list of items> begins  with
633                   an  open paren, and the close paren is not on the same line
634                   as the open, then the list continues until a line that  be‐
635                   gins with a close paren is read from the submit file.
636
637matching  Each  item  in the <list of items with file glob‐
638                   bing> will be matched against the names of files and direc‐
639                   tories relative to the current directory, the set of match‐
640                   ing names is the resulting list of items.
641
642files Only filenames will matched.
643
644dirs Only directory names will be matched.
645
646from <file name> | <list of items>  Each  line  from  <file
647                   name>  or <list of items> is a single item, this allows for
648                   multiple variables to be set  for  each  item.  Lines  from
649                   <file  name>  or  <list  of  items>  will be split on comma
650                   and/or space until there are values for each of  the  vari‐
651                   ables  specified  in  <list of varnames>. The last variable
652                   will contain the remainder of the line. When the  <list  of
653                   items>  form  is  used,  the list continues until the first
654                   line that begins with a close paren,  and  lines  beginning
655                   with  pound  sign  ('#')  will  be skipped.  When using the
656                   <file name> form, if the <file name> ends with |,  then  it
657                   will  be executed as a script whatever the script writes to
658                   stdout will be the list of items.
659
660                 The optional argument <varname> or <list of varnames> is  the
661                 name  or  names of of variables that will be set to the value
662                 of the current item when queuing the job. If no <varname>  is
663                 specified  the variable ITEM will be used. Leading and trail‐
664                 ing whitespace be trimmed. The optional argument <slice> is a
665                 python  style  slice  selecting only some of the items in the
666                 list of items. Negative step values are not supported.
667
668                 A submit file may contain more than  one  queue    statement,
669                 and if desired, any commands may be placed between subsequent
670                 queue   commands, such as new input  ,  output   ,  error   ,
671                 initialdir   ,  or  arguments   commands.  This is handy when
672                 submitting multiple runs into one cluster with one submit de‐
673                 scription file.
674
675
676          universe = <vanilla | standard | scheduler | local | grid | java| vm
677          | parallel | docker>
678                 Specifies which HTCondor universe to use  when  running  this
679                 job.  The  HTCondor  universe specifies an HTCondor execution
680                 environment.
681
682                 The vanilla universe is the default (except where the config‐
683                 uration variable DEFAULT_UNIVERSE
684                   defines  it otherwise), and is an execution environment for
685                 jobs which do not use HTCondor's mechanisms for taking check‐
686                 points; these are ones that have not been linked with the HT‐
687                 Condor libraries. Use the vanilla universe  to  submit  shell
688                 scripts to HTCondor.
689
690                 The  standard  universe tells HTCondor that this job has been
691                 re-linked via condor_compile with the HTCondor libraries  and
692                 therefore  supports  taking  checkpoints  and  remote  system
693                 calls.
694
695                 The scheduler universe is for a job that is to run on the ma‐
696                 chine  where  the job is submitted. This universe is intended
697                 for a job that acts as a metascheduler and will not  be  pre‐
698                 empted.
699
700                 The local universe is for a job that is to run on the machine
701                 where the job is submitted. This universe runs the job  imme‐
702                 diately and will not preempt the job.
703
704                 The grid universe forwards the job to an external job manage‐
705                 ment system. Further specification of the  grid  universe  is
706                 done with the grid_resource command.
707
708                 The java universe is for programs written to the Java Virtual
709                 Machine.
710
711                 The vm universe facilitates the execution of  a  virtual  ma‐
712                 chine.
713
714                 The  parallel  universe  is for parallel jobs (e.g. MPI) that
715                 require multiple machines in order to run.
716
717                 The docker universe runs a docker container  as  an  HTCondor
718                 job.
719
720       COMMANDS FOR MATCHMAKING
721
722          rank = <ClassAd Float Expression>
723                 A  ClassAd  Floating-Point expression that states how to rank
724                 machines which have already met the requirements  expression.
725                 Essentially,  rank  expresses  preference.  A  higher numeric
726                 value equals better rank. HTCondor will give the job the  ma‐
727                 chine with the highest rank.  For example,
728
729                     request_memory = max({60, Target.TotalSlotMemory})
730                     rank = Memory
731
732                 asks  HTCondor  to find all available machines with more than
733                 60 megabytes of memory and give to the job the  machine  with
734                 the  most  amount  of memory. The HTCondor User's Manual con‐
735                 tains complete information on the syntax  and  available  at‐
736                 tributes that can be used in the ClassAd expression.
737
738
739          request_cpus = <num-cpus>
740                 A  requested  number  of  CPUs (cores). If not specified, the
741                 number requested will be 1. If specified, the expression
742
743                     && (RequestCpus <= Target.Cpus)
744
745                 is appended to the requirements expression for the job.
746
747                 For pools that  enable  dynamic  condor_startd  provisioning,
748                 specifies  the minimum number of CPUs requested for this job,
749                 resulting in a dynamic slot  being  created  with  this  many
750                 cores.
751
752
753          request_disk = <quantity>
754                 The  requested amount of disk space in KiB requested for this
755                 job. If not specified, it will be set to the job ClassAd  at‐
756                 tribute DiskUsage. The expression
757
758                     && (RequestDisk <= Target.Disk)
759
760                 is appended to the requirements expression for the job.
761
762                 For  pools  that enable dynamic condor_startd provisioning, a
763                 dynamic slot will be created with at  least  this  much  disk
764                 space.
765
766                 Characters  may  be appended to a numerical value to indicate
767                 units.  K or KB indicates KiB, 210 numbers of bytes. M or  MB
768                 indicates  MiB,  220 numbers of bytes. G or GB indicates GiB,
769                 230 numbers of bytes. T or TB indicates TiB, 240  numbers  of
770                 bytes.
771
772
773          request_memory = <quantity>
774                 The  required  amount of memory in MiB that this job needs to
775                 avoid excessive swapping. If not  specified  and  the  submit
776                 command  vm_memory    is  specified, then the value specified
777                 for vm_memory   defines request_memory   .   If  neither  re‐
778                 quest_memory  nor  vm_memory   is specified, the value is set
779                 by the configuration variable JOB_DEFAULT_REQUESTMEMORY
780                  . The actual amount of memory used by a job  is  represented
781                 by the job ClassAd attribute MemoryUsage.
782
783                 For  pools  that enable dynamic condor_startd provisioning, a
784                 dynamic slot will be created with at least this much RAM.
785
786                 The expression
787
788                     && (RequestMemory <= Target.Memory)
789
790                 is appended to the requirements expression for the job.
791
792                 Characters may be appended to a numerical value  to  indicate
793                 units.   K or KB indicates KiB, 210 numbers of bytes. M or MB
794                 indicates MiB, 220 numbers of bytes. G or GB  indicates  GiB,
795                 230  numbers  of bytes. T or TB indicates TiB, 240 numbers of
796                 bytes.
797
798
799
800
801          request_<name> = <quantity>
802                 The required amount of the custom machine resource identified
803                 by <name> that this job needs. The custom machine resource is
804                 defined in the machine's configuration.  Machines  that  have
805                 available GPUs will define <name> to be GPUs.
806
807
808          requirements = <ClassAd Boolean Expression>
809                 The  requirements  command  is  a  boolean ClassAd expression
810                 which uses C-like operators. In order for  any  job  in  this
811                 cluster  to run on a given machine, this requirements expres‐
812                 sion must evaluate to true on the given machine.
813
814                 For scheduler and local universe jobs, the  requirements  ex‐
815                 pression  is  evaluated  against  the Scheduler ClassAd which
816                 represents the the condor_schedd daemon running on the submit
817                 machine,  rather  than a remote machine. Like all commands in
818                 the submit description file, if  multiple  requirements  com‐
819                 mands  are  present, all but the last one are ignored. By de‐
820                 fault, condor_submit appends the following clauses to the re‐
821                 quirements expression:
822
823                 1. Arch  and OpSys are set equal to the Arch and OpSys of the
824                    submit machine. In other words: unless you request  other‐
825                    wise,  HTCondor  will give your job machines with the same
826                    architecture and operating system version as  the  machine
827                    running condor_submit.
828
829                 2. Cpus  >=  RequestCpus,  if  the  job ClassAd attribute Re‐
830                    questCpus is defined.
831
832                 3. Disk >= RequestDisk, if the job ClassAd attribute Request‐
833                    Disk  is defined. Otherwise, Disk >= DiskUsage is appended
834                    to the requirements. The DiskUsage attribute  is  initial‐
835                    ized  to  the  size of the executable plus the size of any
836                    files specified in a transfer_input_files command. It  ex‐
837                    ists  to  ensure  there is enough disk space on the target
838                    machine for HTCondor to copy over both the executable  and
839                    needed input files. The DiskUsage attribute represents the
840                    maximum amount of total disk space required by the job  in
841                    kilobytes.  HTCondor  automatically  updates the DiskUsage
842                    attribute approximately every 20  minutes  while  the  job
843                    runs with the amount of space being used by the job on the
844                    execute machine.
845
846                 4. Memory >= RequestMemory, if the job ClassAd attribute  Re‐
847                    questMemory is defined.
848
849                 5. If  Universe  is  set  to Vanilla, FileSystemDomain is set
850                    equal to the submit machine's FileSystemDomain.
851
852                 View the requirements of a job which has already been submit‐
853                 ted  (along  with everything else about the job ClassAd) with
854                 the command  condor_q  -l;  see  the  command  reference  for
855                 /man-pages/condor_q.  Also, see the HTCondor Users Manual for
856                 complete information on the syntax and  available  attributes
857                 that can be used in the ClassAd expression.
858
859       FILE TRANSFER COMMANDS
860
861
862
863          dont_encrypt_input_files = < file1,file2,file... >
864                 A  comma  and/or space separated list of input files that are
865                 not to be network encrypted when transferred  with  the  file
866                 transfer  mechanism.  Specification  of  files in this manner
867                 overrides configuration that would use encryption. Each input
868                 file  must  also be in the list given by transfer_input_files
869                 .  When a path to an input file or  directory  is  specified,
870                 this  specifies  the  path  to the file on the submit side. A
871                 single wild card character (*) may be used in each file name.
872
873
874
875          dont_encrypt_output_files = < file1,file2,file... >
876                 A comma and/or space separated list of output files that  are
877                 not  to  be  network encrypted when transferred back with the
878                 file transfer mechanism. Specification of files in this  man‐
879                 ner  overrides  configuration  that would use encryption. The
880                 output file(s) must also either  be  in  the  list  given  by
881                 transfer_output_files  or be discovered and to be transferred
882                 back with the file transfer mechanism. When a path to an out‐
883                 put  file  or directory is specified, this specifies the path
884                 to the file on the execute side. A single wild card character
885                 (*) may be used in each file name.
886
887
888          encrypt_execute_directory = <True | False>
889                 Defaults  to False. If set to True, HTCondor will encrypt the
890                 contents of the remote scratch directory of the machine where
891                 the  job  is  executed. This encryption is transparent to the
892                 job itself, but ensures that files left behind on  the  local
893                 disk  of  the execute machine, perhaps due to a system crash,
894                 will remain private. In addition, condor_submit  will  append
895                 to the job's requirements expression
896
897                     && (TARGET.HasEncryptExecuteDirectory)
898
899                 to  ensure the job is matched to a machine that is capable of
900                 encrypting the contents of the execute directory.  This  sup‐
901                 port  is  limited to Windows platforms that use the NTFS file
902                 system and Linux platforms with  the  ecryptfs-utils  package
903                 installed.
904
905
906
907          encrypt_input_files = < file1,file2,file... >
908                 A  comma  and/or space separated list of input files that are
909                 to be network encrypted when transferred with the file trans‐
910                 fer  mechanism.   Specification of files in this manner over‐
911                 rides configuration that would not use encryption. Each input
912                 file  must  also be in the list given by transfer_input_files
913                 .  When a path to an input file or  directory  is  specified,
914                 this  specifies  the  path  to the file on the submit side. A
915                 single wild card character (*) may be used in each file name.
916                 The  method  of encryption utilized will be as agreed upon in
917                 security negotiation; if that negotiation  failed,  then  the
918                 file  transfer  mechanism must also fail for files to be net‐
919                 work encrypted.
920
921
922
923          encrypt_output_files = < file1,file2,file... >
924                 A comma and/or space separated list of output files that  are
925                 to  be  network encrypted when transferred back with the file
926                 transfer mechanism. Specification of  files  in  this  manner
927                 overrides  configuration  that  would not use encryption. The
928                 output file(s) must also either  be  in  the  list  given  by
929                 transfer_output_files  or be discovered and to be transferred
930                 back with the file transfer mechanism. When a path to an out‐
931                 put  file  or directory is specified, this specifies the path
932                 to the file on the execute side. A single wild card character
933                 (*)  may  be used in each file name. The method of encryption
934                 utilized will be as agreed upon in security  negotiation;  if
935                 that  negotiation  failed,  then  the file transfer mechanism
936                 must also fail for files to be network encrypted.
937
938
939          max_transfer_input_mb = <ClassAd Integer Expression>
940                 This integer expression specifies the maximum  allowed  total
941                 size  in  MiB  of  the input files that are transferred for a
942                 job. This expression does not apply to grid  universe,  stan‐
943                 dard   universe,  or  files  transferred  via  file  transfer
944                 plug-ins. The expression may refer to attributes of the  job.
945                 The  special value -1 indicates no limit. If not defined, the
946                 value set  by  configuration  variable  MAX_TRANSFER_INPUT_MB
947                 is  used.  If  the observed size of all input files at submit
948                 time is larger than the limit, the job  will  be  immediately
949                 placed  on hold with a HoldReasonCode value of 32. If the job
950                 passes this initial test, but the size of the input files in‐
951                 creases or the limit decreases so that the limit is violated,
952                 the job will be placed on hold at  the  time  when  the  file
953                 transfer is attempted.
954
955
956          max_transfer_output_mb = <ClassAd Integer Expression>
957                 This  integer  expression specifies the maximum allowed total
958                 size in MiB of the output files that are  transferred  for  a
959                 job.  This  expression does not apply to grid universe, stan‐
960                 dard  universe,  or  files  transferred  via  file   transfer
961                 plug-ins.  The expression may refer to attributes of the job.
962                 The special value -1 indicates no  limit.  If  not  set,  the
963                 value  set  by  configuration variable MAX_TRANSFER_OUTPUT_MB
964                 is used. If the total size of the job's output  files  to  be
965                 transferred  is larger than the limit, the job will be placed
966                 on hold with a HoldReasonCode value of 33. The output will be
967                 transferred  up  to  the point when the limit is hit, so some
968                 files may be fully transferred, some partially, and some  not
969                 at all.
970
971
972
973          output_destination = <destination-URL>
974                 When present, defines a URL that specifies both a plug-in and
975                 a destination for the transfer of the entire  output  sandbox
976                 or  a  subset of output files as specified by the submit com‐
977                 mand transfer_output_files  .  The plug-in does the  transfer
978                 of  files,  and no files are sent back to the submit machine.
979                 The HTCondor Administrator's manual has full details.
980
981
982          should_transfer_files = <YES | NO | IF_NEEDED >
983                 The should_transfer_files setting is used to define if HTCon‐
984                 dor  should  transfer  files  to  and from the remote machine
985                 where the job runs. The file transfer mechanism  is  used  to
986                 run  jobs  which  are  not  in the standard universe (and can
987                 therefore use remote system calls for  file  access)  on  ma‐
988                 chines which do not have a shared file system with the submit
989                 machine.  should_transfer_files equal to YES will  cause  HT‐
990                 Condor  to always transfer files for the job. NO disables HT‐
991                 Condor's file transfer mechanism. IF_NEEDED will not transfer
992                 files  for  the  job  if it is matched with a resource in the
993                 same FileSystemDomain as the submit machine  (and  therefore,
994                 on a machine with the same shared file system). If the job is
995                 matched with a remote resource in a  different  FileSystemDo‐
996                 main, HTCondor will transfer the necessary files.
997
998                 For more information about this and other settings related to
999                 transferring files, see the HTCondor User's manual section on
1000                 the file transfer mechanism.
1001
1002                 Note  that  should_transfer_files  is  not supported for jobs
1003                 submitted to the grid universe.
1004
1005
1006          skip_filechecks = <True | False>
1007                 When True, file permission checks for the submitted  job  are
1008                 disabled.  When  False, file permissions are checked; this is
1009                 the behavior when this command is not present in  the  submit
1010                 description  file. File permissions are checked for read per‐
1011                 missions on all input files, such as those  defined  by  com‐
1012                 mands  input   and transfer_input_files  , and for write per‐
1013                 mission to output files, such as a log file  defined  by  log
1014                 and  output  files  defined  with  output    or transfer_out‐
1015                 put_files  .
1016
1017
1018          stream_error = <True | False>
1019                 If True, then stderr is streamed back  to  the  machine  from
1020                 which  the  job was submitted. If False, stderr is stored lo‐
1021                 cally and transferred back when the job completes. This  com‐
1022                 mand  is  ignored if the job ClassAd attribute TransferErr is
1023                 False.  The default value is False. This command must be used
1024                 in  conjunction  with  error  , otherwise stderr will sent to
1025                 /dev/null on Unix machines and ignored on Windows machines.
1026
1027
1028          stream_input = <True | False>
1029                 If True, then stdin is streamed from the machine on which the
1030                 job was submitted. The default value is False. The command is
1031                 only relevant for jobs submitted to the vanilla or java  uni‐
1032                 verses,  and it is ignored by the grid universe. This command
1033                 must be used in conjunction with  input   ,  otherwise  stdin
1034                 will be /dev/null on Unix machines and ignored on Windows ma‐
1035                 chines.
1036
1037          stream_output = <True | False>
1038                 If True, then stdout is streamed back  to  the  machine  from
1039                 which  the  job was submitted. If False, stdout is stored lo‐
1040                 cally and transferred back when the job completes. This  com‐
1041                 mand  is  ignored if the job ClassAd attribute TransferOut is
1042                 False.  The default value is False. This command must be used
1043                 in  conjunction  with output  , otherwise stdout will sent to
1044                 /dev/null on Unix machines and ignored on Windows machines.
1045
1046
1047          transfer_executable = <True | False>
1048                 This command is applicable to jobs submitted to the grid  and
1049                 vanilla  universes.  If  transfer_executable is set to False,
1050                 then HTCondor looks for the executable on the remote machine,
1051                 and does not transfer the executable over. This is useful for
1052                 an already pre-staged executable; HTCondor behaves more  like
1053                 rsh. The default value is True.
1054
1055
1056          transfer_input_files = < file1,file2,file... >
1057                 A comma-delimited list of all the files and directories to be
1058                 transferred into the working directory for  the  job,  before
1059                 the job is started. By default, the file specified in the ex‐
1060                 ecutable   command and any file specified in the input   com‐
1061                 mand (for example, stdin) are transferred.
1062
1063                 When  a path to an input file or directory is specified, this
1064                 specifies the path to the file on the submit side.  The  file
1065                 is placed in the job's temporary scratch directory on the ex‐
1066                 ecute side, and it is named using the base name of the origi‐
1067                 nal path. For example, /path/to/input_file becomes input_file
1068                 in the job's scratch directory.
1069
1070                 A directory may be specified by appending the  forward  slash
1071                 character  (/)  as  a trailing path separator. This syntax is
1072                 used for both Windows and Linux submit hosts. A directory ex‐
1073                 ample  using a trailing path separator is input_data/. When a
1074                 directory is specified with the trailing path separator,  the
1075                 contents  of the directory are transferred, but the directory
1076                 itself is not transferred. It is as  if  each  of  the  items
1077                 within  the  directory were listed in the transfer list. When
1078                 there is no trailing path separator, the directory is  trans‐
1079                 ferred,  its contents are transferred, and these contents are
1080                 placed inside the transferred directory.
1081
1082                 For grid universe jobs other than HTCondor-C, the transfer of
1083                 directories is not currently supported.
1084
1085                 Symbolic  links  to  files  are transferred as the files they
1086                 point to.  Transfer of symbolic links to directories  is  not
1087                 currently supported.
1088
1089                 For  vanilla  and vm universe jobs only, a file may be speci‐
1090                 fied by giving a URL, instead of a file name. The implementa‐
1091                 tion for URL transfers requires both configuration and avail‐
1092                 able plug-in.
1093
1094
1095          transfer_output_files = < file1,file2,file... >
1096                 This command forms an explicit list of output files  and  di‐
1097                 rectories  to  be transferred back from the temporary working
1098                 directory on the execute machine to the  submit  machine.  If
1099                 there are multiple files, they must be delimited with commas.
1100                 Setting transfer_output_files to the empty string ("")  means
1101                 that no files are to be transferred.
1102
1103                 For  HTCondor-C jobs and all other non-grid universe jobs, if
1104                 transfer_output_files is not specified, HTCondor  will  auto‐
1105                 matically  transfer  back  all  files  in the job's temporary
1106                 working directory which have been modified or created by  the
1107                 job.  Subdirectories are not scanned for output, so if output
1108                 from subdirectories is desired, the output list must  be  ex‐
1109                 plicitly  specified. For grid universe jobs other than HTCon‐
1110                 dor-C, desired output files must also be  explicitly  listed.
1111                 Another  reason  to explicitly list output files is for a job
1112                 that creates many files, and the user  wants  only  a  subset
1113                 transferred back.
1114
1115                 For  grid  universe jobs other than with grid type condor, to
1116                 have files other than  standard  output  and  standard  error
1117                 transferred  from  the execute machine back to the submit ma‐
1118                 chine, do use transfer_output_files, listing all files to  be
1119                 transferred.  These files are found on the execute machine in
1120                 the working directory of the job.
1121
1122                 When a path to an output file or directory is  specified,  it
1123                 specifies the path to the file on the execute side. As a des‐
1124                 tination on the submit side, the file is placed in the  job's
1125                 initial  working  directory,  and  it is named using the base
1126                 name of the original path.  For example,  path/to/output_file
1127                 becomes  output_file  in the job's initial working directory.
1128                 The name and path of the file that is written on  the  submit
1129                 side may be modified by using transfer_output_remaps  .  Note
1130                 that this remap function only works with files but  not  with
1131                 directories.
1132
1133                 A directory may be specified using a trailing path separator.
1134                 An example of a trailing path separator is the slash  charac‐
1135                 ter  on  Unix platforms; a directory example using a trailing
1136                 path separator is input_data/. When a directory is  specified
1137                 with a trailing path separator, the contents of the directory
1138                 are transferred, but the directory itself is not transferred.
1139                 It  is  as  if  each  of  the items within the directory were
1140                 listed in the transfer list. When there is no  trailing  path
1141                 separator,  the  directory  is  transferred, its contents are
1142                 transferred, and these contents are placed inside the  trans‐
1143                 ferred directory.
1144
1145                 For grid universe jobs other than HTCondor-C, the transfer of
1146                 directories is not currently supported.
1147
1148                 Symbolic links to files are transferred  as  the  files  they
1149                 point  to.   Transfer of symbolic links to directories is not
1150                 currently supported.
1151
1152          transfer_output_remaps = <  name = newname ; name2 = newname2 ...  >
1153                 This specifies the name (and optionally  path)  to  use  when
1154                 downloading  output  files  from the completed job. Normally,
1155                 output files are transferred back to the initial working  di‐
1156                 rectory  with  the same name they had in the execution direc‐
1157                 tory. This gives you the option to save them with a different
1158                 path  or name. If you specify a relative path, the final path
1159                 will be relative to the job's initial working directory.
1160
1161                 name describes an output file name produced by your job,  and
1162                 newname  describes  the file name it should be downloaded to.
1163                 Multiple remaps can be specified by separating  each  with  a
1164                 semicolon.  If  you  wish  to  remap  file names that contain
1165                 equals signs or semicolons, these special characters  may  be
1166                 escaped  with  a backslash. You cannot specify directories to
1167                 be remapped.
1168
1169                 Note that whether an output file is transferred is controlled
1170                 by  transfer_output_files.  Listing  a  file in transfer_out‐
1171                 put_remaps is not sufficient to cause it to be transferred.
1172
1173
1174          when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT >
1175                 Setting when_to_transfer_output equal to ON_EXIT  will  cause
1176                 HTCondor  to transfer the job's output files back to the sub‐
1177                 mitting machine only when the job  completes  (exits  on  its
1178                 own).
1179
1180                 The  ON_EXIT_OR_EVICT  option  is intended for fault tolerant
1181                 jobs which periodically save their own state and can  restart
1182                 where  they  left off. In this case, files are spooled to the
1183                 submit machine any time the job leaves a remote site,  either
1184                 because  it exited on its own, or was evicted by the HTCondor
1185                 system for any reason prior  to  job  completion.  The  files
1186                 spooled  back  are placed in a directory defined by the value
1187                 of the SPOOL configuration variable. Any output files  trans‐
1188                 ferred back to the submit machine are automatically sent back
1189                 out again as input files if the job restarts.
1190
1191       POLICY COMMANDS
1192
1193          max_retries = <integer>
1194                 The maximum number of retries allowed for this job  (must  be
1195                 non-negative).  If the job fails (does not exit with the suc‐
1196                 cess_exit_code exit code) it will be retried  up  to  max_re‐
1197                 tries  times  (unless  retries  are  ceased  because  of  the
1198                 retry_until command). If max_retries is not defined, and  ei‐
1199                 ther  retry_until  or  success_exit_code is, the value of DE‐
1200                 FAULT_JOB_MAX_RETRIES will be used for the maximum number  of
1201                 retries.
1202
1203                 The  combination  of  the  max_retries, retry_until, and suc‐
1204                 cess_exit_code commands causes  an  appropriate  OnExitRemove
1205                 expression to be automatically generated. If retry command(s)
1206                 and on_exit_remove are both defined, the OnExitRemove expres‐
1207                 sion  will be generated by OR'ing the expression specified in
1208                 OnExitRemove and the expression generated by the  retry  com‐
1209                 mands.
1210
1211
1212          retry_until <Integer | ClassAd Boolean Expression>
1213                 An  integer value or boolean expression that prevents further
1214                 retries from taking place, even if max_retries have not  been
1215                 exhausted.   If  retry_until  is  an integer, the job exiting
1216                 with that exit code will cause retries to cease. If retry_un‐
1217                 til  is  a  ClassAd  expression, the expression evaluating to
1218                 True will cause retries to cease.
1219
1220          success_exit_code = <integer>
1221                 The exit code that is considered successful for this job. De‐
1222                 faults to 0 if not defined.
1223
1224                 Note:  non-zero  values of success_exit_code should generally
1225                 not be used for DAG node jobs.  At  the  present  time,  con‐
1226                 dor_dagman  does  not  take  into  account  the value of suc‐
1227                 cess_exit_code. This means that, if success_exit_code is  set
1228                 to  a  non-zero  value,  condor_dagman  will consider the job
1229                 failed when it actually succeeds. For  single-proc  DAG  node
1230                 jobs,  this can be overcome by using a POST script that takes
1231                 into account the value of success_exit_code (although this is
1232                 not recommended). For multi-proc DAG node jobs, there is cur‐
1233                 rently no way to overcome this limitation.
1234
1235
1236          hold = <True | False>
1237                 If hold is set to True, then the submitted job will be placed
1238                 into  the Hold state. Jobs in the Hold state will not run un‐
1239                 til released by condor_release. Defaults to False.
1240
1241
1242          keep_claim_idle = <integer>
1243                 An integer number of seconds that a  job  requests  the  con‐
1244                 dor_schedd  to  wait before releasing its claim after the job
1245                 exits or after the job is removed.
1246
1247                 The process by which the condor_schedd claims a condor_startd
1248                 is  somewhat  time-consuming. To amortize this cost, the con‐
1249                 dor_schedd tries to reuse claims to run subsequent jobs,  af‐
1250                 ter a job using a claim is done. However, it can only do this
1251                 if there is an idle job in the queue at the moment the previ‐
1252                 ous  job  completes.  Sometimes,  and especially for the node
1253                 jobs when using DAGMan, there is a subsequent job about to be
1254                 submitted,  but  it has not yet arrived in the queue when the
1255                 previous job completes. As a result,  the  condor_schedd  re‐
1256                 leases  the claim, and the next job must wait an entire nego‐
1257                 tiation cycle to start. When this submit command  is  defined
1258                 with  a  non-negative  integer,  when the job exits, the con‐
1259                 dor_schedd tries as usual to reuse the claim. If  it  cannot,
1260                 instead  of  releasing the claim, the condor_schedd keeps the
1261                 claim until either the number of seconds given as  a  parame‐
1262                 ter, or a new job which matches that claim arrives, whichever
1263                 comes first. The condor_startd in question will remain in the
1264                 Claimed/Idle  state,  and  the original job will be "charged"
1265                 (in terms of priority) for the time in this state.
1266
1267
1268          leave_in_queue = <ClassAd Boolean Expression>
1269                 When the ClassAd Expression evaluates to True, the job is not
1270                 removed  from the queue upon completion. This allows the user
1271                 of a remotely spooled job to retrieve output files  in  cases
1272                 where HTCondor would have removed them as part of the cleanup
1273                 associated with completion. The job will only exit the  queue
1274                 once it has been marked for removal (via condor_rm, for exam‐
1275                 ple) and the  leave_in_queue  expression  has  become  False.
1276                 leave_in_queue defaults to False.
1277
1278                 As an example, if the job is to be removed once the output is
1279                 retrieved with condor_transfer_data, then use
1280
1281                     leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED) ||\
1282                                      (StageOutFinish == 0))
1283
1284
1285
1286          next_job_start_delay = <ClassAd Boolean Expression>
1287                 This expression specifies the number of seconds to delay  af‐
1288                 ter  starting up this job before the next job is started. The
1289                 maximum allowed delay is specified by the HTCondor configura‐
1290                 tion variable MAX_NEXT_JOB_START_DELAY
1291                  ,  which defaults to 10 minutes. This command does not apply
1292                 to scheduler or local universe jobs.
1293
1294                 This command has been historically used to implement  a  form
1295                 of job start throttling from the job submitter's perspective.
1296                 It was effective for the  case  of  multiple  job  submission
1297                 where  the transfer of extremely large input data sets to the
1298                 execute machine caused machine performance  to  suffer.  This
1299                 command  is  no longer useful, as throttling should be accom‐
1300                 plished through configuration of the condor_schedd daemon.
1301
1302
1303          on_exit_hold = <ClassAd Boolean Expression>
1304                 The ClassAd expression is checked when the job exits, and  if
1305                 True,  places  the job into the Hold state. If False (the de‐
1306                 fault value when not defined), then nothing happens  and  the
1307                 on_exit_remove  expression  is  checked  to determine if that
1308                 needs to be applied.
1309
1310                 For example: Suppose a job is known to run for a  minimum  of
1311                 an  hour.   If the job exits after less than an hour, the job
1312                 should be placed on hold and an e-mail notification sent, in‐
1313                 stead of being allowed to leave the queue.
1314
1315                     on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
1316
1317                 This  expression  places  the job on hold if it exits for any
1318                 reason before running for an hour. An e-mail will be sent  to
1319                 the  user  explaining that the job was placed on hold because
1320                 this expression became True.
1321
1322                 periodic_* expressions take precedence over on_exit_* expres‐
1323                 sions, and *_hold expressions take precedence over a *_remove
1324                 expressions.
1325
1326                 Only job ClassAd attributes will be defined for use  by  this
1327                 ClassAd  expression.  This  expression  is  available for the
1328                 vanilla, java, parallel, grid, local and scheduler universes.
1329                 It  is additionally available, when submitted from a Unix ma‐
1330                 chine, for the standard universe.
1331
1332          on_exit_hold_reason = <ClassAd String Expression>
1333                 When the job is placed on hold due to  the  on_exit_hold  ex‐
1334                 pression  becoming  True, this expression is evaluated to set
1335                 the value of HoldReason in the job ClassAd. If  this  expres‐
1336                 sion  is  UNDEFINED or produces an empty or invalid string, a
1337                 default description is used.
1338
1339
1340          on_exit_hold_subcode = <ClassAd Integer Expression>
1341                 When the job is placed on hold due to  the  on_exit_hold  ex‐
1342                 pression  becoming  True, this expression is evaluated to set
1343                 the value of HoldReasonSubCode in the job  ClassAd.  The  de‐
1344                 fault  subcode  is  0.  The  HoldReasonCode will be set to 3,
1345                 which indicates that the job went on hold due to a job policy
1346                 expression.
1347
1348
1349          on_exit_remove = <ClassAd Boolean Expression>
1350                 The  ClassAd expression is checked when the job exits, and if
1351                 True (the default value when undefined), then it  allows  the
1352                 job  to  leave  the queue normally. If False, then the job is
1353                 placed back into the Idle state. If the user job  runs  under
1354                 the  vanilla  universe, then the job restarts from the begin‐
1355                 ning. If the user job runs under the standard universe,  then
1356                 it  continues  from  where it left off, using the last check‐
1357                 point.
1358
1359                 For  example,  suppose  a  job  occasionally  segfaults,  but
1360                 chances  are that the job will finish successfully if the job
1361                 is run again with the same data. The  on_exit_remove  expres‐
1362                 sion  can  cause the job to run again with the following com‐
1363                 mand. Assume that the signal identifier for the  segmentation
1364                 fault is 11 on the platform where the job will be running.
1365
1366                     on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)
1367
1368                 This  expression  lets the job leave the queue if the job was
1369                 not killed by a signal or if it was killed by a signal  other
1370                 than 11, representing segmentation fault in this example. So,
1371                 if the exited due to signal 11,  it  will  stay  in  the  job
1372                 queue.  In  any  other  case of the job exiting, the job will
1373                 leave the queue as it normally would have done.
1374
1375                 As another example, if the job should only leave the queue if
1376                 it  exited  on its own with status 0, this on_exit_remove ex‐
1377                 pression works well:
1378
1379                     on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
1380
1381                 If the job was killed by a signal or exited with  a  non-zero
1382                 exit status, HTCondor would leave the job in the queue to run
1383                 again.
1384
1385                 periodic_* expressions take precedence over on_exit_* expres‐
1386                 sions, and *_hold expressions take precedence over a *_remove
1387                 expressions.
1388
1389                 Only job ClassAd attributes will be defined for use  by  this
1390                 ClassAd expression.
1391
1392          periodic_hold = <ClassAd Boolean Expression>
1393                 This  expression  is checked periodically when the job is not
1394                 in the Held state. If it becomes True, the job will be placed
1395                 on hold.  If unspecified, the default value is False.
1396
1397                 periodic_* expressions take precedence over on_exit_* expres‐
1398                 sions, and *_hold expressions take precedence over a *_remove
1399                 expressions.
1400
1401                 Only  job  ClassAd attributes will be defined for use by this
1402                 ClassAd expression. Note that, by default, this expression is
1403                 only checked once every 60 seconds. The period of these eval‐
1404                 uations can be adjusted by setting  the  PERIODIC_EXPR_INTER‐
1405                 VAL,  MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1406                 configuration macros.
1407
1408
1409          periodic_hold_reason = <ClassAd String Expression>
1410                 When the job is placed on hold due to the  periodic_hold  ex‐
1411                 pression  becoming  True, this expression is evaluated to set
1412                 the value of HoldReason in the job ClassAd. If  this  expres‐
1413                 sion  is  UNDEFINED or produces an empty or invalid string, a
1414                 default description is used.
1415
1416
1417          periodic_hold_subcode = <ClassAd Integer Expression>
1418                 When the job is placed on hold due to the  periodic_hold  ex‐
1419                 pression  becoming  true, this expression is evaluated to set
1420                 the value of HoldReasonSubCode in the job  ClassAd.  The  de‐
1421                 fault  subcode  is  0.  The  HoldReasonCode will be set to 3,
1422                 which indicates that the job went on hold due to a job policy
1423                 expression.
1424
1425
1426          periodic_release = <ClassAd Boolean Expression>
1427                 This  expression  is  checked periodically when the job is in
1428                 the Held state. If the expression becomes True, the job  will
1429                 be released.
1430
1431                 Only  job  ClassAd attributes will be defined for use by this
1432                 ClassAd expression. Note that, by default, this expression is
1433                 only checked once every 60 seconds. The period of these eval‐
1434                 uations can be adjusted by setting  the  PERIODIC_EXPR_INTER‐
1435                 VAL,  MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1436                 configuration macros.
1437
1438
1439          periodic_remove = <ClassAd Boolean Expression>
1440                 This expression is checked periodically. If it becomes  True,
1441                 the  job  is  removed from the queue. If unspecified, the de‐
1442                 fault value is False.
1443
1444                 See the Examples section of this manual page for  an  example
1445                 of a periodic_remove expression.
1446
1447                 periodic_* expressions take precedence over on_exit_* expres‐
1448                 sions, and *_hold expressions take precedence over a *_remove
1449                 expressions.  So, the periodic_remove expression takes prece‐
1450                 dent over the on_exit_remove expression, if the two  describe
1451                 conflicting actions.
1452
1453                 Only  job  ClassAd attributes will be defined for use by this
1454                 ClassAd expression. Note that, by default, this expression is
1455                 only checked once every 60 seconds. The period of these eval‐
1456                 uations can be adjusted by setting  the  PERIODIC_EXPR_INTER‐
1457                 VAL,  MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE
1458                 configuration macros.
1459
1460       COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
1461
1462
1463          allow_startup_script = <True | False>
1464                 If True, a standard universe job will execute  a  script  in‐
1465                 stead of submitting the job, and the consistency check to see
1466                 if the executable has been  linked  using  condor_compile  is
1467                 omitted.  The executable   command within the submit descrip‐
1468                 tion file specifies the name of the script.   The  script  is
1469                 used  to  do  preprocessing before the job is submitted.  The
1470                 shell script ends with an exec of the  job  executable,  such
1471                 that  the process id of the executable is the same as that of
1472                 the shell script. Here is an example script that gets a  copy
1473                 of a machine-specific executable before the exec.
1474
1475                     #! /bin/sh
1476
1477                     # get the host name of the machine
1478                     $host=`uname -n`
1479
1480                     # grab a standard universe executable designed specifically
1481                     # for this host
1482                     scp elsewhere@cs.wisc.edu:${host} executable
1483
1484                     # The PID MUST stay the same, so exec the new standard universe process.
1485                     exec executable ${1+"$@"}
1486
1487                 If  this command is not present (defined), then the value de‐
1488                 faults to false.
1489
1490          append_files = file1, file2, ...
1491                 If your job attempts to access a file mentioned in this list,
1492                 HTCondor will force all writes to that file to be appended to
1493                 the end. Furthermore, condor_submit  will  not  truncate  it.
1494                 This  list  uses  the  same  syntax  as compress_files, shown
1495                 above.
1496
1497                 This option may yield some  surprising  results.  If  several
1498                 jobs  attempt  to write to the same file, their output may be
1499                 intermixed.  If a job is evicted from one  or  more  machines
1500                 during  the course of its lifetime, such an output file might
1501                 contain several copies of the results. This option should  be
1502                 only  be used when you wish a certain file to be treated as a
1503                 running log instead of a precise result.
1504
1505                 This option only applies to standard-universe jobs.
1506
1507
1508
1509
1510          buffer_files   =   <    name   =   (size,block-size)   ;   name2   =
1511          (size,block-size)  ...   >;  buffer_size  =  <bytes-in-buffer>; buf‐
1512          fer_block_size = <bytes-in-block>
1513                 HTCondor keeps a buffer of recently-used data for each file a
1514                 job accesses. This buffer is used both to cache commonly-used
1515                 data and to consolidate small reads and  writes  into  larger
1516                 operations  that  get better throughput. The default settings
1517                 should produce reasonable results for most programs.
1518
1519                 These options only apply to standard-universe jobs.
1520
1521                 If needed, you may set the buffer controls  individually  for
1522                 each  file using the buffer_files option. For example, to set
1523                 the buffer size to 1 MiB and the block size to  256  KiB  for
1524                 the file input.data, use this command:
1525
1526                     buffer_files = "input.data=(1000000,256000)"
1527
1528                 Alternatively,  you  may use these two options to set the de‐
1529                 fault sizes for all files used by your job:
1530
1531                     buffer_size = 1000000
1532                     buffer_block_size = 256000
1533
1534                 If you do not set these, HTCondor will use the  values  given
1535                 by these two configuration file macros:
1536
1537                     DEFAULT_IO_BUFFER_SIZE = 1000000
1538                     DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
1539
1540                 Finally,  if no other settings are present, HTCondor will use
1541                 a buffer of 512 KiB and a block size of 32 KiB.
1542
1543
1544          compress_files = file1, file2, ...
1545                 If your job attempts to access any of the files mentioned  in
1546                 this  list,  HTCondor  will  automatically  compress them (if
1547                 writing) or decompress them (if reading). The compress format
1548                 is the same as used by GNU gzip.
1549
1550                 The files given in this list may be simple file names or com‐
1551                 plete paths and may include * as a wild  card.  For  example,
1552                 this  list  causes  the  file  /tmp/data.gz,  any  file named
1553                 event.gz, and any file ending in .gzip  to  be  automatically
1554                 compressed or decompressed as needed:
1555
1556                     compress_files = /tmp/data.gz, event.gz, *.gzip
1557
1558                 Due to the nature of the compression format, compressed files
1559                 must only be accessed sequentially. Random access reading  is
1560                 allowed but is very slow, while random access writing is sim‐
1561                 ply not possible.  This restriction may be avoided  by  using
1562                 both  compress_files  and  fetch_files at the same time. When
1563                 this is done, a file is kept in the decompressed state at the
1564                 execution  machine,  but  is  compressed  for transfer to its
1565                 original location.
1566
1567                 This option only applies to standard universe jobs.
1568
1569
1570          fetch_files = file1, file2, ...
1571                 If your job attempts to access a file mentioned in this list,
1572                 HTCondor  will  automatically copy the whole file to the exe‐
1573                 cuting machine, where it can be accessed quickly.  When  your
1574                 job  closes  the file, it will be copied back to its original
1575                 location. This list uses the same syntax  as  compress_files,
1576                 shown above.
1577
1578                 This option only applies to standard universe jobs.
1579
1580
1581          file_remaps = <  name = newname ; name2 = newname2 ...  >
1582                 Directs  HTCondor  to  use a new file name in place of an old
1583                 one.  name describes a file name that your job may attempt to
1584                 open,  and  newname  describes the file name it should be re‐
1585                 placed with.  newname may include an optional leading  access
1586                 specifier,  local:  or  remote:. If left unspecified, the de‐
1587                 fault access specifier is remote:.  Multiple  remaps  can  be
1588                 specified by separating each with a semicolon.
1589
1590                 This option only applies to standard universe jobs.
1591
1592                 If  you wish to remap file names that contain equals signs or
1593                 semicolons, these special characters may be  escaped  with  a
1594                 backslash.
1595
1596                     Example One:
1597                            Suppose   that   your   job  reads  a  file  named
1598                            dataset.1. To instruct HTCondor to force your  job
1599                            to  read  other.dataset  instead,  add this to the
1600                            submit file:
1601
1602                               file_remaps = "dataset.1=other.dataset"
1603
1604                     Example Two:
1605                            Suppose that your run many jobs which all read  in
1606                            the same large file, called very.big. If this file
1607                            can be found in the same place on a local disk  in
1608                            every machine in the pool, (say /bigdisk/bigfile,)
1609                            you can instruct HTCondor of this fact  by  remap‐
1610                            ping  very.big  to /bigdisk/bigfile and specifying
1611                            that the file is to be read locally, which will be
1612                            much faster than reading over the network.
1613
1614                               file_remaps = "very.big = local:/bigdisk/bigfile"
1615
1616                     Example Three:
1617                            Several  remaps can be applied at once by separat‐
1618                            ing each with a semicolon.
1619
1620                               file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 = other.dataset"
1621
1622
1623
1624          local_files = file1, file2, ...
1625                 If your job attempts to access a file mentioned in this list,
1626                 HTCondor will cause it to be read or written at the execution
1627                 machine. This is most useful for temporary files not used for
1628                 input  or  output.  This  list  uses  the same syntax as com‐
1629                 press_files, shown above.
1630
1631                     local_files = /tmp/*
1632
1633                 This option only applies to standard universe jobs.
1634
1635
1636          want_remote_io = <True | False>
1637                 This option controls how a file is opened and manipulated  in
1638                 a standard universe job. If this option is true, which is the
1639                 default, then the condor_shadow makes all decisions about how
1640                 each  and  every  file should be opened by the executing job.
1641                 This entails a network round trip (or more) from the  job  to
1642                 the  condor_shadow  and back again for every single open() in
1643                 addition to other needed information about the file.  If  set
1644                 to false, then when the job queries the condor_shadow for the
1645                 first time about how to open a file, the  condor_shadow  will
1646                 inform  the  job to automatically perform all of its file ma‐
1647                 nipulation on the local file system on  the  execute  machine
1648                 and any file remapping will be ignored. This means that there
1649                 must be a shared file system (such as NFS or AFS) between the
1650                 execute  machine  and  the  submit machine and that ALL paths
1651                 that the job could open on the execute machine must be valid.
1652                 The  ability of the standard universe job to checkpoint, pos‐
1653                 sibly to a checkpoint server, is not affected by this  attri‐
1654                 bute.  However, when the job resumes it will be expecting the
1655                 same file system conditions that were present  when  the  job
1656                 checkpointed.
1657
1658       COMMANDS FOR THE GRID
1659
1660          azure_admin_key = <pathname>
1661                 For grid type azure jobs, specifies the path and file name of
1662                 a file that contains an SSH public key. This key can be  used
1663                 to  log  into  the  administrator account of the instance via
1664                 SSH.
1665
1666
1667          azure_admin_username = <account name>
1668                 For grid type azure jobs, specifies the name of  an  adminis‐
1669                 trator  account  to  be created in the instance. This account
1670                 can be logged into via SSH.
1671
1672          azure_auth_file = <pathname>
1673                 For grid type azure jobs, specifies a path and file  name  of
1674                 the authorization file that grants permission for HTCondor to
1675                 use the Azure account. If it's  not  defined,  then  HTCondor
1676                 will  attempt to use the default credentials of the Azure CLI
1677                 tools.
1678
1679
1680          azure_image = <image id>
1681                 For grid type azure jobs, identifies the  disk  image  to  be
1682                 used  for  the boot disk of the instance. This image must al‐
1683                 ready be registered within Azure.
1684
1685
1686          azure_location = <image id>
1687                 For grid type azure  jobs,  identifies  the  location  within
1688                 Azure  where  the  instance should be run. As an example, one
1689                 current location is centralus.
1690
1691
1692          azure_size = <machine type>
1693                 For grid type azure jobs, the hardware configuration that the
1694                 virtual machine instance is to run on.
1695
1696
1697          batch_queue = <queuename>
1698                 Used  for pbs, lsf, and sge grid universe jobs. Specifies the
1699                 name of the PBS/LSF/SGE job queue into which the  job  should
1700                 be submitted. If not specified, the default queue is used.
1701
1702
1703          boinc_authenticator_file = <pathname>
1704                 For  grid  type boinc jobs, specifies a path and file name of
1705                 the authorization file that grants permission for HTCondor to
1706                 use  the  BOINC  service.  There is no default value when not
1707                 specified.
1708
1709
1710          cream_attributes = <name=value;...;name=value>
1711                 Provides a list of attribute/value pairs to be set in a CREAM
1712                 job description of a grid universe job destined for the CREAM
1713                 grid system. The pairs are separated by semicolons, and writ‐
1714                 ten in New ClassAd syntax.
1715
1716
1717          delegate_job_GSI_credentials_lifetime = <seconds>
1718                 Specifies  the  maximum number of seconds for which delegated
1719                 proxies should be valid. The default behavior when this  com‐
1720                 mand  is  not  specified  is  determined by the configuration
1721                 variable DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME
1722                  , which defaults to one day. A value of 0 indicates that the
1723                 delegated proxy should be valid for as long as allowed by the
1724                 credential used to create the proxy. This  setting  currently
1725                 only  applies  to proxies delegated for non-grid jobs and for
1726                 HTCondor-C jobs. It does not currently apply to  globus  grid
1727                 jobs, which always behave as though this setting were 0. This
1728                 variable has no effect if the  configuration  variable  DELE‐
1729                 GATE_JOB_GSI_CREDENTIALS
1730                   is  False,  because  in  that  case the job proxy is copied
1731                 rather than delegated.
1732
1733
1734          ec2_access_key_id = <pathname>
1735                 For grid type ec2 jobs, identifies the  file  containing  the
1736                 access key.
1737
1738          ec2_ami_id = <EC2 xMI ID>
1739                 For  grid  type  ec2 jobs, identifies the machine image. Ser‐
1740                 vices compatible with the EC2 Query API may  refer  to  these
1741                 with  abbreviations  other than AMI, for example EMI is valid
1742                 for Eucalyptus.
1743
1744          ec2_availability_zone = <zone name>
1745                 For grid type ec2 jobs, specifies the Availability Zone  that
1746                 the  instance should be run in. This command is optional, un‐
1747                 less ec2_ebs_volumes is set. As an example, one current  zone
1748                 is us-east-1b.
1749
1750
1751          ec2_block_device_mapping = <block-device>:<kernel-device>,<block-de‐
1752          vice>:<kernel-device>, ...
1753                 For grid type ec2 jobs, specifies the block device to  kernel
1754                 device mapping. This command is optional.
1755
1756
1757          ec2_ebs_volumes   =   <ebs  name>:<device  name>,<ebs  name>:<device
1758          name>,...
1759                 For grid type ec2 jobs, optionally specifies a list of  Elas‐
1760                 tic Block Store (EBS) volumes to be made available to the in‐
1761                 stance and the device names they should have in the instance.
1762
1763
1764          ec2_elastic_ip = <elastic IP address>
1765                 For grid type ec2 jobs,  and  optional  specification  of  an
1766                 Elastic IP address that should be assigned to this instance.
1767
1768
1769          ec2_iam_profile_arn = <IAM profile ARN>
1770                 For grid type ec2 jobs, an Amazon Resource Name (ARN) identi‐
1771                 fying which Identity and Access Management  (IAM)  (instance)
1772                 profile to associate with the instance.
1773
1774
1775          ec2_iam_profile_name= <IAM profile name>
1776                 For grid type ec2 jobs, a name identifying which Identity and
1777                 Access Management (IAM) (instance) profile to associate  with
1778                 the instance.
1779
1780          ec2_instance_type = <instance type>
1781                 For grid type ec2 jobs, identifies the instance type. Differ‐
1782                 ent services may offer different instance types,  so  no  de‐
1783                 fault value is set.
1784
1785          ec2_keypair = <ssh key-pair name>
1786                 For grid type ec2 jobs, specifies the name of an SSH key-pair
1787                 that is already registered with the EC2 service. The  associ‐
1788                 ated  private key can be used to ssh into the virtual machine
1789                 once it is running.
1790
1791          ec2_keypair_file = <pathname>
1792                 For grid type ec2 jobs, specifies the complete path and  file
1793                 name  of a file into which HTCondor will write an SSH key for
1794                 use with ec2 jobs. The key can be used to ssh into  the  vir‐
1795                 tual  machine  once it is running. If ec2_keypair   is speci‐
1796                 fied for a job, ec2_keypair_file is ignored.
1797
1798          ec2_parameter_names = ParameterName1, ParameterName2, ...
1799                 For grid type ec2 jobs, a space or comma  separated  list  of
1800                 the names of additional parameters to pass when instantiating
1801                 an instance.
1802
1803          ec2_parameter_<name> = <value>
1804                 For grid type ec2 jobs, specifies the value  for  the  corre‐
1805                 spondingly  named  (instance instantiation) parameter. <name>
1806                 is the parameter name specified in the submit command ec2_pa‐
1807                 rameter_names   ,  but  with  any  periods replaced by under‐
1808                 scores.
1809
1810
1811          ec2_secret_access_key = <pathname>
1812                 For grid type ec2 jobs, specifies the path and file name con‐
1813                 taining the secret access key.
1814
1815
1816          ec2_security_groups = group1, group2, ...
1817                 For  grid  type  ec2  jobs,  defines the list of EC2 security
1818                 groups which should be associated with the job.
1819
1820
1821          ec2_security_ids = id1, id2, ...
1822                 For grid type ec2 jobs, defines  the  list  of  EC2  security
1823                 group IDs which should be associated with the job.
1824
1825
1826          ec2_spot_price = <bid>
1827                 For  grid  type  ec2  jobs,  specifies the spot instance bid,
1828                 which is the most that the job submitter is  willing  to  pay
1829                 per hour to run this job.
1830
1831          ec2_tag_names = <name0,name1,name...>
1832                 For  grid type ec2 jobs, specifies the case of tag names that
1833                 will be associated with the running instance.  This  is  only
1834                 necessary  if  a  tag  name case matters. By default the list
1835                 will be automatically generated.
1836
1837
1838          ec2_tag_<name> = <value>
1839                 For grid type ec2 jobs, specifies a tag to be associated with
1840                 the  running  instance. The tag name will be lower-cased, use
1841                 ec2_tag_names to change the case.
1842
1843          WantNameTag = <True | False>
1844                 For grid type ec2 jobs, a job may request that its 'name' tag
1845                 be (not) set by HTCondor. If the job does not otherwise spec‐
1846                 ify any tags, not setting its name tag will eliminate a  call
1847                 by the EC2 GAHP, improving performance.
1848
1849
1850          ec2_user_data = <data>
1851                 For  grid type ec2 jobs, provides a block of data that can be
1852                 accessed by the virtual machine. If  both  ec2_user_data  and
1853                 ec2_user_data_file are specified for a job, the two blocks of
1854                 data are concatenated, with the data from this  ec2_user_data
1855                 submit command occurring first.
1856
1857          ec2_user_data_file = <pathname>
1858                 For  grid type ec2 jobs, specifies a path and file name whose
1859                 contents can be accessed by  the  virtual  machine.  If  both
1860                 ec2_user_data and ec2_user_data_file are specified for a job,
1861                 the two blocks of data are concatenated, with the  data  from
1862                 that ec2_user_data submit command occurring first.
1863
1864          ec2_vpc_ip = <a.b.c.d>
1865                 For  grid  type  ec2 jobs, that are part of a Virtual Private
1866                 Cloud (VPC), an optional specification of the IP address that
1867                 this instance should have within the VPC.
1868
1869
1870          ec2_vpc_subnet = <subnet specification string>
1871                 For grid type ec2 jobs, an optional specification of the Vir‐
1872                 tual Private Cloud (VPC) that this instance should be a  part
1873                 of.
1874
1875
1876          gce_account = <account name>
1877                 For  grid  type gce jobs, specifies the Google cloud services
1878                 account to use. If this submit command isn't specified,  then
1879                 a  random  account  from  the  authorization  file  given  by
1880                 gce_auth_file will be used.
1881
1882          gce_auth_file = <pathname>
1883                 For grid type gce jobs, specifies a path and file name of the
1884                 authorization file that grants permission for HTCondor to use
1885                 the Google account. If this command is  not  specified,  then
1886                 the  default  file  of  the Google command-line tools will be
1887                 used.
1888
1889
1890          gce_image = <image id>
1891                 For grid type gce jobs, the identifier of the virtual machine
1892                 image  representing  the HTCondor job to be run. This virtual
1893                 machine image must already be register with GCE and reside in
1894                 Google's Cloud Storage service.
1895
1896          gce_json_file = <pathname>
1897                 For grid type gce jobs, specifies the path and file name of a
1898                 file that contains JSON elements that should be added to  the
1899                 instance description submitted to the GCE service.
1900
1901
1902          gce_machine_type = <machine type>
1903                 For  grid  type  gce  jobs, the long form of the URL that de‐
1904                 scribes the machine configuration that  the  virtual  machine
1905                 instance is to run on.
1906
1907          gce_metadata = <name=value,...,name=value>
1908                 For  grid  type  gce jobs, a comma separated list of name and
1909                 value pairs that define metadata for a  virtual  machine  in‐
1910                 stance that is an HTCondor job.
1911
1912          gce_metadata_file = <pathname>
1913                 For grid type gce jobs, specifies a path and file name of the
1914                 file that contains metadata for a  virtual  machine  instance
1915                 that is an HTCondor job. Within the file, each name and value
1916                 pair is on its own line; so, the pairs are separated  by  the
1917                 newline character.
1918
1919
1920          gce_preemptible = <True | False>
1921                 For grid type gce jobs, specifies whether the virtual machine
1922                 instance should be preemptible. The default is  for  the  in‐
1923                 stance to not be preemptible.
1924
1925          globus_rematch = <ClassAd Boolean Expression>
1926                 This  expression is evaluated by the condor_gridmanager when‐
1927                 ever:
1928
1929                 1. the globus_resubmit expression evaluates to True
1930
1931                 2. the condor_gridmanager decides it needs to retry a submis‐
1932                    sion (as when a previous submission failed to commit)
1933
1934                 If  globus_rematch  evaluates to True, then before the job is
1935                 submitted again to globus, the  condor_gridmanager  will  re‐
1936                 quest  that  the  condor_schedd  daemon  renegotiate with the
1937                 matchmaker (the condor_negotiator). The result  is  this  job
1938                 will be matched again.
1939
1940
1941          globus_resubmit = <ClassAd Boolean Expression>
1942                 The  expression  is  evaluated by the condor_gridmanager each
1943                 time the condor_gridmanager gets a job ad to  manage.  There‐
1944                 fore, the expression is evaluated:
1945
1946                 1. when a grid universe job is first submitted to HTCondor-G
1947
1948                 2. when a grid universe job is released from the hold state
1949
1950                 3. when  HTCondor-G  is restarted (specifically, whenever the
1951                    condor_gridmanager is restarted)
1952
1953                 If the expression evaluates to True, then any  previous  sub‐
1954                 mission  to  the grid universe will be forgotten and this job
1955                 will be submitted again as a fresh  submission  to  the  grid
1956                 universe.  This may be useful if there is a desire to give up
1957                 on a previous submission and try again. Note  that  this  may
1958                 result  in  the same job running more than once. Do not treat
1959                 this operation lightly.
1960
1961
1962          globus_rsl = <RSL-string>
1963                 Used to provide any additional Globus RSL  string  attributes
1964                 which  are  not covered by other submit description file com‐
1965                 mands or job attributes. Used for grid universe  jobs,  where
1966                 the grid resource has a grid-type-string of gt2.
1967
1968
1969          grid_resource = <grid-type-string> <grid-specific-parameter-list>
1970                 For  each grid-type-string value, there are further type-spe‐
1971                 cific values that must  specified.  This  submit  description
1972                 file  command  allows  each  to be given in a space-separated
1973                 list. Allowable grid-type-string values  are  batch,  condor,
1974                 cream,  ec2, gt2, gt5, lsf, nordugrid, pbs, sge, and unicore.
1975                 The HTCondor manual chapter on Grid Computing details the va‐
1976                 riety of grid types.
1977
1978                 For  a grid-type-string of batch, the single parameter is the
1979                 name of the local batch system, and will be one of pbs,  lsf,
1980                 or sge.
1981
1982                 For  a grid-type-string of condor, the first parameter is the
1983                 name of the remote condor_schedd daemon. The second parameter
1984                 is  the  name  of  the pool to which the remote condor_schedd
1985                 daemon belongs.
1986
1987                 For a grid-type-string of cream, there are three  parameters.
1988                 The  first parameter is the web services address of the CREAM
1989                 server.  The second parameter is the name of the batch system
1990                 that  sits behind the CREAM server. The third parameter iden‐
1991                 tifies a site-specific queue within the batch system.
1992
1993                 For a grid-type-string of ec2, one additional parameter spec‐
1994                 ifies the EC2 URL.
1995
1996                 For  a  grid-type-string  of gt2, the single parameter is the
1997                 name of the pre-WS GRAM resource to be used.
1998
1999                 For a grid-type-string of gt5, the single  parameter  is  the
2000                 name  of  the  pre-WS  GRAM resource to be used, which is the
2001                 same as for the grid-type-string of gt2.
2002
2003                 For a grid-type-string of lsf, no additional  parameters  are
2004                 used.
2005
2006                 For  a grid-type-string of nordugrid, the single parameter is
2007                 the name of the NorduGrid resource to be used.
2008
2009                 For a grid-type-string of pbs, no additional  parameters  are
2010                 used.
2011
2012                 For  a  grid-type-string of sge, no additional parameters are
2013                 used.
2014
2015                 For a grid-type-string of unicore, the first parameter is the
2016                 name of the Unicore Usite to be used. The second parameter is
2017                 the name of the Unicore Vsite to be used.
2018
2019
2020          keystore_alias = <name>
2021                 A string to locate the certificate in a Java  keystore  file,
2022                 as used for a unicore job.
2023
2024
2025          keystore_file = <pathname>
2026                 The  complete  path  and  file name of the Java keystore file
2027                 containing the certificate to be used for a unicore job.
2028
2029
2030          keystore_passphrase_file = <pathname>
2031                 The complete path and file name to the  file  containing  the
2032                 passphrase  protecting  a  Java  keystore file containing the
2033                 certificate. Relevant for a unicore job.
2034
2035
2036          MyProxyCredentialName = <symbolic name>
2037                 The symbolic name that identifies a credential to the MyProxy
2038                 server.  This  symbolic name is set as the credential is ini‐
2039                 tially stored on the server (using myproxy-init).
2040
2041
2042          MyProxyHost = <host>:<port>
2043                 The Internet address of the host that is the MyProxy  server.
2044                 The  host  may  be  specified  by  either  a host name (as in
2045                 head.example.com) or an IP address (of the form 123.456.7.8).
2046                 The port number is an integer.
2047
2048
2049          MyProxyNewProxyLifetime = <number-of-minutes>
2050                 The  new  lifetime  (in minutes) of the proxy after it is re‐
2051                 freshed.
2052
2053
2054          MyProxyPassword = <password>
2055                 The password needed to refresh a credential  on  the  MyProxy
2056                 server.   This password is set when the user initially stores
2057                 credentials on the server (using myproxy-init). As an  alter‐
2058                 native  to  using  MyProxyPassword  in the submit description
2059                 file, the password may be specified as a command  line  argu‐
2060                 ment to condor_submit with the -password argument.
2061
2062          MyProxyRefreshThreshold = <number-of-seconds>
2063                 The  time  (in seconds) before the expiration of a proxy that
2064                 the proxy should be refreshed.  For  example,  if  MyProxyRe‐
2065                 freshThreshold is set to the value 600, the proxy will be re‐
2066                 freshed 10 minutes before it expires.
2067
2068          MyProxyServerDN = <credential subject>
2069                 A string that specifies the expected Distinguished Name (cre‐
2070                 dential  subject,  abbreviated  DN) of the MyProxy server. It
2071                 must be specified when the MyProxy server DN does not  follow
2072                 the conventional naming scheme of a host credential. This oc‐
2073                 curs, for example, when the MyProxy server DN begins  with  a
2074                 user credential.
2075
2076
2077          nordugrid_rsl = <RSL-string>
2078                 Used  to  provide  any additional RSL string attributes which
2079                 are not covered by regular submit  description  file  parame‐
2080                 ters.  Used  when  the universe is grid, and the type of grid
2081                 system is nordugrid.
2082
2083          transfer_error = <True | False>
2084                 For jobs submitted to the grid universe only. If  True,  then
2085                 the  error  output  (from stderr) from the job is transferred
2086                 from the remote machine back to the submit machine. The  name
2087                 of  the  file after transfer is given by the error   command.
2088                 If False, no transfer takes place (from the remote machine to
2089                 submit machine), and the name of the file is given by the er‐
2090                 ror   command. The default value is True.
2091
2092
2093          transfer_input = <True | False>
2094                 For jobs submitted to the grid universe only. If  True,  then
2095                 the  job  input (stdin) is transferred from the machine where
2096                 the job was submitted to the remote machine. The name of  the
2097                 file  that is transferred is given by the input   command. If
2098                 False, then the job's input is taken from a  pre-staged  file
2099                 on  the  remote machine, and the name of the file is given by
2100                 the input   command. The default value is True.
2101
2102                 For transferring files other  than  stdin,  see  transfer_in‐
2103                 put_files  .
2104
2105
2106          transfer_output = <True | False>
2107                 For  jobs  submitted to the grid universe only. If True, then
2108                 the output (from stdout) from the job is transferred from the
2109                 remote  machine  back  to the submit machine. The name of the
2110                 file after transfer is given  by  the  output    command.  If
2111                 False,  no  transfer  takes place (from the remote machine to
2112                 submit machine), and the name of the file  is  given  by  the
2113                 output   command. The default value is True.
2114
2115                 For  transferring  files other than stdout, see transfer_out‐
2116                 put_files  .
2117
2118
2119          use_x509userproxy = <True | False>
2120                 Set this command to True to indicate that the job requires an
2121                 X.509  user proxy. If x509userproxy is set, then that file is
2122                 used for the proxy. Otherwise, the proxy is looked for in the
2123                 standard  locations. If x509userproxy is set or if the job is
2124                 a grid universe job of grid type gt2,  gt5,  cream,  or  nor‐
2125                 dugrid,  then  the  value  of  use_x509userproxy is forced to
2126                 True. Defaults to False.
2127
2128
2129          x509userproxy = <full-pathname>
2130                 Used to override the default path name for  X.509  user  cer‐
2131                 tificates.   The  default  location  for X.509 proxies is the
2132                 /tmp directory, which is generally a local file system.  Set‐
2133                 ting this value would allow HTCondor to access the proxy in a
2134                 shared file system (for example, AFS). HTCondor will use  the
2135                 proxy  specified  in  the  submit  description file first. If
2136                 nothing is specified in the submit description file, it  will
2137                 use  the  environment variable X509_USER_PROXY. If that vari‐
2138                 able is not present, it will search in the default  location.
2139                 Note  that  proxies  are  only valid for a limited time. Con‐
2140                 dor_submit will not submit a job with an  expired  proxy,  it
2141                 will  return  an  error. Also, if the configuration parameter
2142                 CRED_MIN_TIME_LEFT is set to some number of seconds,  and  if
2143                 the proxy will expire before that many seconds, condor_submit
2144                 will  also  refuse  to  submit   the   job.   That   is,   if
2145                 CRED_MIN_TIME_LEFT is set to 60, condor_submit will refuse to
2146                 submit a job whose proxy will expire 60 seconds from the time
2147                 of submission.
2148
2149                 x509userproxy    is relevant when the universe is vanilla, or
2150                 when the universe is grid and the type of grid system is  one
2151                 of  gt2,  gt5,  condor, cream, or nordugrid. Defining a value
2152                 causes the proxy to be  delegated  to  the  execute  machine.
2153                 Further,  VOMS attributes defined in the proxy will appear in
2154                 the job ClassAd.
2155
2156       COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
2157
2158
2159          hold_kill_sig = <signal-number>
2160                 For the scheduler universe only, signal-number   is the  sig‐
2161                 nal  delivered  to  the  job when the job is put on hold with
2162                 condor_hold.  signal-number may be either  the  platform-spe‐
2163                 cific  name  or  value  of the signal. If this command is not
2164                 present, the value of kill_sig   is used.
2165
2166
2167          jar_files = <file_list>
2168                 Specifies a list of additional JAR files to include when  us‐
2169                 ing  the  Java  universe. JAR files will be transferred along
2170                 with the executable and automatically added to the classpath.
2171
2172
2173          java_vm_args = <argument_list>
2174                 Specifies a list of additional arguments to the Java  VM  it‐
2175                 self,  When HTCondor runs the Java program, these are the ar‐
2176                 guments that go before the class name. This can  be  used  to
2177                 set  VM-specific arguments like stack size, garbage-collector
2178                 arguments and initial property values.
2179
2180          machine_count = <max>
2181                 For the parallel universe, a single value (max) is  required.
2182                 It  is  neither  a  maximum or minimum, but the number of ma‐
2183                 chines to be dedicated toward running the job.
2184
2185
2186          remove_kill_sig = <signal-number>
2187                 For the scheduler universe only, signal-number   is the  sig‐
2188                 nal  delivered  to  the job when the job is removed with con‐
2189                 dor_rm.  signal-number may be  either  the  platform-specific
2190                 name or value of the signal.  This example shows it both ways
2191                 for a Linux signal:
2192
2193                     remove_kill_sig = SIGUSR1
2194                     remove_kill_sig = 10
2195
2196                 If this command is not present, the value  of  kill_sig    is
2197                 used.
2198
2199       COMMANDS FOR THE VM UNIVERSE
2200
2201          vm_disk  = file1:device1:permission1, file2:device2:permission2:for‐
2202          mat2, ...
2203                 A list of comma separated disk files. Each disk file is spec‐
2204                 ified  by  4  colon  separated fields. The first field is the
2205                 path and file name of the disk file. The second field  speci‐
2206                 fies  the  device. The third field specifies permissions, and
2207                 the optional fourth field specifies the image  format.  If  a
2208                 disk  file  will  be  transferred by HTCondor, then the first
2209                 field should just be the simple file name (no  path  informa‐
2210                 tion).
2211
2212                 An example that specifies two disk files:
2213
2214                     vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
2215
2216
2217
2218          vm_checkpoint = <True | False>
2219                 A  boolean  value  specifying  whether  or not to take check‐
2220                 points. If not specified, the default value is False. In  the
2221                 current   implementation,   setting  both  vm_checkpoint  and
2222                 vm_networking to True does not yet work in  all  cases.  Net‐
2223                 working cannot be used if a vm universe job uses a checkpoint
2224                 in order to continue execution after migration to another ma‐
2225                 chine.
2226
2227
2228          vm_macaddr = <MACAddr>
2229                 Defines  that  MAC address that the virtual machine's network
2230                 interface should have, in the standard format of  six  groups
2231                 of two hexadecimal digits separated by colons.
2232
2233
2234          vm_memory = <MBytes-of-memory>
2235                 The  amount  of  memory  in MBytes that a vm universe job re‐
2236                 quires.
2237
2238
2239          vm_networking = <True | False>
2240                 Specifies whether to use networking or not.  In  the  current
2241                 implementation,  setting both vm_checkpoint and vm_networking
2242                 to True does not yet work in all cases. Networking cannot  be
2243                 used  if a vm universe job uses a checkpoint in order to con‐
2244                 tinue execution after migration to another machine.
2245
2246
2247          vm_networking_type = <nat | bridge >
2248                 When vm_networking is  True,  this  definition  augments  the
2249                 job's  requirements to match only machines with the specified
2250                 networking. If not specified,  then  either  networking  type
2251                 matches.
2252
2253
2254          vm_no_output_vm = <True | False>
2255                 When  True,  prevents HTCondor from transferring output files
2256                 back to the machine from which the vm universe job  was  sub‐
2257                 mitted. If not specified, the default value is False.
2258
2259
2260          vm_type = <vmware | xen | kvm>
2261                 Specifies  the  underlying virtual machine software that this
2262                 job expects.
2263
2264          vmware_dir = <pathname>
2265                 The complete path and name of the directory where VMware-spe‐
2266                 cific  files  and  applications such as the VMDK (Virtual Ma‐
2267                 chine Disk Format) and VMX  (Virtual  Machine  Configuration)
2268                 reside.  This  command  is  optional; when not specified, all
2269                 relevant VMware image files are to  be  listed  using  trans‐
2270                 fer_input_files  .
2271
2272
2273          vmware_should_transfer_files = <True | False>
2274                 Specifies  whether  HTCondor  will  transfer  VMware-specific
2275                 files located as specified by vmware_dir   to the execute ma‐
2276                 chine  (True)  or rely on access through a shared file system
2277                 (False). Omission of this required  command  (for  VMware  vm
2278                 universe  jobs)  results in an error message from condor_sub‐
2279                 mit, and the job will not be submitted.
2280
2281
2282          vmware_snapshot_disk = <True | False>
2283                 When True, causes HTCondor to utilize a VMware snapshot  disk
2284                 for  new  or  modified  files.  If not specified, the default
2285                 value is True.
2286
2287          xen_initrd = <image-file>
2288                 When xen_kernel gives a file name for  the  kernel  image  to
2289                 use,  this  optional  command may specify a path to a ramdisk
2290                 (initrd) image file. If the image file will be transferred by
2291                 HTCondor,  then the value should just be the simple file name
2292                 (no path information).
2293
2294
2295          xen_kernel = <included | path-to-kernel>
2296                 A value of included specifies that the kernel is included  in
2297                 the  disk file. If not one of these values, then the value is
2298                 a path and file name of the kernel to be used.  If  a  kernel
2299                 file  will  be transferred by HTCondor, then the value should
2300                 just be the simple file name (no path information).
2301
2302          xen_kernel_params = <string>
2303                 A string that is appended to the Xen kernel command line.
2304
2305
2306          xen_root = <string>
2307                 A string that is appended to the Xen kernel command  line  to
2308                 specify  the  root  device.  This  string  is  required  when
2309                 xen_kernel   gives a path to a kernel. Omission for this  re‐
2310                 quired case results in an error message during submission.
2311
2312       COMMANDS FOR THE DOCKER UNIVERSE
2313
2314
2315          docker_image = < image-name >
2316                 Defines  the  name  of the Docker image that is the basis for
2317                 the docker container.
2318
2319       ADVANCED COMMANDS
2320
2321          accounting_group = <accounting-group-name>
2322                 Causes jobs to negotiate under the  given  accounting  group.
2323                 This value is advertised in the job ClassAd as AcctGroup. The
2324                 HTCondor Administrator's  manual  contains  more  information
2325                 about accounting groups.
2326
2327
2328          accounting_group_user = <accounting-group-user-name>
2329                 Sets  the  name  associated  with this job to be used for re‐
2330                 source usage accounting  purposes,  such  as  computation  of
2331                 fair-share  priority  and  reporting via condor_userprio.  If
2332                 not set, defaults to the value of the job  ClassAd  attribute
2333                 Owner.  This  value is advertised in the job ClassAd as Acct‐
2334                 GroupUser.
2335
2336
2337          concurrency_limits = <string-list>
2338                 A list of resources that this job needs.  The  resources  are
2339                 presumed to have concurrency limits placed upon them, thereby
2340                 limiting the number of concurrent  jobs  in  execution  which
2341                 need  the named resource. Commas and space characters delimit
2342                 the items in the list.  Each item in the  list  is  a  string
2343                 that identifies the limit, or it is a ClassAd expression that
2344                 evaluates to a string, and it is evaluated in the context  of
2345                 machine ClassAd being considered as a match. Each item in the
2346                 list also may specify a numerical value identifying the inte‐
2347                 ger  number  of  resources  required for the job.  The syntax
2348                 follows the resource name by a colon character  (:)  and  the
2349                 numerical value. Details on concurrency limits are in the HT‐
2350                 Condor Administrator's manual.
2351
2352
2353          concurrency_limits_expr = <ClassAd String Expression>
2354                 A ClassAd expression that represents the  list  of  resources
2355                 that  this job needs after evaluation. The ClassAd expression
2356                 may specify machine ClassAd  attributes  that  are  evaluated
2357                 against  a  matched  machine. After evaluation, the list sets
2358                 concurrency_limits.
2359
2360
2361          copy_to_spool = <True | False>
2362                 If copy_to_spool is True, then condor_submit copies the  exe‐
2363                 cutable  to  the local spool directory before running it on a
2364                 remote host. As copying can be quite time consuming  and  un‐
2365                 necessary,  the  default value is False for all job universes
2366                 other than the standard universe.  When False,  condor_submit
2367                 does  not copy the executable to a local spool directory. The
2368                 default is True in standard universe, because resuming execu‐
2369                 tion  from  a checkpoint can only be guaranteed to work using
2370                 precisely the same executable that created the checkpoint.
2371
2372          coresize = <size>
2373                 Should the user's program abort  and  produce  a  core  file,
2374                 coresize specifies the maximum size in bytes of the core file
2375                 which the user wishes to keep. If coresize is  not  specified
2376                 in  the  command file, the system's user resource limit core‐
2377                 dumpsize is used (note that coredumpsize is not  an  HTCondor
2378                 parameter  -  it is an operating system parameter that can be
2379                 viewed with the limit or ulimit command on Unix  and  in  the
2380                 Registry on Windows).  A value of -1 results in no limits be‐
2381                 ing applied to the core file size. If HTCondor is running  as
2382                 root, a coresize setting greater than the system coredumpsize
2383                 limit will override the system setting; if  HTCondor  is  not
2384                 running  as root, the system coredumpsize limit will override
2385                 coresize.
2386
2387
2388          cron_day_of_month = <Cron-evaluated Day>
2389                 The set of days of the month for which a  deferral  time  ap‐
2390                 plies.  The HTCondor User's manual section on Time Scheduling
2391                 for Job Execution has further details.
2392
2393
2394          cron_day_of_week = <Cron-evaluated Day>
2395                 The set of days of the week for which  a  deferral  time  ap‐
2396                 plies.  The HTCondor User's manual section on Time Scheduling
2397                 for Job Execution has further details.
2398
2399          cron_hour = <Cron-evaluated Hour>
2400                 The set of hours of the day for which  a  deferral  time  ap‐
2401                 plies.  The HTCondor User's manual section on Time Scheduling
2402                 for Job Execution has further details.
2403
2404          cron_minute = <Cron-evaluated Minute>
2405                 The set of minutes within an hour for which a  deferral  time
2406                 applies.  The HTCondor User's manual section on Time Schedul‐
2407                 ing for Job Execution has further details.
2408
2409
2410          cron_month = <Cron-evaluated Month>
2411                 The set of months within a year for which a deferral time ap‐
2412                 plies.  The HTCondor User's manual section on Time Scheduling
2413                 for Job Execution has further details.
2414
2415
2416          cron_prep_time = <ClassAd Integer Expression>
2417                 Analogous to deferral_prep_time  .   The  number  of  seconds
2418                 prior  to  a  job's deferral time that the job may be matched
2419                 and sent to an execution machine.
2420
2421
2422          cron_window = <ClassAd Integer Expression>
2423                 Analogous to the submit command deferral_window  .  It allows
2424                 cron jobs that miss their deferral time to begin execution.
2425
2426                 The HTCondor User's manual section on Time Scheduling for Job
2427                 Execution has further details.
2428
2429
2430          dagman_log = <pathname>
2431                 DAGMan inserts this command to specify an event log  that  it
2432                 watches  to  maintain the state of the DAG. If the log   com‐
2433                 mand is not specified in the submit file, DAGMan uses the log
2434                 command to specify the event log.
2435
2436          deferral_prep_time = <ClassAd Integer Expression>
2437                 The number of seconds prior to a job's deferral time that the
2438                 job may be matched and sent to an execution machine.
2439
2440                 The HTCondor User's manual section on Time Scheduling for Job
2441                 Execution has further details.
2442
2443
2444          deferral_time = <ClassAd Integer Expression>
2445                 Allows a job to specify the time at which its execution is to
2446                 begin, instead of beginning execution as soon as  it  arrives
2447                 at  the execution machine. The deferral time is an expression
2448                 that evaluates to a Unix Epoch timestamp (the number of  sec‐
2449                 onds  elapsed  since 00:00:00 on January 1, 1970, Coordinated
2450                 Universal Time). Deferral time is evaluated with  respect  to
2451                 the execution machine. This option delays the start of execu‐
2452                 tion, but not the matching and claiming of a machine for  the
2453                 job. If the job is not available and ready to begin execution
2454                 at the deferral time, it has missed its deferral time. A  job
2455                 that  misses  its  deferral  time  will be put on hold in the
2456                 queue.
2457
2458                 The HTCondor User's manual section on Time Scheduling for Job
2459                 Execution has further details.
2460
2461                 Due  to  implementation  details,  a deferral time may not be
2462                 used for scheduler universe jobs.
2463
2464
2465          deferral_window = <ClassAd Integer Expression>
2466                 The deferral window is used in conjunction  with  the  defer‐
2467                 ral_time  command to allow jobs that miss their deferral time
2468                 to begin execution.
2469
2470                 The HTCondor User's manual section on Time Scheduling for Job
2471                 Execution has further details.
2472
2473
2474          description = <string>
2475                 A  string  that  sets  the value of the job ClassAd attribute
2476                 JobDescription. When set, tools which display the  executable
2477                 such as condor_q will instead use this string.
2478
2479
2480          email_attributes = <list-of-job-ad-attributes>
2481                 A  comma-separated  list  of attributes from the job ClassAd.
2482                 These attributes and their values will  be  included  in  the
2483                 e-mail notification of job completion.
2484
2485
2486          image_size = <size>
2487                 Advice  to HTCondor specifying the maximum virtual image size
2488                 to which the job will grow  during  its  execution.  HTCondor
2489                 will  then execute the job only on machines which have enough
2490                 resources, (such as virtual memory), to support executing the
2491                 job.  If  not  specified,  HTCondor will automatically make a
2492                 (reasonably accurate) estimate about the job's size  and  ad‐
2493                 just this estimate as the program runs.  If specified and un‐
2494                 derestimated, the job may crash due to the inability  to  ac‐
2495                 quire  more address space; for example, if malloc() fails. If
2496                 the image size is overestimated, HTCondor may have difficulty
2497                 finding  machines which have the required resources.  size is
2498                 specified in KiB. For example, for an image size  of  8  MiB,
2499                 size should be 8000.
2500
2501          initialdir = <directory-path>
2502                 Used  to give jobs a directory with respect to file input and
2503                 output.  Also provides a directory (on the machine from which
2504                 the job is submitted) for the job event log, when a full path
2505                 is not specified.
2506
2507                 For vanilla universe jobs where there is a shared  file  sys‐
2508                 tem, it is the current working directory on the machine where
2509                 the job is executed.
2510
2511                 For vanilla or grid universe jobs where file transfer  mecha‐
2512                 nisms are utilized (there is not a shared file system), it is
2513                 the directory on the machine from which the job is  submitted
2514                 where  the  input files come from, and where the job's output
2515                 files go to.
2516
2517                 For standard universe jobs, it is the directory  on  the  ma‐
2518                 chine from which the job is submitted where the condor_shadow
2519                 daemon runs; the current working directory for file input and
2520                 output accomplished through remote system calls.
2521
2522                 For  scheduler  universe jobs, it is the directory on the ma‐
2523                 chine from which the job is submitted where the job runs; the
2524                 current  working directory for file input and output with re‐
2525                 spect to relative path names.
2526
2527                 Note that the path to the executable is not relative to  ini‐
2528                 tialdir   ;  if  it is a relative path, it is relative to the
2529                 directory in which the condor_submit command is run.
2530
2531
2532          job_ad_information_attrs = <attribute-list>
2533                 A comma-separated list of job ClassAd  attribute  names.  The
2534                 named  attributes  and  their  values  are written to the job
2535                 event log whenever any event is being  written  to  the  log.
2536                 This  implements the same thing as the configuration variable
2537                 EVENT_LOG_INFORMATION_ATTRS (see the  admin-manual/configura‐
2538                 tion-macros:daemon  logging configuration file entries page),
2539                 but it applies to the job event log, instead  of  the  system
2540                 event log.
2541
2542
2543          JobBatchName = <batch_name>
2544                 Set  the  batch  name for this submit. The batch name is dis‐
2545                 played by condor_q -batch. It is intended for use by users to
2546                 give meaningful names to their jobs and to influence how con‐
2547                 dor_q groups jobs for display. This value in  a  submit  file
2548                 can  be  overridden by specifying the -batch-name argument on
2549                 the condor_submit command line.
2550
2551
2552          job_lease_duration = <number-of-seconds>
2553                 For vanilla, parallel, VM, and java universe jobs  only,  the
2554                 duration  in  seconds  of  a  job lease. The default value is
2555                 2,400, or forty minutes. If a job lease is not  desired,  the
2556                 value can be explicitly set to 0 to disable the job lease se‐
2557                 mantics. The value can also  be  a  ClassAd  expression  that
2558                 evaluates  to  an integer. The HTCondor User's manual section
2559                 on Special Environment Considerations has further details.
2560
2561
2562          job_machine_attrs = <attr1, attr2, ...>
2563                 A comma and/or space  separated  list  of  machine  attribute
2564                 names  that should be recorded in the job ClassAd in addition
2565                 to the ones specified by the  condor_schedd  daemon's  system
2566                 configuration variable SYSTEM_JOB_MACHINE_ATTRS
2567                  .  When  there are multiple run attempts, history of machine
2568                 attributes from previous run attempts may be kept. The number
2569                 of  run  attempts  to  store  may be extended beyond the sys‐
2570                 tem-specified history length by using the submit file command
2571                 job_machine_attrs_history_length    .   A  machine  attribute
2572                 named X will be inserted into the job ClassAd as an attribute
2573                 named  MachineAttrX0.  The  previous  value of this attribute
2574                 will be named MachineAttrX1, the previous  to  that  will  be
2575                 named  MachineAttrX2,  and so on, up to the specified history
2576                 length. A history of length 1 means that  only  MachineAttrX0
2577                 will  be  recorded.  The value recorded in the job ClassAd is
2578                 the evaluation of the machine attribute in the context of the
2579                 job ClassAd when the condor_schedd daemon initiates the start
2580                 up of the job. If the evaluation results in an  Undefined  or
2581                 Error  result, the value recorded in the job ad will be Unde‐
2582                 fined or Error, respectively.
2583
2584
2585          want_graceful_removal = <boolean expression>
2586                 If true, this job will be given a chance to shut down cleanly
2587                 when  removed.  The job will be given as much time as the ad‐
2588                 ministrator of the execute  resource  allows,  which  may  be
2589                 none.  The default is false.  For details, see the configura‐
2590                 tion setting GRACEFULLY_REMOVE_JOBS.
2591
2592
2593          kill_sig = <signal-number>
2594                 When HTCondor needs to kick a job off of a machine,  it  will
2595                 send  the  job the signal specified by signal-number  .  sig‐
2596                 nal-number needs to be an integer which  represents  a  valid
2597                 signal  on  the  execution machine. For jobs submitted to the
2598                 standard universe, the default value is the number for  SIGT‐
2599                 STP  which  tells the HTCondor libraries to initiate a check‐
2600                 point of the process. For jobs submitted to other  universes,
2601                 the default value, when not defined, is SIGTERM, which is the
2602                 standard way to terminate a program in Unix.
2603
2604          kill_sig_timeout = <seconds>
2605                 This submit command should no longer be used as  of  HTCondor
2606                 version    7.7.3;   use   job_max_vacate_time   instead.   If
2607                 job_max_vacate_time is not defined, this defines  the  number
2608                 of seconds that HTCondor should wait following the sending of
2609                 the kill signal defined by kill_sig    and  forcibly  killing
2610                 the job. The actual amount of time between sending the signal
2611                 and forcibly killing the job is the smallest  of  this  value
2612                 and the configuration variable KILLING_TIMEOUT
2613                  , as defined on the execute machine.
2614
2615
2616          load_profile = <True | False>
2617                 When True, loads the account profile of the dedicated run ac‐
2618                 count for Windows jobs. May not be used with run_as_owner  .
2619
2620
2621          match_list_length = <integer value>
2622                 Defaults to the value zero (0). When match_list_length is de‐
2623                 fined with an integer value greater than zero (0), attributes
2624                 are inserted into the job ClassAd. The maximum number of  at‐
2625                 tributes defined is given by the integer value. The job Clas‐
2626                 sAds introduced are given as
2627
2628                     LastMatchName0 = "most-recent-Name"
2629                     LastMatchName1 = "next-most-recent-Name"
2630
2631                 The value for each introduced ClassAd is given by  the  value
2632                 of  the Name attribute from the machine ClassAd of a previous
2633                 execution (match). As a job is matched, the  definitions  for
2634                 these  attributes  will  roll,  with  LastMatchName1 becoming
2635                 LastMatchName2, LastMatchName0 becoming  LastMatchName1,  and
2636                 LastMatchName0 being set by the most recent value of the Name
2637                 attribute.
2638
2639                 An intended use of these job attributes is  in  the  require‐
2640                 ments  expression. The requirements can allow a job to prefer
2641                 a match with either the same or a different resource  than  a
2642                 previous match.
2643
2644
2645
2646          job_max_vacate_time = <integer expression>
2647                 An integer-valued expression (in seconds) that may be used to
2648                 adjust the time given to an evicted job for gracefully  shut‐
2649                 ting  down.  If the job's setting is less than the machine's,
2650                 the job's is used. If the job's setting is  larger  than  the
2651                 machine's,  the result depends on whether the job has any ex‐
2652                 cess retirement time. If the job  has  more  retirement  time
2653                 left than the machine's max vacate time setting, then retire‐
2654                 ment time will be converted into vacating  time,  up  to  the
2655                 amount requested by the job.
2656
2657                 Setting  this  expression  does not affect the job's resource
2658                 requirements or preferences. For a job to only run on  a  ma‐
2659                 chine  with  a  minimum MachineMaxVacateTime, or to preferen‐
2660                 tially run on such machines, explicitly specify this  in  the
2661                 requirements and/or rank expressions.
2662
2663
2664          max_job_retirement_time = <integer expression>
2665                 An  integer-valued  expression (in seconds) that does nothing
2666                 unless the machine that runs the job has been  configured  to
2667                 provide  retirement  time.  Retirement time is a grace period
2668                 given to a job to finish when a resource claim is about to be
2669                 preempted.  The  default behavior in many cases is to take as
2670                 much retirement time as the machine offers, so  this  command
2671                 will rarely appear in a submit description file.
2672
2673                 When  a resource claim is to be preempted, this expression in
2674                 the submit file specifies the maximum run time of the job (in
2675                 seconds,  since  the job started). This expression has no ef‐
2676                 fect, if it is greater than the maximum retirement time  pro‐
2677                 vided  by  the  machine  policy. If the resource claim is not
2678                 preempted, this expression and the machine retirement  policy
2679                 are  irrelevant.  If  the resource claim is preempted the job
2680                 will be allowed to run until the retirement time expires,  at
2681                 which  point  it  is hard-killed. The job will be soft-killed
2682                 when it is getting close to the end of retirement in order to
2683                 give it time to gracefully shut down. The amount of lead-time
2684                 for soft-killing is determined by the maximum  vacating  time
2685                 granted to the job.
2686
2687                 Standard  universe  jobs  and any jobs running with nice_user
2688                 priority have a default max_job_retirement_time of 0,  so  no
2689                 retirement  time  is utilized by default. In all other cases,
2690                 no default value is provided, so the maximum  amount  of  re‐
2691                 tirement time is utilized by default.
2692
2693                 Setting  this  expression  does not affect the job's resource
2694                 requirements or preferences. For a job to only run on  a  ma‐
2695                 chine  with  a  minimum MaxJobRetirementTime, or to preferen‐
2696                 tially run on such machines, explicitly specify this  in  the
2697                 requirements and/or rank expressions.
2698
2699          nice_user = <True | False>
2700                 Normally,  when  a machine becomes available to HTCondor, HT‐
2701                 Condor decides which job to run based upon user and job  pri‐
2702                 orities.  Setting  nice_user equal to True tells HTCondor not
2703                 to use your regular user priority, but that this  job  should
2704                 have last priority among all users and all jobs. So jobs sub‐
2705                 mitted in this fashion run only on machines  which  no  other
2706                 non-nice_user  job  wants - a true bottom-feeder job! This is
2707                 very handy if a user has some jobs they wish to run,  but  do
2708                 not  wish  to use resources that could instead be used to run
2709                 other people's HTCondor jobs. Jobs submitted in this  fashion
2710                 have  "nice-user."  prepended  to  the owner name when viewed
2711                 from condor_q or condor_userprio. The default value is False.
2712
2713          noop_job = <ClassAd Boolean Expression>
2714                 When this boolean expression is True, the job is  immediately
2715                 removed from the queue, and HTCondor makes no attempt at run‐
2716                 ning the job. The log file for the job will show a  job  sub‐
2717                 mitted  event  and a job terminated event, along with an exit
2718                 code of 0, unless the user specifies a  different  signal  or
2719                 exit code.
2720
2721
2722          noop_job_exit_code = <return value>
2723                 When  noop_job   is in the submit description file and evalu‐
2724                 ates to True, this command allows the job to specify the  re‐
2725                 turn  value  as  shown  in  the job's log file job terminated
2726                 event. If not specified, the job will show as  having  termi‐
2727                 nated  with status 0. This overrides any value specified with
2728                 noop_job_exit_signal  .
2729
2730
2731          noop_job_exit_signal = <signal number>
2732                 When noop_job   is in the submit description file and  evalu‐
2733                 ates to True, this command allows the job to specify the sig‐
2734                 nal number that the job's log event will show the job  having
2735                 terminated with.
2736
2737
2738          remote_initialdir = <directory-path>
2739                 The  path  specifies  the directory in which the job is to be
2740                 executed on the remote machine. This is  currently  supported
2741                 in all universes except for the standard universe.
2742
2743
2744          rendezvousdir = <directory-path>
2745                 Used  to  specify the shared file system directory to be used
2746                 for file system authentication when submitting  to  a  remote
2747                 scheduler. Should be a path to a preexisting directory.
2748
2749
2750          run_as_owner = <True | False>
2751                 A boolean value that causes the job to be run under the login
2752                 of the submitter, if supported by the joint configuration  of
2753                 the  submit and execute machines. On Unix platforms, this de‐
2754                 faults to True, and on  Windows  platforms,  it  defaults  to
2755                 False. May not be used with load_profile  .  See the HTCondor
2756                 manual Platform-Specific Information chapter for  administra‐
2757                 tive details on configuring Windows to support this option.
2758
2759          stack_size = <size in bytes>
2760                 This command applies only to Linux platform jobs that are not
2761                 standard universe jobs. An integer number  of  bytes,  repre‐
2762                 senting  the  amount  of  stack space to be allocated for the
2763                 job. This value replaces  the  default  allocation  of  stack
2764                 space, which is unlimited in size.
2765
2766          submit_event_notes = <note>
2767                 A  string  that  is appended to the submit event in the job's
2768                 log file.  For DAGMan jobs, the  string  DAG  Node:  and  the
2769                 node's  name is automatically defined for submit_event_notes,
2770                 causing the logged submit event to identify the DAG node  job
2771                 submitted.
2772
2773          +<attribute> = <value>
2774                 A line that begins with a '+' (plus) character instructs con‐
2775                 dor_submit to insert the given attribute into the job ClassAd
2776                 with  the  given value. Note that setting an attribute should
2777                 not be used in place of one of the specific  commands  listed
2778                 above.  Often,  the command name does not directly correspond
2779                 to an attribute name; furthermore, many submit  commands  re‐
2780                 sult in actions more complex than simply setting an attribute
2781                 or attributes. See /classad-attributes/job-classad-attributes
2782                 for a list of HTCondor job attributes.
2783
2784       MACROS AND COMMENTS
2785
2786
2787       In addition to commands, the submit description file can contain macros
2788       and comments.
2789
2790          Macros Parameterless macros in the form of $(macro_name:default ini‐
2791                 tial  value) may be used anywhere in HTCondor submit descrip‐
2792                 tion files to provide textual substitution  at  submit  time.
2793                 Macros can be defined by lines in the form of
2794
2795                     <macro_name> = <string>
2796
2797                 Two pre-defined macros are supplied by the submit description
2798                 file parser. The $(Cluster) or  $(ClusterId)  macro  supplies
2799                 the value of the
2800
2801                  ClusterId  job  ClassAd  attribute,  and  the  $(Process) or
2802                 $(ProcId) macro supplies the value of the ProcId job  ClassAd
2803                 attribute. These macros are intended to aid in the specifica‐
2804                 tion of input/output files,  arguments,  etc.,  for  clusters
2805                 with lots of jobs, and/or could be used to supply an HTCondor
2806                 process with its own cluster and process numbers on the  com‐
2807                 mand line.
2808
2809                 The  $(Node) macro is defined for parallel universe jobs, and
2810                 is especially relevant for MPI applications. It is  a  unique
2811                 value  assigned  for the duration of the job that essentially
2812                 identifies the machine (slot) on which a program  is  execut‐
2813                 ing.  Values  assigned start at 0 and increase monotonically.
2814                 The values are assigned as  the  parallel  job  is  about  to
2815                 start.
2816
2817                 Recursive  definition of macros is permitted. An example of a
2818                 construction that works is the following:
2819
2820                     foo = bar
2821                     foo =  snap $(foo)
2822
2823                 As a result, foo = snap bar.
2824
2825                 Note that both left- and right- recursion works, so
2826
2827                     foo = bar
2828                     foo =  $(foo) snap
2829
2830                 has as its result foo = bar snap.
2831
2832                 The construction
2833
2834                     foo = $(foo) bar
2835
2836                 by itself will not work, as it does not have an initial  base
2837                 case.  Mutually recursive constructions such as:
2838
2839                     B = bar
2840                     C = $(B)
2841                     B = $(C) boo
2842
2843                 will not work, and will fill memory with expansions.
2844
2845                 A default value may be specified, for use if the macro has no
2846                 definition. Consider the example
2847
2848                     D = $(E:24)
2849
2850                 Where E is not defined within the  submit  description  file,
2851                 the default value 24 is used, resulting in
2852
2853                     D = 24
2854
2855                 This  is of limited value, as the scope of macro substitution
2856                 is the submit description file. Thus, either the macro is  or
2857                 is  not  defined  within  the submit description file. If the
2858                 macro is defined, then the default value is useless.  If  the
2859                 macro is not defined, then there is no point in using it in a
2860                 submit command.
2861
2862
2863                 To use the dollar sign character ($) as  a  literal,  without
2864                 macro expansion, use
2865
2866                     $(DOLLAR)
2867
2868                 In addition to the normal macro, there is also a special kind
2869                 of macro called a substitution macro
2870                  that allows the substitution of a machine ClassAd  attribute
2871                 value  defined on the resource machine itself (gotten after a
2872                 match to the machine has been made)  into  specific  commands
2873                 within the submit description file. The substitution macro is
2874                 of the form:
2875
2876                     $$(attribute)
2877
2878                 As this form of the  substitution  macro  is  only  evaluated
2879                 within  the  context  of  the machine ClassAd, use of a scope
2880                 resolution prefix TARGET. or MY. is not allowed.
2881
2882                 A common use of this form of the substitution  macro  is  for
2883                 the heterogeneous submission of an executable:
2884
2885                     executable = povray.$$(OpSys).$$(Arch)
2886
2887                 Values  for  the OpSys and Arch attributes are substituted at
2888                 match time for any given resource. This example allows HTCon‐
2889                 dor  to  automatically  choose the correct executable for the
2890                 matched machine.
2891
2892                 An extension to the syntax of the substitution macro provides
2893                 an  alternative string to use if the machine attribute within
2894                 the substitution macro is undefined. The syntax appears as:
2895
2896                     $$(attribute:string_if_attribute_undefined)
2897
2898                 An example using this extended syntax provides a path name to
2899                 a  required  input file. Since the file can be placed in dif‐
2900                 ferent locations on different machines, the file's path  name
2901                 is given as an argument to the program.
2902
2903                     arguments = $$(input_file_path:/usr/foo)
2904
2905                 On  the  machine, if the attribute input_file_path is not de‐
2906                 fined, then the path /usr/foo is used instead.
2907
2908                 A further extension to the syntax of the  substitution  macro
2909                 allows  the  evaluation of a ClassAd expression to define the
2910                 value. In this form, the expression may refer to machine  at‐
2911                 tributes  by prefacing them with the TARGET. scope resolution
2912                 prefix. To place a ClassAd expression into  the  substitution
2913                 macro,  square  brackets are added to delimit the expression.
2914                 The syntax appears as:
2915
2916                     $$([ClassAd expression])
2917
2918                 An example of a job that uses this syntax  may  be  one  that
2919                 wants  to  know  how  much memory it can use. The application
2920                 cannot detect this itself, as it would potentially use all of
2921                 the memory on a multi-slot machine. So the job determines the
2922                 memory per slot, reducing it by 10% to account for  miscella‐
2923                 neous overhead, and passes this as a command line argument to
2924                 the application. In the submit description file will be
2925
2926                     arguments = --memory $$([TARGET.Memory * 0.9])
2927
2928
2929
2930                 To insert two dollar sign characters ($$) as literals into  a
2931                 ClassAd string, use
2932
2933                     $$(DOLLARDOLLAR)
2934
2935
2936
2937
2938
2939                 The  environment macro, $ENV, allows the evaluation of an en‐
2940                 vironment variable to be used in setting a submit description
2941                 file command.  The syntax used is
2942
2943                     $ENV(variable)
2944
2945                 An  example  submit  description  file command that uses this
2946                 functionality evaluates the submitter's home directory in or‐
2947                 der to set the path and file name of a log file:
2948
2949                     log = $ENV(HOME)/jobs/logfile
2950
2951                 The  environment  variable  is  evaluated when the submit de‐
2952                 scription file is processed.
2953
2954
2955
2956
2957                 The $RANDOM_CHOICE macro allows a random choice  to  be  made
2958                 from  a  given  list of parameters at submission time. For an
2959                 expression, if some randomness needs  to  be  generated,  the
2960                 macro may appear as
2961
2962                     $RANDOM_CHOICE(0,1,2,3,4,5,6)
2963
2964                 When evaluated, one of the parameters values will be chosen.
2965
2966          Comments
2967                 Blank lines and lines beginning with a pound sign ('#') char‐
2968                 acter are ignored by the submit description file parser.
2969

SUBMIT VARIABLES

2971       While processing the queue command in a submit file or from the command
2972       line,  condor_submit  will  set  the values of several automatic submit
2973       variables so that they can be referred to by statements in  the  submit
2974       file. With the exception of Cluster and Process, if these variables are
2975       set by the submit file, they will not be modified during  queue    pro‐
2976       cessing.
2977
2978          ClusterId
2979                 Set  to  the  integer value that the ClusterId attribute that
2980                 the job ClassAd will have when the job is submitted. All jobs
2981                 in  a single submit will normally have the same value for the
2982                 ClusterId. If the -dry-run argument is specified,  The  value
2983                 will be 1.
2984
2985          Cluster
2986                 Alternate  name for the ClusterId submit variable. Before HT‐
2987                 Condor version 8.4 this was the only name.
2988
2989          ProcId Set to the integer value that the ProcId attribute of the job
2990                 ClassAd  will  have when the job is submitted. The value will
2991                 start at 0 and increment by 1 for each job submitted.
2992
2993          Process
2994                 Alternate name for the ProcId submit variable. Before  HTCon‐
2995                 dor version 8.4 this was the only name.
2996
2997          Node   For  parallel  universes,  set to the value #pArAlLeLnOdE# or
2998                 #MpInOdE# depending on the parallel universe type  For  other
2999                 universes it is set to nothing.
3000
3001          Step   Set  to  the step value as it varies from 0 to N-1 where N is
3002                 the number provided on the queue    argument.  This  variable
3003                 changes  at  the  same rate as ProcId when it changes at all.
3004                 For submit files that don't make use of the queue number  op‐
3005                 tion, Step will always be 0. For submit files that don't make
3006                 use of any of the foreach options, Step and ProcId  will  al‐
3007                 ways be the same.
3008
3009          ItemIndex
3010                 Set  to the index within the item list being processed by the
3011                 various queue foreach options. For submit  files  that  don't
3012                 make  use of any queue foreach list, ItemIndex will always be
3013                 0 For submit files that make use of a slice  to  select  only
3014                 some  items  in a foreach list, ItemIndex will only be set to
3015                 selected values.
3016
3017          Row    Alternate name for ItemIndex.
3018
3019          Item   when a queue foreach option is used and no variable  list  is
3020                 supplied,  this variable will be set to the value of the cur‐
3021                 rent item.
3022
3023       The automatic variables below are set before parsing the  submit  file,
3024       and  will not vary during processing unless the submit file itself sets
3025       them.
3026
3027          ARCH   Set to the CPU  architecture  of  the  machine  running  con‐
3028                 dor_submit.  The value will be the same as the automatic con‐
3029                 figuration variable of the same name.
3030
3031          OPSYS  Set to the name of the operating system on the  machine  run‐
3032                 ning  condor_submit.  The value will be the same as the auto‐
3033                 matic configuration variable of the same name.
3034
3035          OPSYSANDVER
3036                 Set to the name and major version of the operating system  on
3037                 the machine running condor_submit. The value will be the same
3038                 as the automatic configuration variable of the same name.
3039
3040          OPSYSMAJORVER
3041                 Set to the major version of the operating system on  the  ma‐
3042                 chine  running  condor_submit.  The value will be the same as
3043                 the automatic configuration variable of the same name.
3044
3045          OPSYSVER
3046                 Set to the version of the operating  system  on  the  machine
3047                 running  condor_submit. The value will be the same as the au‐
3048                 tomatic configuration variable of the same name.
3049
3050          SPOOL  Set to the full path of the  HTCondor  spool  directory.  The
3051                 value  will  be the same as the automatic configuration vari‐
3052                 able of the same name.
3053
3054          IsLinux
3055                 Set to true if the operating system of  the  machine  running
3056                 condor_submit is a Linux variant. Set to false otherwise.
3057
3058          IsWindows
3059                 Set  to  true  if the operating system of the machine running
3060                 condor_submit is a Microsoft Windows variant.  Set  to  false
3061                 otherwise.
3062
3063          SUBMIT_FILE
3064                 Set  to  the full pathname of the submit file being processed
3065                 by condor_submit. If submit statements are read from standard
3066                 input, it is set to nothing.
3067

EXIT STATUS

3069       condor_submit  will  exit with a status value of 0 (zero) upon success,
3070       and a non-zero value upon failure.
3071

EXAMPLES

3073       • Submit Description File Example 1: This example queues three jobs for
3074         execution by HTCondor. The first will be given command line arguments
3075         of 15 and 2000, and it will write its standard  output  to  foo.out1.
3076         The  second  will be given command line arguments of 30 and 2000, and
3077         it will write its standard output to foo.out2.  Similarly  the  third
3078         will  have arguments of 45 and 6000, and it will use foo.out3 for its
3079         standard output. Standard error output (if any) from all  three  pro‐
3080         grams will appear in foo.error.
3081
3082            ####################
3083            #
3084            # submit description file
3085            # Example 1: queuing multiple jobs with differing
3086            # command line arguments and output files.
3087            #
3088            ####################
3089
3090            Executable     = foo
3091            Universe       = vanilla
3092
3093            Arguments      = 15 2000
3094            Output  = foo.out0
3095            Error   = foo.err0
3096            Queue
3097
3098            Arguments      = 30 2000
3099            Output  = foo.out1
3100            Error   = foo.err1
3101            Queue
3102
3103            Arguments      = 45 6000
3104            Output  = foo.out2
3105            Error   = foo.err2
3106            Queue
3107
3108         Or  you  can get the same results as the above submit file by using a
3109         list of arguments with the Queue statement
3110
3111            ####################
3112            #
3113            # submit description file
3114            # Example 1b: queuing multiple jobs with differing
3115            # command line arguments and output files, alternate syntax
3116            #
3117            ####################
3118
3119            Executable     = foo
3120            Universe       = vanilla
3121
3122            # generate different output and error filenames for each process
3123            Output  = foo.out$(Process)
3124            Error   = foo.err$(Process)
3125
3126            Queue Arguments From (
3127              15 2000
3128              30 2000
3129              45 6000
3130            )
3131
3132       • Submit Description File Example 2: This submit description file exam‐
3133         ple  queues 150 runs of program foo which must have been compiled and
3134         linked for an Intel x86 processor running RHEL 3.  HTCondor will  not
3135         attempt  to  run  the  processes  on machines which have less than 32
3136         Megabytes of physical memory, and it will run them on machines  which
3137         have  at  least  64 Megabytes, if such machines are available. Stdin,
3138         stdout, and stderr will refer to in.0, out.0, and err.0 for the first
3139         run of this program (process 0). Stdin, stdout, and stderr will refer
3140         to in.1, out.1, and err.1 for process 1, and so  forth.  A  log  file
3141         containing  entries  about where and when HTCondor runs, takes check‐
3142         points, and migrates processes in this cluster will be  written  into
3143         file foo.log.
3144
3145            ####################
3146            #
3147            # Example 2: Show off some fancy features including
3148            # use of pre-defined macros and logging.
3149            #
3150            ####################
3151
3152            Executable     = foo
3153            Universe       = standard
3154            Requirements   = OpSys == "LINUX" && Arch =="INTEL"
3155            Rank           = Memory >= 64
3156            Request_Memory = 32 Mb
3157            Image_Size     = 28 Mb
3158
3159            Error   = err.$(Process)
3160            Input   = in.$(Process)
3161            Output  = out.$(Process)
3162            Log = foo.log
3163            Queue 150
3164
3165       • Submit   Description   File  Example  3:  This  example  targets  the
3166         /bin/sleep program to run only on a platform running a RHEL 6 operat‐
3167         ing system. The example presumes that the pool contains machines run‐
3168         ning more than one version of Linux, and this job needs the  particu‐
3169         lar operating system to run correctly.
3170
3171            ####################
3172            #
3173            # Example 3: Run on a RedHat 6 machine
3174            #
3175            ####################
3176            Universe     = vanilla
3177            Executable   = /bin/sleep
3178            Arguments    = 30
3179            Requirements = (OpSysAndVer == "RedHat6")
3180
3181            Error   = err.$(Process)
3182            Input   = in.$(Process)
3183            Output  = out.$(Process)
3184            Log     = sleep.log
3185            Queue
3186
3187       • Command  Line  example: The following command uses the -append option
3188         to add two commands before the job(s) is queued. A log  file  and  an
3189         error  log  file  are  specified.  The submit description file is un‐
3190         changed.
3191
3192            condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile
3193
3194         Note that each of the added commands is contained within quote  marks
3195         because there are space characters within the command.
3196
3197periodic_remove  example:  A job should be removed from the queue, if
3198         the total suspension time of the job is more than  half  of  the  run
3199         time of the job.
3200
3201         Including the command
3202
3203            periodic_remove = CumulativeSuspensionTime >
3204                              ((RemoteWallClockTime - CumulativeSuspensionTime) / 2.0)
3205
3206         in the submit description file causes this to happen.
3207

GENERAL REMARKS

3209       • For  security reasons, HTCondor will refuse to run any jobs submitted
3210         by user root (UID = 0) or by a user  whose  default  group  is  group
3211         wheel (GID = 0). Jobs submitted by user root or a user with a default
3212         group of wheel will appear to sit forever in the  queue  in  an  idle
3213         state.
3214
3215       • All  path names specified in the submit description file must be less
3216         than 256 characters in length, and command  line  arguments  must  be
3217         less than 4096 characters in length; otherwise, condor_submit gives a
3218         warning message but the jobs will not execute properly.
3219
3220       • Somewhat understandably, behavior gets bizarre if the user makes  the
3221         mistake  of  requesting  multiple  HTCondor jobs to write to the same
3222         file, and/or if the user alters any files that need to be accessed by
3223         an  HTCondor  job  which is still in the queue. For example, the com‐
3224         pressing of data or output files before an HTCondor job has completed
3225         is a common mistake.
3226
3227       • To  disable  checkpointing  for  Standard  Universe jobs, include the
3228         line:
3229
3230            +WantCheckpoint = False
3231
3232         in the submit description file before the queue command(s).
3233

SEE ALSO

3235       HTCondor User Manual
3236

AUTHOR

3238       HTCondor Team
3239
3241       1990-2022, Center for High Throughput Computing, Computer Sciences  De‐
3242       partment,  University  of  Wisconsin-Madison, Madison, WI, US. Licensed
3243       under the Apache License, Version 2.0.
3244
3245
3246
3247
32488.8                              Jan 19, 2022                 CONDOR_SUBMIT(1)
Impressum