1CONDOR_SUBMIT_DAG(1)            HTCondor Manual           CONDOR_SUBMIT_DAG(1)
2
3
4

NAME

6       condor_submit_dag - HTCondor Manual
7
8       Manage  and  queue  jobs within a specified DAG for execution on remote
9       machines
10
11

SYNOPSIS

13       condor_submit_dag [-help | -version ]
14
15       condor_submit_dag [-no_submit ] [-verbose ] [-force  ]  [-maxidle  Num‐
16       berOfProcs]   [-maxjobs  NumberOfClusters]  [-dagman  DagmanExecutable]
17       [-maxpre NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notifica‐
18       tion  value] [-noeventchecks ] [-allowlogerror ] [-r schedd_name] [-de‐
19       bug level] [-usedagdir ] [-outfile_dir directory] [-config  ConfigFile‐
20       Name]   [-insert_sub_file   FileName]  [-append  Command]  [-batch-name
21       batch_name] [-autorescue 0|1] [-dorescuefrom number] [-allowversionmis‐
22       match  ] [-no_recurse ] [-do_recurse ] [-update_submit ] [-import_env ]
23       [-DumpRescue ] [-valgrind ]  [-DontAlwaysRunPost  ]  [-AlwaysRunPost  ]
24       [-priority    number]   [-dont_use_default_node_log   ]   [-schedd-dae‐
25       mon-ad-file FileName] [-schedd-address-file FileName]  [-suppress_noti‐
26       fication  ] [-dont_suppress_notification ] [-DoRecovery ] DAGInputFile1
27       [DAGInputFile2 ...DAGInputFileN ]
28

DESCRIPTION

30       condor_submit_dag is the program for submitting a DAG (directed acyclic
31       graph)  of  jobs for execution under HTCondor. The program enforces the
32       job dependencies defined in one or more DAGInputFile s.  Each DAGInput‐
33       File  contains commands to direct the submission of jobs implied by the
34       nodes of a DAG to HTCondor. Extensive documentation is in the  HTCondor
35       User Manual section on DAGMan.
36
37
38
39       Some  options may be specified on the command line or in the configura‐
40       tion or in a node job's submit description file. Precedence is given to
41       command  line  options or configuration over settings from a submit de‐
42       scription file. An example is e-mail notifications. When  configuration
43       variable DAGMAN_SUPPRESS_NOTIFICATION
44         is  its  default  value  of True, and a node job's submit description
45       file contains
46
47          notification = Complete
48
49       e-mail will not be sent upon completion, as the  value  of  DAGMAN_SUP‐
50       PRESS_NOTIFICATION is enforced.
51

OPTIONS

53          -help  Display usage information and exit.
54
55          -version
56                 Display version information and exit.
57
58          -no_submit
59                 Produce  the HTCondor submit description file for DAGMan, but
60                 do not submit DAGMan as an HTCondor job.
61
62          -verbose
63                 Cause condor_submit_dag to give verbose error messages.
64
65          -force Require condor_submit_dag to overwrite the files that it pro‐
66                 duces,  if the files already exist. Note that dagman.out will
67                 be appended to, not overwritten. If new-style rescue DAG mode
68                 is in effect, and any new-style rescue DAGs exist, the -force
69                 flag will cause them to be renamed, and the original DAG will
70                 be run. If old-style rescue DAG mode is in effect, any exist‐
71                 ing old-style rescue DAGs will be deleted, and  the  original
72                 DAG will be run.
73
74          -maxidle NumberOfProcs
75                 Sets  the  maximum  number  of idle procs allowed before con‐
76                 dor_dagman stops submitting more node  jobs.  Note  that  for
77                 this  argument,  each individual proc within a cluster counts
78                 as a towards the limit, which is inconsistent with -maxjobs .
79                 Once  idle procs start to run, condor_dagman will resume sub‐
80                 mitting jobs once the number of idle procs  falls  below  the
81                 specified limit.  NumberOfProcs is a non-negative integer. If
82                 this option is omitted, the number of idle procs  is  limited
83                 by the configuration variable DAGMAN_MAX_JOBS_IDLE
84                   (see  admin-manual/configuration-macros:configuration  file
85                 entries for dagman), which defaults to 1000. To disable  this
86                 limit,  set  NumberOfProcs to 0. Note that submit description
87                 files that queue multiple procs can cause  the  NumberOfProcs
88                 limit  to  be  exceeded. Setting queue 5000 in the submit de‐
89                 scription file, where -maxidle is set to 250 will result in a
90                 cluster  of  5000  new  procs  being  submitted  to  the con‐
91                 dor_schedd, not 250. In this case, condor_dagman will  resume
92                 submitting  jobs  when  the  number of idle procs falls below
93                 250.
94
95          -maxjobs NumberOfClusters
96                 Sets the maximum number of clusters within the DAG that  will
97                 be  submitted to HTCondor at one time. Note that for this ar‐
98                 gument, each cluster counts as one job, no  matter  how  many
99                 individual  procs  are  in the cluster. NumberOfClusters is a
100                 non-negative integer. If this option is omitted,  the  number
101                 of  clusters  is  limited  by the configuration variable DAG‐
102                 MAN_MAX_JOBS_SUBMITTED
103                   (see  admin-manual/configuration-macros:configuration  file
104                 entries for dagman), which defaults to 0 (unlimited).
105
106          -dagman DagmanExecutable
107                 Allows  the  specification of an alternate condor_dagman exe‐
108                 cutable to be used instead of the one  found  in  the  user's
109                 path. This must be a fully qualified path.
110
111          -maxpre NumberOfPreScripts
112                 Sets  the  maximum  number of PRE scripts within the DAG that
113                 may be running at one time. NumberOfPreScripts is a non-nega‐
114                 tive  integer.   If this option is omitted, the number of PRE
115                 scripts  is  limited  by  the  configuration  variable   DAG‐
116                 MAN_MAX_PRE_SCRIPTS        (see       admin-manual/configura‐
117                 tion-macros:configuration file entries for dagman), which de‐
118                 faults to 20.
119
120          -maxpost NumberOfPostScripts
121                 Sets  the  maximum number of POST scripts within the DAG that
122                 may be running at one time. NumberOfPostScripts is a non-neg‐
123                 ative  integer. If this option is omitted, the number of POST
124                 scripts  is  limited  by  the  configuration  variable   DAG‐
125                 MAN_MAX_POST_SCRIPTS
126                   (see  admin-manual/configuration-macros:configuration  file
127                 entries for dagman), which defaults to 20.
128
129          -notification value
130                 Sets the e-mail notification for DAGMan itself. This informa‐
131                 tion will be used within the HTCondor submit description file
132                 for DAGMan.  This file is produced by condor_submit_dag.  See
133                 the  description  of notification within condor_submit manual
134                 page for a specification of value.
135
136          -noeventchecks
137                 This argument is no longer used; it is now ignored. Its func‐
138                 tionality  is now implemented by the DAGMAN_ALLOW_EVENTS con‐
139                 figuration variable.
140
141          -allowlogerror
142                 As of verson 8.5.5 this argument is no longer supported,  and
143                 setting it will generate a warning.
144
145          -r schedd_name
146                 Submit  condor_dagman  to  a remote machine, specifically the
147                 condor_schedd daemon on that machine. The  condor_dagman  job
148                 will not run on the local condor_schedd (the submit machine),
149                 but on the specified one. This is implemented using the  -re‐
150                 mote  option to condor_submit. Note that this option does not
151                 currently specify input files for condor_dagman, nor the  in‐
152                 dividual nodes to be taken along! It is assumed that any nec‐
153                 essary files will be present on the remote computer, possibly
154                 via  a  shared file system between the local computer and the
155                 remote computer. It is also necessary that the user  has  ap‐
156                 propriate  permissions to submit a job to the remote machine;
157                 the permissions are the same as those required  to  use  con‐
158                 dor_submit  's  -remote option. If other options are desired,
159                 including transfer of other input files, consider  using  the
160                 -no_submit  option,  modifying  the resulting submit file for
161                 specific needs, and then using condor_submit on that.
162
163          -debug level
164                 Passes the the level of  debugging  output  desired  to  con‐
165                 dor_dagman.  level  is  an integer, with values of 0-7 inclu‐
166                 sive, where 7 is the most verbose output. See the condor_dag‐
167                 man manual page for detailed descriptions of these values. If
168                 not specified, no -debug v alue is passed to condor_dagman.
169
170          -usedagdir
171                 This optional argument causes condor_dagman to run each spec‐
172                 ified  DAG as if condor_submit_dag had been run in the direc‐
173                 tory containing that DAG file. This  option  is  most  useful
174                 when  running  multiple  DAGs in a single condor_dagman. Note
175                 that the -usedagdir flag must not be  used  when  running  an
176                 old-style Rescue DAG.
177
178          -outfile_dir directory
179                 Specifies the directory in which the .dagman.out file will be
180                 written. The directory may be specified relative to the  cur‐
181                 rent  working  directory as condor_submit_dag is executed, or
182                 specified with an absolute path.  Without  this  option,  the
183                 .dagman.out file is placed in the same directory as the first
184                 DAG input file listed on the command line.
185
186          -config ConfigFileName
187                 Specifies a configuration file to be  used  for  this  DAGMan
188                 run.  Note  that  the  options specified in the configuration
189                 file apply to all DAGs if multiple DAGs are  specified.  Fur‐
190                 ther  note that it is a fatal error if the configuration file
191                 specified by this option conflicts with a configuration  file
192                 specified in any of the DAG files, if they specify one.
193
194          -insert_sub_file FileName
195                 Specifies  a file to insert into the .condor.sub file created
196                 by condor_submit_dag. The specified file  must  contain  only
197                 legal  submit  file  commands. Only one file can be inserted.
198                 (If both the  DAGMAN_INSERT_SUB_FILE  configuration  variable
199                 and  -insert_sub_file  are  specified, -insert_sub_file over‐
200                 rides DAGMAN_INSERT_SUB_FILE.) The specified file is inserted
201                 into the .condor.sub file before the Queue command and before
202                 any commands specified with the -append option.
203
204          -append Command
205                 Specifies a command to append to the .condor.sub file created
206                 by  condor_submit_dag.  The  specified command is appended to
207                 the .condor.sub file immediately before  the  Queue  command.
208                 Multiple  commands  are specified by using the -append option
209                 multiple times. Each new command is given in a separate  -ap‐
210                 pend  option.   Commands with spaces in them must be enclosed
211                 in double quotes.  Commands specified with the -append option
212                 are  appended to the .condor.sub file after commands inserted
213                 from a file specified by the -insert_sub_file option  or  the
214                 DAGMAN_INSERT_SUB_FILE configuration variable, so the -append
215                 command(s) will override commands from the inserted  file  if
216                 the commands conflict.
217
218          -batch-name batch_name
219                 Set  the  batch name for this DAG/workflow. The batch name is
220                 displayed by condor_q -batch. It is intended for use by users
221                 to  give meaningful names to their workflows and to influence
222                 how condor_q groups jobs for display. As  of  version  8.5.5,
223                 the  batch  name  set with this argument is propagated to all
224                 node jobs of the given DAG (including  sub-DAGs),  overriding
225                 any batch names set in the individual submit files. Note: set
226                 the batch name to ' ' (space) to avoid overriding batch names
227                 specified in node job submit files.  If no batch name is set,
228                 the batch name defaults to DagFile +cluster (where DagFile is
229                 the  primary DAG file of the top-level DAGMan, and cluster is
230                 the HTCondor cluster of the top-level  DAGMan);  the  default
231                 will override any lower-level batch names.
232
233          -autorescue 0|1
234                 Whether  to  automatically  run the newest rescue DAG for the
235                 given DAG file, if one exists (0 = false, 1 = true).
236
237          -dorescuefrom number
238                 Forces condor_dagman to run the specified rescue  DAG  number
239                 for the given DAG. A value of 0 is the same as not specifying
240                 this option. Specifying a non-existent rescue DAG is a  fatal
241                 error.
242
243          -allowversionmismatch
244                 This  optional  argument causes condor_dagman to allow a ver‐
245                 sion mismatch between  condor_dagman  itself  and  the  .con‐
246                 dor.sub  file  produced  by  condor_submit_dag  (or, in other
247                 words, between condor_submit_dag and condor_dagman). WARNING!
248                 This  option should be used only if absolutely necessary. Al‐
249                 lowing version mismatches can cause subtle problems when run‐
250                 ning  DAGs.  (Note  that,  starting  with version 7.4.0, con‐
251                 dor_dagman no longer requires an exact version match  between
252                 itself and the .condor.sub file.  Instead, a "minimum compat‐
253                 ible version" is defined, and any .condor.sub  file  of  that
254                 version or newer is accepted.)
255
256          -no_recurse
257                 This  optional  argument  causes condor_submit_dag to not run
258                 itself recursively on nested DAGs (this is now  the  default;
259                 this flag has been kept mainly for backwards compatibility).
260
261          -do_recurse
262                 This optional argument causes condor_submit_dag to run itself
263                 recursively on nested DAGs. The default is now that  it  does
264                 not run itself recursively; instead the .condor.sub files for
265                 nested DAGs are generated "lazily" by  condor_dagman  itself.
266                 DAG  nodes specified with the SUBDAG EXTERNAL keyword or with
267                 submit file names ending in .condor.sub are considered nested
268                 DAGs.  The DAGMAN_GENERATE_SUBDAG_SUBMITS configuration vari‐
269                 able may be relevant.
270
271          -update_submit
272                 This optional argument causes an existing .condor.sub file to
273                 not be treated as an error; rather, the .condor.sub file will
274                 be overwritten, but the existing values of  -maxjobs,  -maxi‐
275                 dle, -maxpre, and -maxpost will be preserved.
276
277          -import_env
278                 This optional argument causes condor_submit_dag to import the
279                 current environment into the environment command of the .con‐
280                 dor.sub file it generates.
281
282          -DumpRescue
283                 This  optional  argument  tells  condor_dagman to immediately
284                 dump a rescue DAG and then exit, as opposed to actually  run‐
285                 ning  the  DAG.  This feature is mainly intended for testing.
286                 The Rescue DAG file is produced  whether  or  not  there  are
287                 parse errors reading the original DAG input file. The name of
288                 the file differs if there was a parse error.
289
290          -valgrind
291                 This optional argument causes  the  submit  description  file
292                 generated for the submission of condor_dagman to be modified.
293                 The executable becomes valgrind run on condor_dagman, with  a
294                 specific set of arguments intended for testing condor_dagman.
295                 Note that this argument  is  intended  for  testing  purposes
296                 only.  Using  the -valgrind option without the necessary val‐
297                 grind software installed will cause the DAG to fail.  If  the
298                 DAG does run, it will run much more slowly than usual.
299
300          -DontAlwaysRunPost
301                 This  option causes the submit description file generated for
302                 the submission of condor_dagman to  be  modified.  It  causes
303                 condor_dagman to not run the POST script of a node if the PRE
304                 script fails. (This was the default behavior prior to  HTCon‐
305                 dor  version  7.7.2,  and  is again the default behavior from
306                 version 8.5.4 onwards.)
307
308          -AlwaysRunPost
309                 This option causes the submit description file generated  for
310                 the  submission  of  condor_dagman  to be modified. It causes
311                 condor_dagman to always run the POST script of a  node,  even
312                 if  the  PRE script fails. (This was the default behavior for
313                 HTCondor version 7.7.2 through version 8.5.3.)
314
315          -priority number
316                 Sets the minimum job priority of node jobs submitted and run‐
317                 ning  under  the  condor_dagman  job  submitted  by this con‐
318                 dor_submit_dag command.
319
320          -dont_use_default_node_log
321                 This option is disabled as of HTCondor  version  8.3.1.  This
322                 causes  a  compatibility error if the HTCondor version number
323                 of the condor_schedd is 7.9.0 or older.  Tells  condor_dagman
324                 to  use the file specified by the job ClassAd attribute User‐
325                 Log to monitor job status. If this command line  argument  is
326                 used,  then  the  job event log file cannot be defined with a
327                 macro.
328
329          -schedd-daemon-ad-file FileName
330                 Specifies a full path to a daemon ad file dropped by  a  con‐
331                 dor_schedd.  Therefore  this  allows submission to a specific
332                 scheduler if several are available without repeatedly  query‐
333                 ing  the  condor_collector.  The  value for this argument de‐
334                 faults to the configuration attribute SCHEDD_DAEMON_AD_FILE.
335
336          -schedd-address-file FileName
337                 Specifies a full path to an address file dropped  by  a  con‐
338                 dor_schedd.  Therefore  this  allows submission to a specific
339                 scheduler if several are available without repeatedly  query‐
340                 ing  the  condor_collector.  The  value for this argument de‐
341                 faults to the configuration attribute SCHEDD_ADDRESS_FILE.
342
343          -suppress_notification
344                 Causes jobs submitted by condor_dagman to not send email  no‐
345                 tification  for  events.  The  same effect can be achieved by
346                 setting configuration variable DAGMAN_SUPPRESS_NOTIFICATION
347                   to True. This command line option  is  independent  of  the
348                 -notification  command  line option, which controls notifica‐
349                 tion for the condor_dagman job itself.
350
351          -dont_suppress_notification
352                 Causes jobs submitted by condor_dagman to  defer  to  content
353                 within  the  submit  description  file  when deciding to send
354                 email  notification  for  events.  The  same  effect  can  be
355                 achieved   by   setting  configuration  variable  DAGMAN_SUP‐
356                 PRESS_NOTIFICATION
357                   to False. This command line flag is independent of the -no‐
358                 tification  command  line option, which controls notification
359                 for the condor_dagman job itself. If both  -dont_suppress_no‐
360                 tification  and -suppress_notification are specified with the
361                 same command line, the last argument is used.
362
363          -DoRecovery
364                 Causes condor_dagman to start in recovery mode.  (This  means
365                 that  it  reads the relevant job user log(s) and "catches up"
366                 to the given DAG's previous state before submitting  any  new
367                 jobs.)
368

EXIT STATUS

370       condor_submit_dag  will  exit with a status value of 0 (zero) upon suc‐
371       cess, and it will exit with the value 1 (one) upon failure.
372

EXAMPLES

374       To run a single DAG:
375
376          % condor_submit_dag diamond.dag
377
378       To run a DAG when it has already been run and the output files exist:
379
380          % condor_submit_dag -force diamond.dag
381
382       To run a DAG, limiting the number of idle node jobs in  the  DAG  to  a
383       maximum of five:
384
385          % condor_submit_dag -maxidle 5 diamond.dag
386
387       To  run  a DAG, limiting the number of concurrent PRE scripts to 10 and
388       the number of concurrent POST scripts to five:
389
390          % condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
391
392       To run two DAGs, each of which is set up to run in its own directory:
393
394          % condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
395

AUTHOR

397       HTCondor Team
398
400       1990-2022, Center for High Throughput Computing, Computer Sciences  De‐
401       partment,  University  of  Wisconsin-Madison, Madison, WI, US. Licensed
402       under the Apache License, Version 2.0.
403
404
405
406
4078.8                              Jan 19, 2022             CONDOR_SUBMIT_DAG(1)
Impressum