1CONDOR_SUBMIT_DAG(1) HTCondor Manual CONDOR_SUBMIT_DAG(1)
2
6 condor_submit_dag - HTCondor Manual
7 Manage and queue jobs within a specified DAG for execution on remote
9 machines
10
13 condor_submit_dag [-help | -version ]
14 condor_submit_dag [-no_submit ] [-verbose ] [-force ] [-maxidle Num‐
16 berOfProcs] [-maxjobs NumberOfClusters] [-dagman DagmanExecutable]
17 [-maxpre NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notifica‐
18 tion value] [-noeventchecks ] [-allowlogerror ] [-r schedd_name] [-de‐
19 bug level] [-usedagdir ] [-outfile_dir directory] [-config ConfigFile‐
20 Name] [-insert_sub_file FileName] [-append Command] [-batch-name
21 batch_name] [-autorescue 0|1] [-dorescuefrom number] [-allowversionmis‐
22 match ] [-no_recurse ] [-do_recurse ] [-update_submit ] [-import_env ]
23 [-DumpRescue ] [-valgrind ] [-DontAlwaysRunPost ] [-AlwaysRunPost ]
24 [-priority number] [-dont_use_default_node_log ] [-schedd-dae‐
25 mon-ad-file FileName] [-schedd-address-file FileName] [-suppress_noti‐
26 fication ] [-dont_suppress_notification ] [-DoRecovery ] DAGInputFile1
27 [DAGInputFile2 ...DAGInputFileN ]
28
30 condor_submit_dag is the program for submitting a DAG (directed acyclic
31 graph) of jobs for execution under HTCondor. The program enforces the
32 job dependencies defined in one or more DAGInputFile s. Each DAGInput‐
33 File contains commands to direct the submission of jobs implied by the
34 nodes of a DAG to HTCondor. Extensive documentation is in the HTCondor
35 User Manual section on DAGMan.
36 Some options may be specified on the command line or in the configura‐
40 tion or in a node job's submit description file. Precedence is given to
41 command line options or configuration over settings from a submit de‐
42 scription file. An example is e-mail notifications. When configuration
43 variable DAGMAN_SUPPRESS_NOTIFICATION
44 is its default value of True, and a node job's submit description
45 file contains
46 notification = Complete
48 e-mail will not be sent upon completion, as the value of DAGMAN_SUP‐
50 PRESS_NOTIFICATION is enforced.
51
53 -help Display usage information and exit.
54 -version
56 Display version information and exit.
57 -no_submit
59 Produce the HTCondor submit description file for DAGMan, but
60 do not submit DAGMan as an HTCondor job.
61 -verbose
63 Cause condor_submit_dag to give verbose error messages.
64 -force Require condor_submit_dag to overwrite the files that it pro‐
66 duces, if the files already exist. Note that dagman.out will
67 be appended to, not overwritten. If new-style rescue DAG mode
68 is in effect, and any new-style rescue DAGs exist, the -force
69 flag will cause them to be renamed, and the original DAG will
70 be run. If old-style rescue DAG mode is in effect, any exist‐
71 ing old-style rescue DAGs will be deleted, and the original
72 DAG will be run.
73 -maxidle NumberOfProcs
75 Sets the maximum number of idle procs allowed before con‐
76 dor_dagman stops submitting more node jobs. Note that for
77 this argument, each individual proc within a cluster counts
78 as a towards the limit, which is inconsistent with -maxjobs .
79 Once idle procs start to run, condor_dagman will resume sub‐
80 mitting jobs once the number of idle procs falls below the
81 specified limit. NumberOfProcs is a non-negative integer. If
82 this option is omitted, the number of idle procs is limited
83 by the configuration variable DAGMAN_MAX_JOBS_IDLE
84 (see admin-manual/configuration-macros:configuration file
85 entries for dagman), which defaults to 1000. To disable this
86 limit, set NumberOfProcs to 0. Note that submit description
87 files that queue multiple procs can cause the NumberOfProcs
88 limit to be exceeded. Setting queue 5000 in the submit de‐
89 scription file, where -maxidle is set to 250 will result in a
90 cluster of 5000 new procs being submitted to the con‐
91 dor_schedd, not 250. In this case, condor_dagman will resume
92 submitting jobs when the number of idle procs falls below
93 250.
94 -maxjobs NumberOfClusters
96 Sets the maximum number of clusters within the DAG that will
97 be submitted to HTCondor at one time. Note that for this ar‐
98 gument, each cluster counts as one job, no matter how many
99 individual procs are in the cluster. NumberOfClusters is a
100 non-negative integer. If this option is omitted, the number
101 of clusters is limited by the configuration variable DAG‐
102 MAN_MAX_JOBS_SUBMITTED
103 (see admin-manual/configuration-macros:configuration file
104 entries for dagman), which defaults to 0 (unlimited).
105 -dagman DagmanExecutable
107 Allows the specification of an alternate condor_dagman exe‐
108 cutable to be used instead of the one found in the user's
109 path. This must be a fully qualified path.
110 -maxpre NumberOfPreScripts
112 Sets the maximum number of PRE scripts within the DAG that
113 may be running at one time. NumberOfPreScripts is a non-nega‐
114 tive integer. If this option is omitted, the number of PRE
115 scripts is limited by the configuration variable DAG‐
116 MAN_MAX_PRE_SCRIPTS (see admin-manual/configura‐
117 tion-macros:configuration file entries for dagman), which de‐
118 faults to 20.
119 -maxpost NumberOfPostScripts
121 Sets the maximum number of POST scripts within the DAG that
122 may be running at one time. NumberOfPostScripts is a non-neg‐
123 ative integer. If this option is omitted, the number of POST
124 scripts is limited by the configuration variable DAG‐
125 MAN_MAX_POST_SCRIPTS
126 (see admin-manual/configuration-macros:configuration file
127 entries for dagman), which defaults to 20.
128 -notification value
130 Sets the e-mail notification for DAGMan itself. This informa‐
131 tion will be used within the HTCondor submit description file
132 for DAGMan. This file is produced by condor_submit_dag. See
133 the description of notification within condor_submit manual
134 page for a specification of value.
135 -noeventchecks
137 This argument is no longer used; it is now ignored. Its func‐
138 tionality is now implemented by the DAGMAN_ALLOW_EVENTS con‐
139 figuration variable.
140 -allowlogerror
142 As of verson 8.5.5 this argument is no longer supported, and
143 setting it will generate a warning.
144 -r schedd_name
146 Submit condor_dagman to a remote machine, specifically the
147 condor_schedd daemon on that machine. The condor_dagman job
148 will not run on the local condor_schedd (the submit machine),
149 but on the specified one. This is implemented using the -re‐
150 mote option to condor_submit. Note that this option does not
151 currently specify input files for condor_dagman, nor the in‐
152 dividual nodes to be taken along! It is assumed that any nec‐
153 essary files will be present on the remote computer, possibly
154 via a shared file system between the local computer and the
155 remote computer. It is also necessary that the user has ap‐
156 propriate permissions to submit a job to the remote machine;
157 the permissions are the same as those required to use con‐
158 dor_submit 's -remote option. If other options are desired,
159 including transfer of other input files, consider using the
160 -no_submit option, modifying the resulting submit file for
161 specific needs, and then using condor_submit on that.
162 -debug level
164 Passes the the level of debugging output desired to con‐
165 dor_dagman. level is an integer, with values of 0-7 inclu‐
166 sive, where 7 is the most verbose output. See the condor_dag‐
167 man manual page for detailed descriptions of these values. If
168 not specified, no -debug v alue is passed to condor_dagman.
169 -usedagdir
171 This optional argument causes condor_dagman to run each spec‐
172 ified DAG as if condor_submit_dag had been run in the direc‐
173 tory containing that DAG file. This option is most useful
174 when running multiple DAGs in a single condor_dagman. Note
175 that the -usedagdir flag must not be used when running an
176 old-style Rescue DAG.
177 -outfile_dir directory
179 Specifies the directory in which the .dagman.out file will be
180 written. The directory may be specified relative to the cur‐
181 rent working directory as condor_submit_dag is executed, or
182 specified with an absolute path. Without this option, the
183 .dagman.out file is placed in the same directory as the first
184 DAG input file listed on the command line.
185 -config ConfigFileName
187 Specifies a configuration file to be used for this DAGMan
188 run. Note that the options specified in the configuration
189 file apply to all DAGs if multiple DAGs are specified. Fur‐
190 ther note that it is a fatal error if the configuration file
191 specified by this option conflicts with a configuration file
192 specified in any of the DAG files, if they specify one.
193 -insert_sub_file FileName
195 Specifies a file to insert into the .condor.sub file created
196 by condor_submit_dag. The specified file must contain only
197 legal submit file commands. Only one file can be inserted.
198 (If both the DAGMAN_INSERT_SUB_FILE configuration variable
199 and -insert_sub_file are specified, -insert_sub_file over‐
200 rides DAGMAN_INSERT_SUB_FILE.) The specified file is inserted
201 into the .condor.sub file before the Queue command and before
202 any commands specified with the -append option.
203 -append Command
205 Specifies a command to append to the .condor.sub file created
206 by condor_submit_dag. The specified command is appended to
207 the .condor.sub file immediately before the Queue command.
208 Multiple commands are specified by using the -append option
209 multiple times. Each new command is given in a separate -ap‐
210 pend option. Commands with spaces in them must be enclosed
211 in double quotes. Commands specified with the -append option
212 are appended to the .condor.sub file after commands inserted
213 from a file specified by the -insert_sub_file option or the
214 DAGMAN_INSERT_SUB_FILE configuration variable, so the -append
215 command(s) will override commands from the inserted file if
216 the commands conflict.
217 -batch-name batch_name
219 Set the batch name for this DAG/workflow. The batch name is
220 displayed by condor_q -batch. It is intended for use by users
221 to give meaningful names to their workflows and to influence
222 how condor_q groups jobs for display. As of version 8.5.5,
223 the batch name set with this argument is propagated to all
224 node jobs of the given DAG (including sub-DAGs), overriding
225 any batch names set in the individual submit files. Note: set
226 the batch name to ' ' (space) to avoid overriding batch names
227 specified in node job submit files. If no batch name is set,
228 the batch name defaults to DagFile +cluster (where DagFile is
229 the primary DAG file of the top-level DAGMan, and cluster is
230 the HTCondor cluster of the top-level DAGMan); the default
231 will override any lower-level batch names.
232 -autorescue 0|1
234 Whether to automatically run the newest rescue DAG for the
235 given DAG file, if one exists (0 = false, 1 = true).
236 -dorescuefrom number
238 Forces condor_dagman to run the specified rescue DAG number
239 for the given DAG. A value of 0 is the same as not specifying
240 this option. Specifying a non-existent rescue DAG is a fatal
241 error.
242 -allowversionmismatch
244 This optional argument causes condor_dagman to allow a ver‐
245 sion mismatch between condor_dagman itself and the .con‐
246 dor.sub file produced by condor_submit_dag (or, in other
247 words, between condor_submit_dag and condor_dagman). WARNING!
248 This option should be used only if absolutely necessary. Al‐
249 lowing version mismatches can cause subtle problems when run‐
250 ning DAGs. (Note that, starting with version 7.4.0, con‐
251 dor_dagman no longer requires an exact version match between
252 itself and the .condor.sub file. Instead, a "minimum compat‐
253 ible version" is defined, and any .condor.sub file of that
254 version or newer is accepted.)
255 -no_recurse
257 This optional argument causes condor_submit_dag to not run
258 itself recursively on nested DAGs (this is now the default;
259 this flag has been kept mainly for backwards compatibility).
260 -do_recurse
262 This optional argument causes condor_submit_dag to run itself
263 recursively on nested DAGs. The default is now that it does
264 not run itself recursively; instead the .condor.sub files for
265 nested DAGs are generated "lazily" by condor_dagman itself.
266 DAG nodes specified with the SUBDAG EXTERNAL keyword or with
267 submit file names ending in .condor.sub are considered nested
268 DAGs. The DAGMAN_GENERATE_SUBDAG_SUBMITS configuration vari‐
269 able may be relevant.
270 -update_submit
272 This optional argument causes an existing .condor.sub file to
273 not be treated as an error; rather, the .condor.sub file will
274 be overwritten, but the existing values of -maxjobs, -maxi‐
275 dle, -maxpre, and -maxpost will be preserved.
276 -import_env
278 This optional argument causes condor_submit_dag to import the
279 current environment into the environment command of the .con‐
280 dor.sub file it generates.
281 -DumpRescue
283 This optional argument tells condor_dagman to immediately
284 dump a rescue DAG and then exit, as opposed to actually run‐
285 ning the DAG. This feature is mainly intended for testing.
286 The Rescue DAG file is produced whether or not there are
287 parse errors reading the original DAG input file. The name of
288 the file differs if there was a parse error.
289 -valgrind
291 This optional argument causes the submit description file
292 generated for the submission of condor_dagman to be modified.
293 The executable becomes valgrind run on condor_dagman, with a
294 specific set of arguments intended for testing condor_dagman.
295 Note that this argument is intended for testing purposes
296 only. Using the -valgrind option without the necessary val‐
297 grind software installed will cause the DAG to fail. If the
298 DAG does run, it will run much more slowly than usual.
299 -DontAlwaysRunPost
301 This option causes the submit description file generated for
302 the submission of condor_dagman to be modified. It causes
303 condor_dagman to not run the POST script of a node if the PRE
304 script fails. (This was the default behavior prior to HTCon‐
305 dor version 7.7.2, and is again the default behavior from
306 version 8.5.4 onwards.)
307 -AlwaysRunPost
309 This option causes the submit description file generated for
310 the submission of condor_dagman to be modified. It causes
311 condor_dagman to always run the POST script of a node, even
312 if the PRE script fails. (This was the default behavior for
313 HTCondor version 7.7.2 through version 8.5.3.)
314 -priority number
316 Sets the minimum job priority of node jobs submitted and run‐
317 ning under the condor_dagman job submitted by this con‐
318 dor_submit_dag command.
319 -dont_use_default_node_log
321 This option is disabled as of HTCondor version 8.3.1. This
322 causes a compatibility error if the HTCondor version number
323 of the condor_schedd is 7.9.0 or older. Tells condor_dagman
324 to use the file specified by the job ClassAd attribute User‐
325 Log to monitor job status. If this command line argument is
326 used, then the job event log file cannot be defined with a
327 macro.
328 -schedd-daemon-ad-file FileName
330 Specifies a full path to a daemon ad file dropped by a con‐
331 dor_schedd. Therefore this allows submission to a specific
332 scheduler if several are available without repeatedly query‐
333 ing the condor_collector. The value for this argument de‐
334 faults to the configuration attribute SCHEDD_DAEMON_AD_FILE.
335 -schedd-address-file FileName
337 Specifies a full path to an address file dropped by a con‐
338 dor_schedd. Therefore this allows submission to a specific
339 scheduler if several are available without repeatedly query‐
340 ing the condor_collector. The value for this argument de‐
341 faults to the configuration attribute SCHEDD_ADDRESS_FILE.
342 -suppress_notification
344 Causes jobs submitted by condor_dagman to not send email no‐
345 tification for events. The same effect can be achieved by
346 setting configuration variable DAGMAN_SUPPRESS_NOTIFICATION
347 to True. This command line option is independent of the
348 -notification command line option, which controls notifica‐
349 tion for the condor_dagman job itself.
350 -dont_suppress_notification
352 Causes jobs submitted by condor_dagman to defer to content
353 within the submit description file when deciding to send
354 email notification for events. The same effect can be
355 achieved by setting configuration variable DAGMAN_SUP‐
356 PRESS_NOTIFICATION
357 to False. This command line flag is independent of the -no‐
358 tification command line option, which controls notification
359 for the condor_dagman job itself. If both -dont_suppress_no‐
360 tification and -suppress_notification are specified with the
361 same command line, the last argument is used.
362 -DoRecovery
364 Causes condor_dagman to start in recovery mode. (This means
365 that it reads the relevant job user log(s) and "catches up"
366 to the given DAG's previous state before submitting any new
367 jobs.)
368
370 condor_submit_dag will exit with a status value of 0 (zero) upon suc‐
371 cess, and it will exit with the value 1 (one) upon failure.
372
374 To run a single DAG:
375 % condor_submit_dag diamond.dag
377 To run a DAG when it has already been run and the output files exist:
379 % condor_submit_dag -force diamond.dag
381 To run a DAG, limiting the number of idle node jobs in the DAG to a
383 maximum of five:
384 % condor_submit_dag -maxidle 5 diamond.dag
386 To run a DAG, limiting the number of concurrent PRE scripts to 10 and
388 the number of concurrent POST scripts to five:
389 % condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
391 To run two DAGs, each of which is set up to run in its own directory:
393 % condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
395
397 HTCondor Team
398
400 1990-2022, Center for High Throughput Computing, Computer Sciences De‐
401 partment, University of Wisconsin-Madison, Madison, WI, US. Licensed
402 under the Apache License, Version 2.0.
4038.8 Jan 19, 2022 CONDOR_SUBMIT_DAG(1)