1CONDOR_SUBMIT_DAG(1) HTCondor Manual CONDOR_SUBMIT_DAG(1)
2
6 condor_submit_dag - HTCondor Manual
7 Manage and queue jobs within a specified DAG for execution on remote
9 machines
10
13 condor_submit_dag [-help | -version ]
14 condor_submit_dag [-no_submit ] [-verbose ] [-force ] [-maxidle Num‐
16 berOfProcs] [-maxjobs NumberOfClusters] [-dagman DagmanExecutable]
17 [-maxpre NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notifica‐
18 tion value] [-noeventchecks ] [-allowlogerror ] [-r schedd_name]
19 [-debug level] [-usedagdir ] [-outfile_dir directory] [-config Config‐
20 FileName] [-insert_sub_file FileName] [-append Command] [-batch-name
21 batch_name] [-autorescue 0|1] [-dorescuefrom number] [-allowversionmis‐
22 match ] [-no_recurse ] [-do_recurse ] [-update_submit ] [-import_env ]
23 [-DumpRescue ] [-valgrind ] [-DontAlwaysRunPost ] [-AlwaysRunPost ]
24 [-priority number] [-dont_use_default_node_log ] [-schedd-dae‐
25 mon-ad-file FileName] [-schedd-address-file FileName] [-suppress_noti‐
26 fication ] [-dont_suppress_notification ] [-DoRecovery ] DAGInputFile1
27 [DAGInputFile2 ...DAGInputFileN ]
28
30 condor_submit_dag is the program for submitting a DAG (directed acyclic
31 graph) of jobs for execution under HTCondor. The program enforces the
32 job dependencies defined in one or more DAGInputFile s. Each DAGInput‐
33 File contains commands to direct the submission of jobs implied by the
34 nodes of a DAG to HTCondor. Extensive documentation is in the HTCondor
35 User Manual section on DAGMan.
36 Some options may be specified on the command line or in the configura‐
40 tion or in a node job's submit description file. Precedence is given to
41 command line options or configuration over settings from a submit
42 description file. An example is e-mail notifications. When configura‐
43 tion variable DAGMAN_SUPPRESS_NOTIFICATION
44 is its default value of True, and a node job's submit description
45 file contains
46 notification = Complete
48 e-mail will not be sent upon completion, as the value of DAGMAN_SUP‐
50 PRESS_NOTIFICATION is enforced.
51
53 -help Display usage information and exit.
54 -version
56 Display version information and exit.
57 -no_submit
59 Produce the HTCondor submit description file for DAGMan, but
60 do not submit DAGMan as an HTCondor job.
61 -verbose
63 Cause condor_submit_dag to give verbose error messages.
64 -force Require condor_submit_dag to overwrite the files that it pro‐
66 duces, if the files already exist. Note that dagman.out will
67 be appended to, not overwritten. If new-style rescue DAG mode
68 is in effect, and any new-style rescue DAGs exist, the -force
69 flag will cause them to be renamed, and the original DAG will
70 be run. If old-style rescue DAG mode is in effect, any exist‐
71 ing old-style rescue DAGs will be deleted, and the original
72 DAG will be run.
73 -maxidle NumberOfProcs
75 Sets the maximum number of idle procs allowed before con‐
76 dor_dagman stops submitting more node jobs. Note that for
77 this argument, each individual proc within a cluster counts
78 as a towards the limit, which is inconsistent with -maxjobs .
79 Once idle procs start to run, condor_dagman will resume sub‐
80 mitting jobs once the number of idle procs falls below the
81 specified limit. NumberOfProcs is a non-negative integer. If
82 this option is omitted, the number of idle procs is limited
83 by the configuration variable DAGMAN_MAX_JOBS_IDLE
84 (see admin-manual/configuration-macros:configuration file
85 entries for dagman), which defaults to 1000. To disable this
86 limit, set NumberOfProcs to 0. Note that submit description
87 files that queue multiple procs can cause the NumberOfProcs
88 limit to be exceeded. Setting queue 5000 in the submit
89 description file, where -maxidle is set to 250 will result in
90 a cluster of 5000 new procs being submitted to the con‐
91 dor_schedd, not 250. In this case, condor_dagman will resume
92 submitting jobs when the number of idle procs falls below
93 250.
94 -maxjobs NumberOfClusters
96 Sets the maximum number of clusters within the DAG that will
97 be submitted to HTCondor at one time. Note that for this
98 argument, each cluster counts as one job, no matter how many
99 individual procs are in the cluster. NumberOfClusters is a
100 non-negative integer. If this option is omitted, the number
101 of clusters is limited by the configuration variable DAG‐
102 MAN_MAX_JOBS_SUBMITTED
103 (see admin-manual/configuration-macros:configuration file
104 entries for dagman), which defaults to 0 (unlimited).
105 -dagman DagmanExecutable
107 Allows the specification of an alternate condor_dagman exe‐
108 cutable to be used instead of the one found in the user's
109 path. This must be a fully qualified path.
110 -maxpre NumberOfPreScripts
112 Sets the maximum number of PRE scripts within the DAG that
113 may be running at one time. NumberOfPreScripts is a non-nega‐
114 tive integer. If this option is omitted, the number of PRE
115 scripts is limited by the configuration variable DAG‐
116 MAN_MAX_PRE_SCRIPTS (see admin-manual/configura‐
117 tion-macros:configuration file entries for dagman), which
118 defaults to 20.
119 -maxpost NumberOfPostScripts
121 Sets the maximum number of POST scripts within the DAG that
122 may be running at one time. NumberOfPostScripts is a non-neg‐
123 ative integer. If this option is omitted, the number of POST
124 scripts is limited by the configuration variable DAG‐
125 MAN_MAX_POST_SCRIPTS
126 (see admin-manual/configuration-macros:configuration file
127 entries for dagman), which defaults to 20.
128 -notification value
130 Sets the e-mail notification for DAGMan itself. This informa‐
131 tion will be used within the HTCondor submit description file
132 for DAGMan. This file is produced by condor_submit_dag. See
133 the description of notification within condor_submit manual
134 page for a specification of value.
135 -noeventchecks
137 This argument is no longer used; it is now ignored. Its func‐
138 tionality is now implemented by the DAGMAN_ALLOW_EVENTS con‐
139 figuration variable.
140 -allowlogerror
142 As of verson 8.5.5 this argument is no longer supported, and
143 setting it will generate a warning.
144 -r schedd_name
146 Submit condor_dagman to a remote machine, specifically the
147 condor_schedd daemon on that machine. The condor_dagman job
148 will not run on the local condor_schedd (the submit machine),
149 but on the specified one. This is implemented using the
150 -remote option to condor_submit. Note that this option does
151 not currently specify input files for condor_dagman, nor the
152 individual nodes to be taken along! It is assumed that any
153 necessary files will be present on the remote computer, pos‐
154 sibly via a shared file system between the local computer and
155 the remote computer. It is also necessary that the user has
156 appropriate permissions to submit a job to the remote
157 machine; the permissions are the same as those required to
158 use condor_submit 's -remote option. If other options are
159 desired, including transfer of other input files, consider
160 using the -no_submit option, modifying the resulting submit
161 file for specific needs, and then using condor_submit on
162 that.
163 -debug level
165 Passes the the level of debugging output desired to con‐
166 dor_dagman. level is an integer, with values of 0-7 inclu‐
167 sive, where 7 is the most verbose output. See the condor_dag‐
168 man manual page for detailed descriptions of these values. If
169 not specified, no -debug v alue is passed to condor_dagman.
170 -usedagdir
172 This optional argument causes condor_dagman to run each spec‐
173 ified DAG as if condor_submit_dag had been run in the direc‐
174 tory containing that DAG file. This option is most useful
175 when running multiple DAGs in a single condor_dagman. Note
176 that the -usedagdir flag must not be used when running an
177 old-style Rescue DAG.
178 -outfile_dir directory
180 Specifies the directory in which the .dagman.out file will be
181 written. The directory may be specified relative to the cur‐
182 rent working directory as condor_submit_dag is executed, or
183 specified with an absolute path. Without this option, the
184 .dagman.out file is placed in the same directory as the first
185 DAG input file listed on the command line.
186 -config ConfigFileName
188 Specifies a configuration file to be used for this DAGMan
189 run. Note that the options specified in the configuration
190 file apply to all DAGs if multiple DAGs are specified. Fur‐
191 ther note that it is a fatal error if the configuration file
192 specified by this option conflicts with a configuration file
193 specified in any of the DAG files, if they specify one.
194 -insert_sub_file FileName
196 Specifies a file to insert into the .condor.sub file created
197 by condor_submit_dag. The specified file must contain only
198 legal submit file commands. Only one file can be inserted.
199 (If both the DAGMAN_INSERT_SUB_FILE configuration variable
200 and -insert_sub_file are specified, -insert_sub_file over‐
201 rides DAGMAN_INSERT_SUB_FILE.) The specified file is inserted
202 into the .condor.sub file before the Queue command and before
203 any commands specified with the -append option.
204 -append Command
206 Specifies a command to append to the .condor.sub file created
207 by condor_submit_dag. The specified command is appended to
208 the .condor.sub file immediately before the Queue command.
209 Multiple commands are specified by using the -append option
210 multiple times. Each new command is given in a separate
211 -append option. Commands with spaces in them must be
212 enclosed in double quotes. Commands specified with the
213 -append option are appended to the .condor.sub file after
214 commands inserted from a file specified by the
215 -insert_sub_file option or the DAGMAN_INSERT_SUB_FILE config‐
216 uration variable, so the -append command(s) will override
217 commands from the inserted file if the commands conflict.
218 -batch-name batch_name
220 Set the batch name for this DAG/workflow. The batch name is
221 displayed by condor_q -batch. It is intended for use by users
222 to give meaningful names to their workflows and to influence
223 how condor_q groups jobs for display. As of version 8.5.5,
224 the batch name set with this argument is propagated to all
225 node jobs of the given DAG (including sub-DAGs), overriding
226 any batch names set in the individual submit files. Note: set
227 the batch name to ' ' (space) to avoid overriding batch names
228 specified in node job submit files. If no batch name is set,
229 the batch name defaults to DagFile +cluster (where DagFile is
230 the primary DAG file of the top-level DAGMan, and cluster is
231 the HTCondor cluster of the top-level DAGMan); the default
232 will override any lower-level batch names.
233 -autorescue 0|1
235 Whether to automatically run the newest rescue DAG for the
236 given DAG file, if one exists (0 = false, 1 = true).
237 -dorescuefrom number
239 Forces condor_dagman to run the specified rescue DAG number
240 for the given DAG. A value of 0 is the same as not specifying
241 this option. Specifying a non-existent rescue DAG is a fatal
242 error.
243 -allowversionmismatch
245 This optional argument causes condor_dagman to allow a ver‐
246 sion mismatch between condor_dagman itself and the .con‐
247 dor.sub file produced by condor_submit_dag (or, in other
248 words, between condor_submit_dag and condor_dagman). WARNING!
249 This option should be used only if absolutely necessary.
250 Allowing version mismatches can cause subtle problems when
251 running DAGs. (Note that, starting with version 7.4.0, con‐
252 dor_dagman no longer requires an exact version match between
253 itself and the .condor.sub file. Instead, a "minimum compat‐
254 ible version" is defined, and any .condor.sub file of that
255 version or newer is accepted.)
256 -no_recurse
258 This optional argument causes condor_submit_dag to not run
259 itself recursively on nested DAGs (this is now the default;
260 this flag has been kept mainly for backwards compatibility).
261 -do_recurse
263 This optional argument causes condor_submit_dag to run itself
264 recursively on nested DAGs. The default is now that it does
265 not run itself recursively; instead the .condor.sub files for
266 nested DAGs are generated "lazily" by condor_dagman itself.
267 DAG nodes specified with the SUBDAG EXTERNAL keyword or with
268 submit file names ending in .condor.sub are considered nested
269 DAGs. The DAGMAN_GENERATE_SUBDAG_SUBMITS configuration vari‐
270 able may be relevant.
271 -update_submit
273 This optional argument causes an existing .condor.sub file to
274 not be treated as an error; rather, the .condor.sub file will
275 be overwritten, but the existing values of -maxjobs, -maxi‐
276 dle, -maxpre, and -maxpost will be preserved.
277 -import_env
279 This optional argument causes condor_submit_dag to import the
280 current environment into the environment command of the .con‐
281 dor.sub file it generates.
282 -DumpRescue
284 This optional argument tells condor_dagman to immediately
285 dump a rescue DAG and then exit, as opposed to actually run‐
286 ning the DAG. This feature is mainly intended for testing.
287 The Rescue DAG file is produced whether or not there are
288 parse errors reading the original DAG input file. The name of
289 the file differs if there was a parse error.
290 -valgrind
292 This optional argument causes the submit description file
293 generated for the submission of condor_dagman to be modified.
294 The executable becomes valgrind run on condor_dagman, with a
295 specific set of arguments intended for testing condor_dagman.
296 Note that this argument is intended for testing purposes
297 only. Using the -valgrind option without the necessary val‐
298 grind software installed will cause the DAG to fail. If the
299 DAG does run, it will run much more slowly than usual.
300 -DontAlwaysRunPost
302 This option causes the submit description file generated for
303 the submission of condor_dagman to be modified. It causes
304 condor_dagman to not run the POST script of a node if the PRE
305 script fails. (This was the default behavior prior to HTCon‐
306 dor version 7.7.2, and is again the default behavior from
307 version 8.5.4 onwards.)
308 -AlwaysRunPost
310 This option causes the submit description file generated for
311 the submission of condor_dagman to be modified. It causes
312 condor_dagman to always run the POST script of a node, even
313 if the PRE script fails. (This was the default behavior for
314 HTCondor version 7.7.2 through version 8.5.3.)
315 -priority number
317 Sets the minimum job priority of node jobs submitted and run‐
318 ning under the condor_dagman job submitted by this con‐
319 dor_submit_dag command.
320 -dont_use_default_node_log
322 This option is disabled as of HTCondor version 8.3.1. This
323 causes a compatibility error if the HTCondor version number
324 of the condor_schedd is 7.9.0 or older. Tells condor_dagman
325 to use the file specified by the job ClassAd attribute User‐
326 Log to monitor job status. If this command line argument is
327 used, then the job event log file cannot be defined with a
328 macro.
329 -schedd-daemon-ad-file FileName
331 Specifies a full path to a daemon ad file dropped by a con‐
332 dor_schedd. Therefore this allows submission to a specific
333 scheduler if several are available without repeatedly query‐
334 ing the condor_collector. The value for this argument
335 defaults to the configuration attribute SCHEDD_DAE‐
336 MON_AD_FILE.
337 -schedd-address-file FileName
339 Specifies a full path to an address file dropped by a con‐
340 dor_schedd. Therefore this allows submission to a specific
341 scheduler if several are available without repeatedly query‐
342 ing the condor_collector. The value for this argument
343 defaults to the configuration attribute SCHEDD_ADDRESS_FILE.
344 -suppress_notification
346 Causes jobs submitted by condor_dagman to not send email
347 notification for events. The same effect can be achieved by
348 setting configuration variable DAGMAN_SUPPRESS_NOTIFICATION
349 to True. This command line option is independent of the
350 -notification command line option, which controls notifica‐
351 tion for the condor_dagman job itself.
352 -dont_suppress_notification
354 Causes jobs submitted by condor_dagman to defer to content
355 within the submit description file when deciding to send
356 email notification for events. The same effect can be
357 achieved by setting configuration variable DAGMAN_SUP‐
358 PRESS_NOTIFICATION
359 to False. This command line flag is independent of the
360 -notification command line option, which controls notifica‐
361 tion for the condor_dagman job itself. If both -dont_sup‐
362 press_notification and -suppress_notification are specified
363 with the same command line, the last argument is used.
364 -DoRecovery
366 Causes condor_dagman to start in recovery mode. (This means
367 that it reads the relevant job user log(s) and "catches up"
368 to the given DAG's previous state before submitting any new
369 jobs.)
370
372 condor_submit_dag will exit with a status value of 0 (zero) upon suc‐
373 cess, and it will exit with the value 1 (one) upon failure.
374
376 To run a single DAG:
377 % condor_submit_dag diamond.dag
379 To run a DAG when it has already been run and the output files exist:
381 % condor_submit_dag -force diamond.dag
383 To run a DAG, limiting the number of idle node jobs in the DAG to a
385 maximum of five:
386 % condor_submit_dag -maxidle 5 diamond.dag
388 To run a DAG, limiting the number of concurrent PRE scripts to 10 and
390 the number of concurrent POST scripts to five:
391 % condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
393 To run two DAGs, each of which is set up to run in its own directory:
395 % condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
397
399 HTCondor Team
400
402 1990-2020, Center for High Throughput Computing, Computer Sciences
403 Department, University of Wisconsin-Madison, Madison, WI, US. Licensed
404 under the Apache License, Version 2.0.
4058.8 Aug 06, 2020 CONDOR_SUBMIT_DAG(1)