1CONDOR_SUBMIT_DAG(1) HTCondor Manual CONDOR_SUBMIT_DAG(1)
2
6 condor_submit_dag - HTCondor Manual
7 Manage and queue jobs within a specified DAG for execution on remote
9 machines
10
13 condor_submit_dag [-help | -version ]
14 condor_submit_dag [-no_submit ] [-verbose ] [-force ] [-maxidle Num‐
16 berOfProcs] [-maxjobs NumberOfClusters] [-dagman DagmanExecutable]
17 [-maxpre NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notifica‐
18 tion value] [-noeventchecks ] [-allowlogerror ] [-r schedd_name] [-de‐
19 bug level] [-usedagdir ] [-outfile_dir directory] [-config ConfigFile‐
20 Name] [-insert_sub_file FileName] [-append Command] [-batch-name
21 batch_name] [-autorescue 0|1] [-dorescuefrom number] [-allowversionmis‐
22 match ] [-no_recurse ] [-do_recurse ] [-update_submit ] [-import_env ]
23 [-DumpRescue ] [-valgrind ] [-DontAlwaysRunPost ] [-AlwaysRunPost ]
24 [-priority number] [-dont_use_default_node_log ] [-schedd-dae‐
25 mon-ad-file FileName] [-schedd-address-file FileName] [-suppress_noti‐
26 fication ] [-dont_suppress_notification ] [-DoRecovery ] DAGInputFile1
27 [DAGInputFile2 ...DAGInputFileN ]
28
30 condor_submit_dag is the program for submitting a DAG (directed acyclic
31 graph) of jobs for execution under HTCondor. The program enforces the
32 job dependencies defined in one or more DAGInputFile s. Each DAGInput‐
33 File contains commands to direct the submission of jobs implied by the
34 nodes of a DAG to HTCondor. Extensive documentation is in the HTCondor
35 User Manual section on DAGMan.
36 Some options may be specified on the command line or in the configura‐
40 tion or in a node job's submit description file. Precedence is given to
41 command line options or configuration over settings from a submit de‐
42 scription file. An example is e-mail notifications. When configuration
43 variable DAGMAN_SUPPRESS_NOTIFICATION
44 is its default value of True, and a node job's submit description
45 file contains
46 notification = Complete
48 e-mail will not be sent upon completion, as the value of DAGMAN_SUP‐
50 PRESS_NOTIFICATION is enforced.
51
53 -help Display usage information and exit.
54 -version
56 Display version information and exit.
57 -no_submit
59 Produce the HTCondor submit description file for DAGMan, but
60 do not submit DAGMan as an HTCondor job.
61 -verbose
63 Cause condor_submit_dag to give verbose error messages.
64 -force Require condor_submit_dag to overwrite the files that it pro‐
66 duces, if the files already exist. Note that dagman.out will
67 be appended to, not overwritten. If new-style rescue DAG mode
68 is in effect, and any new-style rescue DAGs exist, the -force
69 flag will cause them to be renamed, and the original DAG will
70 be run. If old-style rescue DAG mode is in effect, any exist‐
71 ing old-style rescue DAGs will be deleted, and the original
72 DAG will be run.
73 -maxidle NumberOfProcs
75 Sets the maximum number of idle procs allowed before con‐
76 dor_dagman stops submitting more node jobs. Note that for
77 this argument, each individual proc within a cluster counts
78 as a towards the limit, which is inconsistent with -maxjobs .
79 Once idle procs start to run, condor_dagman will resume sub‐
80 mitting jobs once the number of idle procs falls below the
81 specified limit. NumberOfProcs is a non-negative integer. If
82 this option is omitted, the number of idle procs is limited
83 by the configuration variable DAGMAN_MAX_JOBS_IDLE
84 (see Configuration File Entries for DAGMan), which defaults
85 to 1000. To disable this limit, set NumberOfProcs to 0. Note
86 that submit description files that queue multiple procs can
87 cause the NumberOfProcs limit to be exceeded. Setting queue
88 5000 in the submit description file, where -maxidle is set to
89 250 will result in a cluster of 5000 new procs being submit‐
90 ted to the condor_schedd, not 250. In this case, condor_dag‐
91 man will resume submitting jobs when the number of idle procs
92 falls below 250.
93 -maxjobs NumberOfClusters
95 Sets the maximum number of clusters within the DAG that will
96 be submitted to HTCondor at one time. Note that for this ar‐
97 gument, each cluster counts as one job, no matter how many
98 individual procs are in the cluster. NumberOfClusters is a
99 non-negative integer. If this option is omitted, the number
100 of clusters is limited by the configuration variable DAG‐
101 MAN_MAX_JOBS_SUBMITTED
102 (see Configuration File Entries for DAGMan), which defaults
103 to 0 (unlimited).
104 -dagman DagmanExecutable
106 Allows the specification of an alternate condor_dagman exe‐
107 cutable to be used instead of the one found in the user's
108 path. This must be a fully qualified path.
109 -maxpre NumberOfPreScripts
111 Sets the maximum number of PRE scripts within the DAG that
112 may be running at one time. NumberOfPreScripts is a non-nega‐
113 tive integer. If this option is omitted, the number of PRE
114 scripts is limited by the configuration variable DAG‐
115 MAN_MAX_PRE_SCRIPTS (see Configuration File Entries for DAG‐
116 Man), which defaults to 20.
117 -maxpost NumberOfPostScripts
119 Sets the maximum number of POST scripts within the DAG that
120 may be running at one time. NumberOfPostScripts is a non-neg‐
121 ative integer. If this option is omitted, the number of POST
122 scripts is limited by the configuration variable DAG‐
123 MAN_MAX_POST_SCRIPTS
124 (see Configuration File Entries for DAGMan), which defaults
125 to 20.
126 -notification value
128 Sets the e-mail notification for DAGMan itself. This informa‐
129 tion will be used within the HTCondor submit description file
130 for DAGMan. This file is produced by condor_submit_dag. See
131 the description of notification within condor_submit manual
132 page for a specification of value.
133 -noeventchecks
135 This argument is no longer used; it is now ignored. Its func‐
136 tionality is now implemented by the DAGMAN_ALLOW_EVENTS con‐
137 figuration variable.
138 -allowlogerror
140 As of verson 8.5.5 this argument is no longer supported, and
141 setting it will generate a warning.
142 -r schedd_name
144 Submit condor_dagman to a remote machine, specifically the
145 condor_schedd daemon on that machine. The condor_dagman job
146 will not run on the local condor_schedd (the submit machine),
147 but on the specified one. This is implemented using the -re‐
148 mote option to condor_submit. Note that this option does not
149 currently specify input files for condor_dagman, nor the in‐
150 dividual nodes to be taken along! It is assumed that any nec‐
151 essary files will be present on the remote computer, possibly
152 via a shared file system between the local computer and the
153 remote computer. It is also necessary that the user has ap‐
154 propriate permissions to submit a job to the remote machine;
155 the permissions are the same as those required to use con‐
156 dor_submit 's -remote option. If other options are desired,
157 including transfer of other input files, consider using the
158 -no_submit option, modifying the resulting submit file for
159 specific needs, and then using condor_submit on that.
160 -debug level
162 Passes the the level of debugging output desired to con‐
163 dor_dagman. level is an integer, with values of 0-7 inclu‐
164 sive, where 7 is the most verbose output. See the condor_dag‐
165 man manual page for detailed descriptions of these values. If
166 not specified, no -debug v alue is passed to condor_dagman.
167 -usedagdir
169 This optional argument causes condor_dagman to run each spec‐
170 ified DAG as if condor_submit_dag had been run in the direc‐
171 tory containing that DAG file. This option is most useful
172 when running multiple DAGs in a single condor_dagman. Note
173 that the -usedagdir flag must not be used when running an
174 old-style Rescue DAG.
175 -outfile_dir directory
177 Specifies the directory in which the .dagman.out file will be
178 written. The directory may be specified relative to the cur‐
179 rent working directory as condor_submit_dag is executed, or
180 specified with an absolute path. Without this option, the
181 .dagman.out file is placed in the same directory as the first
182 DAG input file listed on the command line.
183 -config ConfigFileName
185 Specifies a configuration file to be used for this DAGMan
186 run. Note that the options specified in the configuration
187 file apply to all DAGs if multiple DAGs are specified. Fur‐
188 ther note that it is a fatal error if the configuration file
189 specified by this option conflicts with a configuration file
190 specified in any of the DAG files, if they specify one.
191 -insert_sub_file FileName
193 Specifies a file to insert into the .condor.sub file created
194 by condor_submit_dag. The specified file must contain only
195 legal submit file commands. Only one file can be inserted.
196 (If both the DAGMAN_INSERT_SUB_FILE configuration variable
197 and -insert_sub_file are specified, -insert_sub_file over‐
198 rides DAGMAN_INSERT_SUB_FILE.) The specified file is inserted
199 into the .condor.sub file before the Queue command and before
200 any commands specified with the -append option.
201 -append Command
203 Specifies a command to append to the .condor.sub file created
204 by condor_submit_dag. The specified command is appended to
205 the .condor.sub file immediately before the Queue command.
206 Multiple commands are specified by using the -append option
207 multiple times. Each new command is given in a separate -ap‐
208 pend option. Commands with spaces in them must be enclosed
209 in double quotes. Commands specified with the -append option
210 are appended to the .condor.sub file after commands inserted
211 from a file specified by the -insert_sub_file option or the
212 DAGMAN_INSERT_SUB_FILE configuration variable, so the -append
213 command(s) will override commands from the inserted file if
214 the commands conflict.
215 -batch-name batch_name
217 Set the batch name for this DAG/workflow. The batch name is
218 displayed by condor_q -batch. It is intended for use by users
219 to give meaningful names to their workflows and to influence
220 how condor_q groups jobs for display. As of version 8.5.5,
221 the batch name set with this argument is propagated to all
222 node jobs of the given DAG (including sub-DAGs), overriding
223 any batch names set in the individual submit files. Note: set
224 the batch name to ' ' (space) to avoid overriding batch names
225 specified in node job submit files. If no batch name is set,
226 the batch name defaults to DagFile +cluster (where DagFile is
227 the primary DAG file of the top-level DAGMan, and cluster is
228 the HTCondor cluster of the top-level DAGMan); the default
229 will override any lower-level batch names.
230 -autorescue 0|1
232 Whether to automatically run the newest rescue DAG for the
233 given DAG file, if one exists (0 = false, 1 = true).
234 -dorescuefrom number
236 Forces condor_dagman to run the specified rescue DAG number
237 for the given DAG. A value of 0 is the same as not specifying
238 this option. Specifying a non-existent rescue DAG is a fatal
239 error.
240 -allowversionmismatch
242 This optional argument causes condor_dagman to allow a ver‐
243 sion mismatch between condor_dagman itself and the .con‐
244 dor.sub file produced by condor_submit_dag (or, in other
245 words, between condor_submit_dag and condor_dagman). WARNING!
246 This option should be used only if absolutely necessary. Al‐
247 lowing version mismatches can cause subtle problems when run‐
248 ning DAGs. (Note that, starting with version 7.4.0, con‐
249 dor_dagman no longer requires an exact version match between
250 itself and the .condor.sub file. Instead, a "minimum compat‐
251 ible version" is defined, and any .condor.sub file of that
252 version or newer is accepted.)
253 -no_recurse
255 This optional argument causes condor_submit_dag to not run
256 itself recursively on nested DAGs (this is now the default;
257 this flag has been kept mainly for backwards compatibility).
258 -do_recurse
260 This optional argument causes condor_submit_dag to run itself
261 recursively on nested DAGs. The default is now that it does
262 not run itself recursively; instead the .condor.sub files for
263 nested DAGs are generated "lazily" by condor_dagman itself.
264 DAG nodes specified with the SUBDAG EXTERNAL keyword or with
265 submit file names ending in .condor.sub are considered nested
266 DAGs. The DAGMAN_GENERATE_SUBDAG_SUBMITS configuration vari‐
267 able may be relevant.
268 -update_submit
270 This optional argument causes an existing .condor.sub file to
271 not be treated as an error; rather, the .condor.sub file will
272 be overwritten, but the existing values of -maxjobs, -maxi‐
273 dle, -maxpre, and -maxpost will be preserved.
274 -import_env
276 This optional argument causes condor_submit_dag to import the
277 current environment into the environment command of the .con‐
278 dor.sub file it generates.
279 -DumpRescue
281 This optional argument tells condor_dagman to immediately
282 dump a rescue DAG and then exit, as opposed to actually run‐
283 ning the DAG. This feature is mainly intended for testing.
284 The Rescue DAG file is produced whether or not there are
285 parse errors reading the original DAG input file. The name of
286 the file differs if there was a parse error.
287 -valgrind
289 This optional argument causes the submit description file
290 generated for the submission of condor_dagman to be modified.
291 The executable becomes valgrind run on condor_dagman, with a
292 specific set of arguments intended for testing condor_dagman.
293 Note that this argument is intended for testing purposes
294 only. Using the -valgrind option without the necessary val‐
295 grind software installed will cause the DAG to fail. If the
296 DAG does run, it will run much more slowly than usual.
297 -DontAlwaysRunPost
299 This option causes the submit description file generated for
300 the submission of condor_dagman to be modified. It causes
301 condor_dagman to not run the POST script of a node if the PRE
302 script fails. (This was the default behavior prior to HTCon‐
303 dor version 7.7.2, and is again the default behavior from
304 version 8.5.4 onwards.)
305 -AlwaysRunPost
307 This option causes the submit description file generated for
308 the submission of condor_dagman to be modified. It causes
309 condor_dagman to always run the POST script of a node, even
310 if the PRE script fails. (This was the default behavior for
311 HTCondor version 7.7.2 through version 8.5.3.)
312 -priority number
314 Sets the minimum job priority of node jobs submitted and run‐
315 ning under the condor_dagman job submitted by this con‐
316 dor_submit_dag command.
317 -dont_use_default_node_log
319 This option is disabled as of HTCondor version 8.3.1. This
320 causes a compatibility error if the HTCondor version number
321 of the condor_schedd is 7.9.0 or older. Tells condor_dagman
322 to use the file specified by the job ClassAd attribute User‐
323 Log to monitor job status. If this command line argument is
324 used, then the job event log file cannot be defined with a
325 macro.
326 -schedd-daemon-ad-file FileName
328 Specifies a full path to a daemon ad file dropped by a con‐
329 dor_schedd. Therefore this allows submission to a specific
330 scheduler if several are available without repeatedly query‐
331 ing the condor_collector. The value for this argument de‐
332 faults to the configuration attribute SCHEDD_DAEMON_AD_FILE.
333 -schedd-address-file FileName
335 Specifies a full path to an address file dropped by a con‐
336 dor_schedd. Therefore this allows submission to a specific
337 scheduler if several are available without repeatedly query‐
338 ing the condor_collector. The value for this argument de‐
339 faults to the configuration attribute SCHEDD_ADDRESS_FILE.
340 -suppress_notification
342 Causes jobs submitted by condor_dagman to not send email no‐
343 tification for events. The same effect can be achieved by
344 setting configuration variable DAGMAN_SUPPRESS_NOTIFICATION
345 to True. This command line option is independent of the
346 -notification command line option, which controls notifica‐
347 tion for the condor_dagman job itself.
348 -dont_suppress_notification
350 Causes jobs submitted by condor_dagman to defer to content
351 within the submit description file when deciding to send
352 email notification for events. The same effect can be
353 achieved by setting configuration variable DAGMAN_SUP‐
354 PRESS_NOTIFICATION
355 to False. This command line flag is independent of the -no‐
356 tification command line option, which controls notification
357 for the condor_dagman job itself. If both -dont_suppress_no‐
358 tification and -suppress_notification are specified with the
359 same command line, the last argument is used.
360 -DoRecovery
362 Causes condor_dagman to start in recovery mode. (This means
363 that it reads the relevant job user log(s) and "catches up"
364 to the given DAG's previous state before submitting any new
365 jobs.)
366
368 condor_submit_dag will exit with a status value of 0 (zero) upon suc‐
369 cess, and it will exit with the value 1 (one) upon failure.
370
372 To run a single DAG:
373 % condor_submit_dag diamond.dag
375 To run a DAG when it has already been run and the output files exist:
377 % condor_submit_dag -force diamond.dag
379 To run a DAG, limiting the number of idle node jobs in the DAG to a
381 maximum of five:
382 % condor_submit_dag -maxidle 5 diamond.dag
384 To run a DAG, limiting the number of concurrent PRE scripts to 10 and
386 the number of concurrent POST scripts to five:
387 % condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
389 To run two DAGs, each of which is set up to run in its own directory:
391 % condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
393
395 HTCondor Team
396
398 1990-2022, Center for High Throughput Computing, Computer Sciences De‐
399 partment, University of Wisconsin-Madison, Madison, WI, US. Licensed
400 under the Apache License, Version 2.0.
4018.8 Jun 13, 2022 CONDOR_SUBMIT_DAG(1)