1condor_submit_dag(1) General Commands Manual condor_submit_dag(1)
2
3
4
6 condor_submit_dagManage and queue jobs within a specified DAG for exe‐
7 cution on remote machines
8
10 condor_submit_dag[-help | -version]
11
12 condor_submit_dag[-no_submit] [-verbose] [-force] [-maxidle NumberOf‐
13 Procs] [-maxjobs NumberOfClusters] [-dagman DagmanExecutable] [-maxpre
14 NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notification
15 value] [-noeventchecks] [-allowlogerror] [-r schedd_name] [-debug
16 level] [-usedagdir] [-outfile_dir directory] [-config ConfigFileName]
17 [-insert_sub_file FileName] [-append Command] [-batch-name batch_name]
18 [-autorescue 0|1] [-dorescuefrom number] [-allowversionmismatch]
19 [-no_recurse] [-do_recurse] [-update_submit] [-import_env] [-DumpRes‐
20 cue] [-valgrind] [-DontAlwaysRunPost] [-AlwaysRunPost] [-priority num‐
21 ber] [-dont_use_default_node_log] [-schedd-daemon-ad-file FileName]
22 [-schedd-address-file FileName] [-suppress_notification] [-dont_sup‐
23 press_notification] [-DoRecovery] DAGInputFile1[DAGInputFile2 ...DAGIn‐
24 putFileN ]
25
27 condor_submit_dagis the program for submitting a DAG (directed acyclic
28 graph) of jobs for execution under HTCondor. The program enforces the
29 job dependencies defined in one or more DAGInputFiles. Each DAGInput‐
30 Filecontains commands to direct the submission of jobs implied by the
31 nodes of a DAG to HTCondor. Extensive documentation is in the HTCondor
32 User Manual section on DAGMan.
33
34 Some options may be specified on the command line or in the configura‐
35 tion or in a node job's submit description file. Precedence is given to
36 command line options or configuration over settings from a submit
37 description file. An example is e-mail notifications. When configura‐
38 tion variable DAGMAN_SUPPRESS_NOTIFICATIONis its default value of True,
39 and a node job's submit description file contains
40
41 notification = Complete
42
43 e-mail will notbe sent upon completion, as the value of DAGMAN_SUP‐
44 PRESS_NOTIFICATIONis enforced.
45
47 -help
48
49 Display usage information and exit.
50
51
52
53 -version
54
55 Display version information and exit.
56
57
58
59 -no_submit
60
61 Produce the HTCondor submit description file for DAGMan, but do not
62 submit DAGMan as an HTCondor job.
63
64
65
66 -verbose
67
68 Cause condor_submit_dagto give verbose error messages.
69
70
71
72 -force
73
74 Require condor_submit_dagto overwrite the files that it produces, if
75 the files already exist. Note that dagman.outwill be appended to,
76 not overwritten. If new-style rescue DAG mode is in effect, and any
77 new-style rescue DAGs exist, the -forceflag will cause them to be
78 renamed, and the original DAG will be run. If old-style rescue DAG
79 mode is in effect, any existing old-style rescue DAGs will be
80 deleted, and the original DAG will be run.
81
82
83
84 -maxidle NumberOfProcs
85
86 Sets the maximum number of idle procs allowed before condor_dag‐
87 manstops submitting more node jobs. Note that for this argument,
88 each individual proc within a cluster counts as a towards the limit,
89 which is inconsistent with -maxjobs .Once idle procs start to run,
90 condor_dagmanwill resume submitting jobs once the number of idle
91 procs falls below the specified limit. NumberOfProcsis a non-nega‐
92 tive integer. If this option is omitted, the number of idle procs is
93 limited by the configuration variable DAGMAN_MAX_JOBS_IDLE(see ),
94 which defaults to 1000. To disable this limit, set NumberOfProcsto
95 0. Note that submit description files that queue multiple procs can
96 cause the NumberOfProcslimit to be exceeded. Setting queue 5000in
97 the submit description file, where -maxidleis set to 250 will result
98 in a cluster of 5000 new procs being submitted to the condor_schedd,
99 not 250. In this case, condor_dagmanwill resume submitting jobs when
100 the number of idle procs falls below 250.
101
102
103
104 -maxjobs NumberOfClusters
105
106 Sets the maximum number of clusters within the DAG that will be sub‐
107 mitted to HTCondor at one time. Note that for this argument, each
108 cluster counts as one job, no matter how many individual procs are
109 in the cluster. NumberOfClustersis a non-negative integer. If this
110 option is omitted, the number of clusters is limited by the configu‐
111 ration variable DAGMAN_MAX_JOBS_SUBMITTED(see ), which defaults to 0
112 (unlimited).
113
114
115
116 -dagman DagmanExecutable
117
118 Allows the specification of an alternate condor_dagmanexecutable to
119 be used instead of the one found in the user's path. This must be a
120 fully qualified path.
121
122
123
124 -maxpre NumberOfPreScripts
125
126 Sets the maximum number of PRE scripts within the DAG that may be
127 running at one time. NumberOfPreScriptsis a non-negative integer. If
128 this option is omitted, the number of PRE scripts is limited by the
129 configuration variable DAGMAN_MAX_PRE_SCRIPTS(see ), which defaults
130 to 20.
131
132
133
134 -maxpost NumberOfPostScripts
135
136 Sets the maximum number of POST scripts within the DAG that may be
137 running at one time. NumberOfPostScriptsis a non-negative integer.
138 If this option is omitted, the number of POST scripts is limited by
139 the configuration variable DAGMAN_MAX_POST_SCRIPTS(see ), which
140 defaults to 20.
141
142
143
144 -notification value
145
146 Sets the e-mail notification for DAGMan itself. This information
147 will be used within the HTCondor submit description file for DAGMan.
148 This file is produced by condor_submit_dag. See the description of
149 notificationwithin condor_submitmanual page for a specification of
150 value.
151
152
153
154 -noeventchecks
155
156 This argument is no longer used; it is now ignored. Its functional‐
157 ity is now implemented by the DAGMAN_ALLOW_EVENTSconfiguration vari‐
158 able.
159
160
161
162 -allowlogerror
163
164 As of verson 8.5.5 this argument is no longer supported, and setting
165 it will generate a warning.
166
167
168
169 -r schedd_name
170
171 Submit condor_dagmanto a remote machine, specifically the con‐
172 dor_schedddaemon on that machine. The condor_dagmanjob will not run
173 on the local condor_schedd(the submit machine), but on the specified
174 one. This is implemented using the -remoteoption to condor_submit.
175 Note that this option does not currently specify input files for
176 condor_dagman, nor the individual nodes to be taken along! It is
177 assumed that any necessary files will be present on the remote com‐
178 puter, possibly via a shared file system between the local computer
179 and the remote computer. It is also necessary that the user has
180 appropriate permissions to submit a job to the remote machine; the
181 permissions are the same as those required to use condor_submit's
182 -remoteoption. If other options are desired, including transfer of
183 other input files, consider using the -no_submitoption, modifying
184 the resulting submit file for specific needs, and then using con‐
185 dor_submiton that.
186
187
188
189 -debug level
190
191 Passes the the levelof debugging output desired to condor_dagman.
192 levelis an integer, with values of 0-7 inclusive, where 7 is the
193 most verbose output. See the condor_dagmanmanual page for detailed
194 descriptions of these values. If not specified, no -debug value is
195 passed to condor_dagman.
196
197
198
199 -usedagdir
200
201 This optional argument causes condor_dagmanto run each specified DAG
202 as if condor_submit_daghad been run in the directory containing that
203 DAG file. This option is most useful when running multiple DAGs in a
204 single condor_dagman. Note that the -usedagdirflag must not be used
205 when running an old-style Rescue DAG.
206
207
208
209 -outfile_dir directory
210
211 Specifies the directory in which the .dagman.outfile will be writ‐
212 ten. The directorymay be specified relative to the current working
213 directory as condor_submit_dagis executed, or specified with an
214 absolute path. Without this option, the .dagman.outfile is placed in
215 the same directory as the first DAG input file listed on the command
216 line.
217
218
219
220 -config ConfigFileName
221
222 Specifies a configuration file to be used for this DAGMan run. Note
223 that the options specified in the configuration file apply to all
224 DAGs if multiple DAGs are specified. Further note that it is a fatal
225 error if the configuration file specified by this option conflicts
226 with a configuration file specified in any of the DAG files, if they
227 specify one.
228
229
230
231 -insert_sub_file FileName
232
233 Specifies a file to insert into the .condor.subfile created by con‐
234 dor_submit_dag. The specified file must contain only legal submit
235 file commands. Only one file can be inserted. (If both the DAG‐
236 MAN_INSERT_SUB_FILE configuration variable and -insert_sub_fileare
237 specified, -insert_sub_fileoverrides DAGMAN_INSERT_SUB_FILE.) The
238 specified file is inserted into the .condor.subfile before the Queue
239 command and before any commands specified with the -appendoption.
240
241
242
243 -append Command
244
245 Specifies a command to append to the .condor.subfile created by con‐
246 dor_submit_dag. The specified command is appended to the .con‐
247 dor.subfile immediately before the Queue command. Multiple commands
248 are specified by using the -appendoption multiple times. Each new
249 command is given in a separate -appendoption. Commands with spaces
250 in them must be enclosed in double quotes. Commands specified with
251 the -appendoption are appended to the .condor.subfile aftercommands
252 inserted from a file specified by the -insert_sub_fileoption or the
253 DAGMAN_INSERT_SUB_FILE configuration variable, so the -appendcom‐
254 mand(s) will override commands from the inserted file if the com‐
255 mands conflict.
256
257
258
259 -batch-name batch_name
260
261 Set the batch name for this DAG/workflow. The batch name is dis‐
262 played by condor_q-batch. It is intended for use by users to give
263 meaningful names to their workflows and to influence how con‐
264 dor_qgroups jobs for display. As of version 8.5.5, the batch name
265 set with this argument is propagated to all node jobs of the given
266 DAG (including sub-DAGs), overriding any batch names set in the
267 individual submit files. Note: set the batch name to ' ' (space) to
268 avoid overriding batch names specified in node job submit files. If
269 no batch name is set, the batch name defaults to DagFile+clus‐
270 ter(where DagFileis the primary DAG file of the top-level DAGMan,
271 and clusteris the HTCondor cluster of the top-level DAGMan); the
272 default willoverride any lower-level batch names.
273
274
275
276 -autorescue 0|1
277
278 Whether to automatically run the newest rescue DAG for the given DAG
279 file, if one exists (0 = false, 1 = true).
280
281
282
283 -dorescuefrom number
284
285 Forces condor_dagmanto run the specified rescue DAG number for the
286 given DAG. A value of 0 is the same as not specifying this option.
287 Specifying a non-existent rescue DAG is a fatal error.
288
289
290
291 -allowversionmismatch
292
293 This optional argument causes condor_dagmanto allow a version mis‐
294 match between condor_dagmanitself and the .condor.subfile produced
295 by condor_submit_dag(or, in other words, between condor_sub‐
296 mit_dagand condor_dagman). WARNING! This option should be used only
297 if absolutely necessary. Allowing version mismatches can cause sub‐
298 tle problems when running DAGs. (Note that, starting with version
299 7.4.0, condor_dagmanno longer requires an exact version match
300 between itself and the .condor.subfile. Instead, a "minimum compati‐
301 ble version" is defined, and any .condor.subfile of that version or
302 newer is accepted.)
303
304
305
306 -no_recurse
307
308 This optional argument causes condor_submit_dagto notrun itself
309 recursively on nested DAGs (this is now the default; this flag has
310 been kept mainly for backwards compatibility).
311
312
313
314 -do_recurse
315
316 This optional argument causes condor_submit_dagto run itself recur‐
317 sively on nested DAGs. The default is now that it does notrun itself
318 recursively; instead the .condor.subfiles for nested DAGs are gener‐
319 ated "lazily" by condor_dagmanitself. DAG nodes specified with the
320 SUBDAG EXTERNALkeyword or with submit file names ending in .con‐
321 dor.subare considered nested DAGs. The DAGMAN_GENERATE_SUBDAG_SUB‐
322 MITSconfiguration variable may be relevant.
323
324
325
326 -update_submit
327
328 This optional argument causes an existing .condor.subfile to not be
329 treated as an error; rather, the .condor.subfile will be overwrit‐
330 ten, but the existing values of -maxjobs, -maxidle, -maxpre, and
331 -maxpostwill be preserved.
332
333
334
335 -import_env
336
337 This optional argument causes condor_submit_dagto import the current
338 environment into the environmentcommand of the .condor.subfile it
339 generates.
340
341
342
343 -DumpRescue
344
345 This optional argument tells condor_dagmanto immediately dump a res‐
346 cue DAG and then exit, as opposed to actually running the DAG. This
347 feature is mainly intended for testing. The Rescue DAG file is pro‐
348 duced whether or not there are parse errors reading the original DAG
349 input file. The name of the file differs if there was a parse error.
350
351
352
353 -valgrind
354
355 This optional argument causes the submit description file generated
356 for the submission of condor_dagmanto be modified. The executable
357 becomes valgrindrun on condor_dagman, with a specific set of argu‐
358 ments intended for testing condor_dagman. Note that this argument is
359 intended for testing purposes only. Using the -valgrindoption with‐
360 out the necessary valgrindsoftware installed will cause the DAG to
361 fail. If the DAG does run, it will run much more slowly than usual.
362
363
364
365 -DontAlwaysRunPost
366
367 This option causes the submit description file generated for the
368 submission of condor_dagmanto be modified. It causes condor_dagmanto
369 not run the POST script of a node if the PRE script fails. (This was
370 the default behavior prior to HTCondor version 7.7.2, and is again
371 the default behavior from version 8.5.4 onwards.)
372
373
374
375 -AlwaysRunPost
376
377 This option causes the submit description file generated for the
378 submission of condor_dagmanto be modified. It causes condor_dagmanto
379 always run the POST script of a node, even if the PRE script fails.
380 (This was the default behavior for HTCondor version 7.7.2 through
381 version 8.5.3.)
382
383
384
385 -priority number
386
387 Sets the minimum job priority of node jobs submitted and running
388 under the condor_dagmanjob submitted by this condor_submit_dagcom‐
389 mand.
390
391
392
393 -dont_use_default_node_log
394
395 This option is disabled as of HTCondor version 8.3.1. This causes a
396 compatibility error if the HTCondor version number of the con‐
397 dor_scheddis 7.9.0 or older.Tells condor_dagmanto use the file spec‐
398 ified by the job ClassAd attribute UserLogto monitor job status. If
399 this command line argument is used, then the job event log file can‐
400 not be defined with a macro.
401
402
403
404 -schedd-daemon-ad-file FileName
405
406 Specifies a full path to a daemon ad file dropped by a con‐
407 dor_schedd. Therefore this allows submission to a specific scheduler
408 if several are available without repeatedly querying the condor_col‐
409 lector. The value for this argument defaults to the configuration
410 attribute SCHEDD_DAEMON_AD_FILE.
411
412
413
414 -schedd-address-file FileName
415
416 Specifies a full path to an address file dropped by a condor_schedd.
417 Therefore this allows submission to a specific scheduler if several
418 are available without repeatedly querying the condor_collector. The
419 value for this argument defaults to the configuration attribute
420 SCHEDD_ADDRESS_FILE.
421
422
423
424 -suppress_notification
425
426 Causes jobs submitted by condor_dagmanto not send email notification
427 for events. The same effect can be achieved by setting configuration
428 variable DAGMAN_SUPPRESS_NOTIFICATIONto True. This command line
429 option is independent of the -notificationcommand line option, which
430 controls notification for the condor_dagmanjob itself.
431
432
433
434 -dont_suppress_notification
435
436 Causes jobs submitted by condor_dagmanto defer to content within the
437 submit description file when deciding to send email notification for
438 events. The same effect can be achieved by setting configuration
439 variable DAGMAN_SUPPRESS_NOTIFICATIONto False. This command line
440 flag is independent of the -notificationcommand line option, which
441 controls notification for the condor_dagmanjob itself. If both
442 -dont_suppress_notificationand -suppress_notificationare specified
443 with the same command line, the last argument is used.
444
445
446
447 -DoRecovery
448
449 Causes condor_dagmanto start in recovery mode. (This means that it
450 reads the relevant job user log(s) and "catches up" to the given
451 DAG's previous state before submitting any new jobs.)
452
453
454
456 condor_submit_dagwill exit with a status value of 0 (zero) upon suc‐
457 cess, and it will exit with the value 1 (one) upon failure.
458
460 To run a single DAG:
461
462 % condor_submit_dag diamond.dag
463
464 To run a DAG when it has already been run and the output files exist:
465
466 % condor_submit_dag -force diamond.dag
467
468 To run a DAG, limiting the number of idle node jobs in the DAG to a
469 maximum of five:
470
471 % condor_submit_dag -maxidle 5 diamond.dag
472
473 To run a DAG, limiting the number of concurrent PRE scripts to 10 and
474 the number of concurrent POST scripts to five:
475
476 % condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
477
478 To run two DAGs, each of which is set up to run in its own directory:
479
480 % condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
481
483 Center for High Throughput Computing, University of Wiscon‐
484 sin–Madison
485
487 Copyright © 1990-2019 Center for High Throughput Computing, Computer
488 Sciences Department, University of Wisconsin-Madison, Madison, WI. All
489 Rights Reserved. Licensed under the Apache License, Version 2.0.
490
491
492
493 date condor_submit_dag(1)