1condor_submit_dag(1) General Commands Manual condor_submit_dag(1)
2
6 condor_submit_dagManage and queue jobs within a specified DAG for exe‐
7 cution on remote machines
8
10 condor_submit_dag[-help | -version]
11 condor_submit_dag[-no_submit] [-verbose] [-force] [-maxidle NumberOf‐
13 Procs] [-maxjobs NumberOfClusters] [-dagman DagmanExecutable] [-maxpre
14 NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notification
15 value] [-noeventchecks] [-allowlogerror] [-r schedd_name] [-debug
16 level] [-usedagdir] [-outfile_dir directory] [-config ConfigFileName]
17 [-insert_sub_file FileName] [-append Command] [-batch-name batch_name]
18 [-autorescue 0|1] [-dorescuefrom number] [-allowversionmismatch]
19 [-no_recurse] [-do_recurse] [-update_submit] [-import_env] [-DumpRes‐
20 cue] [-valgrind] [-DontAlwaysRunPost] [-AlwaysRunPost] [-priority num‐
21 ber] [-dont_use_default_node_log] [-schedd-daemon-ad-file FileName]
22 [-schedd-address-file FileName] [-suppress_notification] [-dont_sup‐
23 press_notification] [-DoRecovery] DAGInputFile1[DAGInputFile2 ...DAGIn‐
24 putFileN ]
25
27 condor_submit_dagis the program for submitting a DAG (directed acyclic
28 graph) of jobs for execution under HTCondor. The program enforces the
29 job dependencies defined in one or more DAGInputFiles. Each DAGInput‐
30 Filecontains commands to direct the submission of jobs implied by the
31 nodes of a DAG to HTCondor. Extensive documentation is in the HTCondor
32 User Manual section on DAGMan.
33 Some options may be specified on the command line or in the configura‐
35 tion or in a node job's submit description file. Precedence is given to
36 command line options or configuration over settings from a submit
37 description file. An example is e-mail notifications. When configura‐
38 tion variable DAGMAN_SUPPRESS_NOTIFICATIONis its default value of True,
39 and a node job's submit description file contains
40 notification = Complete
42 e-mail will notbe sent upon completion, as the value of DAGMAN_SUP‐
44 PRESS_NOTIFICATIONis enforced.
45
47 -help
48 Display usage information and exit.
50 -version
54 Display version information and exit.
56 -no_submit
60 Produce the HTCondor submit description file for DAGMan, but do not
62 submit DAGMan as an HTCondor job.
63 -verbose
67 Cause condor_submit_dagto give verbose error messages.
69 -force
73 Require condor_submit_dagto overwrite the files that it produces, if
75 the files already exist. Note that dagman.outwill be appended to,
76 not overwritten. If new-style rescue DAG mode is in effect, and any
77 new-style rescue DAGs exist, the -forceflag will cause them to be
78 renamed, and the original DAG will be run. If old-style rescue DAG
79 mode is in effect, any existing old-style rescue DAGs will be
80 deleted, and the original DAG will be run.
81 -maxidle NumberOfProcs
85 Sets the maximum number of idle procs allowed before condor_dag‐
87 manstops submitting more node jobs. Note that for this argument,
88 each individual proc within a cluster counts as a towards the limit,
89 which is inconsistent with -maxjobs .Once idle procs start to run,
90 condor_dagmanwill resume submitting jobs once the number of idle
91 procs falls below the specified limit. NumberOfProcsis a non-nega‐
92 tive integer. If this option is omitted, the number of idle procs is
93 limited by the configuration variable DAGMAN_MAX_JOBS_IDLE(see ),
94 which defaults to 1000. To disable this limit, set NumberOfProcsto
95 0. Note that submit description files that queue multiple procs can
96 cause the NumberOfProcslimit to be exceeded. Setting queue 5000in
97 the submit description file, where -maxidleis set to 250 will result
98 in a cluster of 5000 new procs being submitted to the condor_schedd,
99 not 250. In this case, condor_dagmanwill resume submitting jobs when
100 the number of idle procs falls below 250.
101 -maxjobs NumberOfClusters
105 Sets the maximum number of clusters within the DAG that will be sub‐
107 mitted to HTCondor at one time. Note that for this argument, each
108 cluster counts as one job, no matter how many individual procs are
109 in the cluster. NumberOfClustersis a non-negative integer. If this
110 option is omitted, the number of clusters is limited by the configu‐
111 ration variable DAGMAN_MAX_JOBS_SUBMITTED(see ), which defaults to 0
112 (unlimited).
113 -dagman DagmanExecutable
117 Allows the specification of an alternate condor_dagmanexecutable to
119 be used instead of the one found in the user's path. This must be a
120 fully qualified path.
121 -maxpre NumberOfPreScripts
125 Sets the maximum number of PRE scripts within the DAG that may be
127 running at one time. NumberOfPreScriptsis a non-negative integer. If
128 this option is omitted, the number of PRE scripts is limited by the
129 configuration variable DAGMAN_MAX_PRE_SCRIPTS(see ), which defaults
130 to 20.
131 -maxpost NumberOfPostScripts
135 Sets the maximum number of POST scripts within the DAG that may be
137 running at one time. NumberOfPostScriptsis a non-negative integer.
138 If this option is omitted, the number of POST scripts is limited by
139 the configuration variable DAGMAN_MAX_POST_SCRIPTS(see ), which
140 defaults to 20.
141 -notification value
145 Sets the e-mail notification for DAGMan itself. This information
147 will be used within the HTCondor submit description file for DAGMan.
148 This file is produced by condor_submit_dag. See the description of
149 notificationwithin condor_submitmanual page for a specification of
150 value.
151 -noeventchecks
155 This argument is no longer used; it is now ignored. Its functional‐
157 ity is now implemented by the DAGMAN_ALLOW_EVENTSconfiguration vari‐
158 able.
159 -allowlogerror
163 As of verson 8.5.5 this argument is no longer supported, and setting
165 it will generate a warning.
166 -r schedd_name
170 Submit condor_dagmanto a remote machine, specifically the con‐
172 dor_schedddaemon on that machine. The condor_dagmanjob will not run
173 on the local condor_schedd(the submit machine), but on the specified
174 one. This is implemented using the -remoteoption to condor_submit.
175 Note that this option does not currently specify input files for
176 condor_dagman, nor the individual nodes to be taken along! It is
177 assumed that any necessary files will be present on the remote com‐
178 puter, possibly via a shared file system between the local computer
179 and the remote computer. It is also necessary that the user has
180 appropriate permissions to submit a job to the remote machine; the
181 permissions are the same as those required to use condor_submit's
182 -remoteoption. If other options are desired, including transfer of
183 other input files, consider using the -no_submitoption, modifying
184 the resulting submit file for specific needs, and then using con‐
185 dor_submiton that.
186 -debug level
190 Passes the the levelof debugging output desired to condor_dagman.
192 levelis an integer, with values of 0-7 inclusive, where 7 is the
193 most verbose output. See the condor_dagmanmanual page for detailed
194 descriptions of these values. If not specified, no -debug value is
195 passed to condor_dagman.
196 -usedagdir
200 This optional argument causes condor_dagmanto run each specified DAG
202 as if condor_submit_daghad been run in the directory containing that
203 DAG file. This option is most useful when running multiple DAGs in a
204 single condor_dagman. Note that the -usedagdirflag must not be used
205 when running an old-style Rescue DAG.
206 -outfile_dir directory
210 Specifies the directory in which the .dagman.outfile will be writ‐
212 ten. The directorymay be specified relative to the current working
213 directory as condor_submit_dagis executed, or specified with an
214 absolute path. Without this option, the .dagman.outfile is placed in
215 the same directory as the first DAG input file listed on the command
216 line.
217 -config ConfigFileName
221 Specifies a configuration file to be used for this DAGMan run. Note
223 that the options specified in the configuration file apply to all
224 DAGs if multiple DAGs are specified. Further note that it is a fatal
225 error if the configuration file specified by this option conflicts
226 with a configuration file specified in any of the DAG files, if they
227 specify one.
228 -insert_sub_file FileName
232 Specifies a file to insert into the .condor.subfile created by con‐
234 dor_submit_dag. The specified file must contain only legal submit
235 file commands. Only one file can be inserted. (If both the DAG‐
236 MAN_INSERT_SUB_FILE configuration variable and -insert_sub_fileare
237 specified, -insert_sub_fileoverrides DAGMAN_INSERT_SUB_FILE.) The
238 specified file is inserted into the .condor.subfile before the Queue
239 command and before any commands specified with the -appendoption.
240 -append Command
244 Specifies a command to append to the .condor.subfile created by con‐
246 dor_submit_dag. The specified command is appended to the .con‐
247 dor.subfile immediately before the Queue command. Multiple commands
248 are specified by using the -appendoption multiple times. Each new
249 command is given in a separate -appendoption. Commands with spaces
250 in them must be enclosed in double quotes. Commands specified with
251 the -appendoption are appended to the .condor.subfile aftercommands
252 inserted from a file specified by the -insert_sub_fileoption or the
253 DAGMAN_INSERT_SUB_FILE configuration variable, so the -appendcom‐
254 mand(s) will override commands from the inserted file if the com‐
255 mands conflict.
256 -batch-name batch_name
260 Set the batch name for this DAG/workflow. The batch name is dis‐
262 played by condor_q-batch. It is intended for use by users to give
263 meaningful names to their workflows and to influence how con‐
264 dor_qgroups jobs for display. As of version 8.5.5, the batch name
265 set with this argument is propagated to all node jobs of the given
266 DAG (including sub-DAGs), overriding any batch names set in the
267 individual submit files. Note: set the batch name to ' ' (space) to
268 avoid overriding batch names specified in node job submit files. If
269 no batch name is set, the batch name defaults to DagFile+clus‐
270 ter(where DagFileis the primary DAG file of the top-level DAGMan,
271 and clusteris the HTCondor cluster of the top-level DAGMan); the
272 default willoverride any lower-level batch names.
273 -autorescue 0|1
277 Whether to automatically run the newest rescue DAG for the given DAG
279 file, if one exists (0 = false, 1 = true).
280 -dorescuefrom number
284 Forces condor_dagmanto run the specified rescue DAG number for the
286 given DAG. A value of 0 is the same as not specifying this option.
287 Specifying a non-existent rescue DAG is a fatal error.
288 -allowversionmismatch
292 This optional argument causes condor_dagmanto allow a version mis‐
294 match between condor_dagmanitself and the .condor.subfile produced
295 by condor_submit_dag(or, in other words, between condor_sub‐
296 mit_dagand condor_dagman). WARNING! This option should be used only
297 if absolutely necessary. Allowing version mismatches can cause sub‐
298 tle problems when running DAGs. (Note that, starting with version
299 7.4.0, condor_dagmanno longer requires an exact version match
300 between itself and the .condor.subfile. Instead, a "minimum compati‐
301 ble version" is defined, and any .condor.subfile of that version or
302 newer is accepted.)
303 -no_recurse
307 This optional argument causes condor_submit_dagto notrun itself
309 recursively on nested DAGs (this is now the default; this flag has
310 been kept mainly for backwards compatibility).
311 -do_recurse
315 This optional argument causes condor_submit_dagto run itself recur‐
317 sively on nested DAGs. The default is now that it does notrun itself
318 recursively; instead the .condor.subfiles for nested DAGs are gener‐
319 ated "lazily" by condor_dagmanitself. DAG nodes specified with the
320 SUBDAG EXTERNALkeyword or with submit file names ending in .con‐
321 dor.subare considered nested DAGs. The DAGMAN_GENERATE_SUBDAG_SUB‐
322 MITSconfiguration variable may be relevant.
323 -update_submit
327 This optional argument causes an existing .condor.subfile to not be
329 treated as an error; rather, the .condor.subfile will be overwrit‐
330 ten, but the existing values of -maxjobs, -maxidle, -maxpre, and
331 -maxpostwill be preserved.
332 -import_env
336 This optional argument causes condor_submit_dagto import the current
338 environment into the environmentcommand of the .condor.subfile it
339 generates.
340 -DumpRescue
344 This optional argument tells condor_dagmanto immediately dump a res‐
346 cue DAG and then exit, as opposed to actually running the DAG. This
347 feature is mainly intended for testing. The Rescue DAG file is pro‐
348 duced whether or not there are parse errors reading the original DAG
349 input file. The name of the file differs if there was a parse error.
350 -valgrind
354 This optional argument causes the submit description file generated
356 for the submission of condor_dagmanto be modified. The executable
357 becomes valgrindrun on condor_dagman, with a specific set of argu‐
358 ments intended for testing condor_dagman. Note that this argument is
359 intended for testing purposes only. Using the -valgrindoption with‐
360 out the necessary valgrindsoftware installed will cause the DAG to
361 fail. If the DAG does run, it will run much more slowly than usual.
362 -DontAlwaysRunPost
366 This option causes the submit description file generated for the
368 submission of condor_dagmanto be modified. It causes condor_dagmanto
369 not run the POST script of a node if the PRE script fails. (This was
370 the default behavior prior to HTCondor version 7.7.2, and is again
371 the default behavior from version 8.5.4 onwards.)
372 -AlwaysRunPost
376 This option causes the submit description file generated for the
378 submission of condor_dagmanto be modified. It causes condor_dagmanto
379 always run the POST script of a node, even if the PRE script fails.
380 (This was the default behavior for HTCondor version 7.7.2 through
381 version 8.5.3.)
382 -priority number
386 Sets the minimum job priority of node jobs submitted and running
388 under the condor_dagmanjob submitted by this condor_submit_dagcom‐
389 mand.
390 -dont_use_default_node_log
394 This option is disabled as of HTCondor version 8.3.1. This causes a
396 compatibility error if the HTCondor version number of the con‐
397 dor_scheddis 7.9.0 or older.Tells condor_dagmanto use the file spec‐
398 ified by the job ClassAd attribute UserLogto monitor job status. If
399 this command line argument is used, then the job event log file can‐
400 not be defined with a macro.
401 -schedd-daemon-ad-file FileName
405 Specifies a full path to a daemon ad file dropped by a con‐
407 dor_schedd. Therefore this allows submission to a specific scheduler
408 if several are available without repeatedly querying the condor_col‐
409 lector. The value for this argument defaults to the configuration
410 attribute SCHEDD_DAEMON_AD_FILE.
411 -schedd-address-file FileName
415 Specifies a full path to an address file dropped by a condor_schedd.
417 Therefore this allows submission to a specific scheduler if several
418 are available without repeatedly querying the condor_collector. The
419 value for this argument defaults to the configuration attribute
420 SCHEDD_ADDRESS_FILE.
421 -suppress_notification
425 Causes jobs submitted by condor_dagmanto not send email notification
427 for events. The same effect can be achieved by setting configuration
428 variable DAGMAN_SUPPRESS_NOTIFICATIONto True. This command line
429 option is independent of the -notificationcommand line option, which
430 controls notification for the condor_dagmanjob itself.
431 -dont_suppress_notification
435 Causes jobs submitted by condor_dagmanto defer to content within the
437 submit description file when deciding to send email notification for
438 events. The same effect can be achieved by setting configuration
439 variable DAGMAN_SUPPRESS_NOTIFICATIONto False. This command line
440 flag is independent of the -notificationcommand line option, which
441 controls notification for the condor_dagmanjob itself. If both
442 -dont_suppress_notificationand -suppress_notificationare specified
443 with the same command line, the last argument is used.
444 -DoRecovery
448 Causes condor_dagmanto start in recovery mode. (This means that it
450 reads the relevant job user log(s) and "catches up" to the given
451 DAG's previous state before submitting any new jobs.)
452
456 condor_submit_dagwill exit with a status value of 0 (zero) upon suc‐
457 cess, and it will exit with the value 1 (one) upon failure.
458
460 To run a single DAG:
461 % condor_submit_dag diamond.dag
463 To run a DAG when it has already been run and the output files exist:
465 % condor_submit_dag -force diamond.dag
467 To run a DAG, limiting the number of idle node jobs in the DAG to a
469 maximum of five:
470 % condor_submit_dag -maxidle 5 diamond.dag
472 To run a DAG, limiting the number of concurrent PRE scripts to 10 and
474 the number of concurrent POST scripts to five:
475 % condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
477 To run two DAGs, each of which is set up to run in its own directory:
479 % condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
481
483 Center for High Throughput Computing, University of Wisconsin-Madison
484
486 Copyright (C) 1990-2019 Center for High Throughput Computing, Computer
487 Sciences Department, University of Wisconsin-Madison, Madison, WI. All
488 Rights Reserved. Licensed under the Apache License, Version 2.0.
489 date condor_submit_dag(1)