1CONDOR_SUBMIT_DAG(1) HTCondor Manual CONDOR_SUBMIT_DAG(1)
2
6 condor_submit_dag - HTCondor Manual
7 Manage and queue jobs within a specified DAG for execution on remote
9 machines
10
13 condor_submit_dag [-help | -version ]
14 condor_submit_dag [-no_submit ] [-verbose ] [-force ] [-dagman Dag‐
16 manExecutable] [-maxidle NumberOfProcs] [-maxjobs NumberOfClusters]
17 [-maxpre NumberOfPreScripts] [-maxpost NumberOfPostScripts] [-notifica‐
18 tion value] [-r schedd_name] [-debug level] [-usedagdir ] [-outfile_dir
19 directory] [-config ConfigFileName] [-insert_sub_file FileName] [-ap‐
20 pend Command] [-batch-name batch_name] [-autorescue 0|1] [-dorescuefrom
21 number] [-load_save filename] [-allowversionmismatch ] [-no_recurse ]
22 [-do_recurse ] [-update_submit ] [-import_env ] [-include_env Vari‐
23 ables] [-insert_env Key=Value] [-DumpRescue ] [-valgrind ] [-DontAl‐
24 waysRunPost ] [-AlwaysRunPost ] [-priority number] [-schedd-dae‐
25 mon-ad-file FileName] [-schedd-address-file FileName] [-suppress_noti‐
26 fication ] [-dont_suppress_notification ] [-DoRecovery ] DAGInputFile1
27 [DAGInputFile2 ... DAGInputFileN ]
28
30 condor_submit_dag is the program for submitting a DAG (directed acyclic
31 graph) of jobs for execution under HTCondor. The program enforces the
32 job dependencies defined in one or more DAGInputFiles. Each DAGInput‐
33 File contains commands to direct the submission of jobs implied by the
34 nodes of a DAG to HTCondor. Extensive documentation is in the HTCondor
35 User Manual section on DAGMan.
36 Some options may be specified on the command line or in the configura‐
40 tion or in a node job's submit description file. Precedence is given to
41 command line options or configuration over settings from a submit de‐
42 scription file. An example is e-mail notifications. When configuration
43 variable DAGMAN_SUPPRESS_NOTIFICATION is its default value of True,
44 and a node job's submit description file contains
45 notification = Complete
47 e-mail will not be sent upon completion, as the value of DAGMAN_SUP‐
49 PRESS_NOTIFICATION is enforced.
50
52 -help Display usage information and exit.
53 -version
55 Display version information and exit.
56 -no_submit
58 Produce the HTCondor submit description file for DAGMan, but
59 do not submit DAGMan as an HTCondor job.
60 -verbose
62 Cause condor_submit_dag to give verbose error messages.
63 -force Require condor_submit_dag to overwrite the files that it pro‐
65 duces, if the files already exist. Note that dagman.out will
66 be appended to, not overwritten. If rescue files exist then
67 DAGMan will run the original DAG and rename the rescue files.
68 Any old-style rescue files will be deleted.
69 -dagman DagmanExecutable
71 Allows the specification of an alternate condor_dagman exe‐
72 cutable to be used instead of the one found in the user's
73 path. This must be a fully qualified path.
74 -maxidle NumberOfProcs
76 Sets the maximum number of idle procs allowed before con‐
77 dor_dagman stops submitting more node jobs. If this option is
78 omitted then the number of idle procs is limited by the con‐
79 figuration variable DAGMAN_MAX_JOBS_IDLE which defaults to
80 1000. To disable this limit, set NumberOfProcs to 0. The
81 NumberOfProcs can be exceeded if a nodes job has a queue com‐
82 mand with more than one proc to queue. i.e. queue 500 will
83 submit all procs even if NumberOfProcs is 250. In this case
84 DAGMan will wait for for the number of idle procs to fall be‐
85 low 250 before submitting more jobs to the condor_schedd.
86 -maxjobs NumberOfClusters
88 Sets the maximum number of clusters within the DAG that will
89 be submitted to HTCondor at one time. Each cluster is associ‐
90 ated with one node job no matter how many individual procs
91 are in the cluster. NumberOfClusters is a non-negative inte‐
92 ger. If this option is omitted then the number of clusters is
93 limited by the configuration variable
94 DAGMAN_MAX_JOBS_SUBMITTED which defaults to 0 (unlimited).
95 -maxpre NumberOfPreScripts
97 Sets the maximum number of PRE scripts within the DAG that
98 may be running at one time. NumberOfPreScripts is a non-nega‐
99 tive integer. If this option is omitted, the number of PRE
100 scripts is limited by the configuration variable
101 DAGMAN_MAX_PRE_SCRIPTS which defaults to 20.
102 -maxpost NumberOfPostScripts
104 Sets the maximum number of POST scripts within the DAG that
105 may be running at one time. NumberOfPostScripts is a non-neg‐
106 ative integer. If this option is omitted, the number of POST
107 scripts is limited by the configuration variable
108 DAGMAN_MAX_POST_SCRIPTS which defaults to 20.
109 -notification value
111 Sets the e-mail notification for DAGMan itself. This informa‐
112 tion will be used within the HTCondor submit description file
113 for DAGMan. This file is produced by condor_submit_dag. See
114 the description of notification within condor_submit manual
115 page for a specification of value.
116 -r schedd_name
118 Submit condor_dagman to a condor_schedd on a remote machine.
119 It is assumed that any necessary files will be present on the
120 remote machine via some method like a shared filesystem be‐
121 tween the local and remote machines. The user also requires
122 the correct permissions to submit remotely similarly to con‐
123 dor_submit's -remote option. If other options are desired,
124 including transfer of other input files, consider using the
125 -no_submit option and modifying the resulting submit file for
126 specific needs before using condor_submit on the prouduced
127 DAGMan job submit file.
128 -debug level
130 Passes the the level of debugging output desired to con‐
131 dor_dagman. level is an integer, with values of 0-7 inclu‐
132 sive, where 7 is the most verbose output. See the condor_dag‐
133 man manual page for detailed descriptions of these values. If
134 not specified, no -debug Value is passed to condor_dagman.
135 -usedagdir
137 This optional argument causes condor_dagman to run each spec‐
138 ified DAG as if condor_submit_dag had been run in the direc‐
139 tory containing that DAG file. This option is most useful
140 when running multiple DAGs in a single condor_dagman. Note
141 that the -usedagdir flag must not be used when running an
142 old-style Rescue DAG.
143 -outfile_dir directory
145 Specifies the directory in which the .dagman.out file will be
146 written. The directory may be specified relative to the cur‐
147 rent working directory as condor_submit_dag is executed, or
148 specified with an absolute path. Without this option, the
149 .dagman.out file is placed in the same directory as the first
150 DAG input file listed on the command line.
151 -config ConfigFileName
153 Specifies a configuration file to be used for this DAGMan
154 run. This configuration will apply to all DAGs submitted in
155 via DAGMan. Note that only one custom configuration file can
156 be specified for a DAGMan workflow which will cause a failure
157 if used in conjuntion with a DAG using the CONFIG command.
158 -insert_sub_file FileName
160 Specifies a file to insert into the .condor.sub file created
161 by condor_submit_dag. The specified file must contain only
162 legal submit file commands. Only one file can be inserted.
163 The specified file will override the file set by the configu‐
164 ration variable DAGMAN_INSERT_SUB_FILE. The specified file is
165 inserted into the .condor.sub file before the queue command
166 and any commands specified with the -append option.
167 -append Command
169 Specifies a command to append to the .condor.sub file created
170 by condor_submit_dag. The specified command is appended to
171 the .condor.sub file immediately before the queue command and
172 after any commands added via -insert_sub_file or
173 DAGMAN_INSERT_SUB_FILE. Multiple commands are specified by
174 using the -append option multiple times. Commands with spaces
175 in them must be enclosed in double quotes.
176 -batch-name batch_name
178 Set the batch name for this DAG/workflow. The batch name is
179 displayed by condor_q. If omitted DAGMan will set the batch
180 name to DagFile+ClusterId where DagFile is the name of the
181 primary DAG submitted DAGMan and ClusterId is the DAGMan
182 proper jobs ClusterId. The batch name is set in all jobs sub‐
183 mitted by DAGMan and propagated down into sub-DAGs. Note: set
184 the batch name to ' ' (space) to avoid overriding batch names
185 specified in node job submit files.
186 -autorescue 0|1
188 Whether to automatically run the newest rescue DAG for the
189 given DAG file, if one exists (0 = false, 1 = true).
190 -dorescuefrom number
192 Forces condor_dagman to run the specified rescue DAG number
193 for the given DAG. A value of 0 is the same as not specifying
194 this option. Specifying a non-existent rescue DAG is a fatal
195 error.
196 -load_save filename
198 Specify a file with saved DAG progress to re-run the DAG
199 from. If given a path DAGMan will attempt to read that file
200 following that path. Otherwise, DAGMan will check for the
201 file in the DAG's save_files sub-directory.
202 -allowversionmismatch
204 This optional argument causes condor_dagman to allow a ver‐
205 sion mismatch between condor_dagman itself and the .con‐
206 dor.sub file produced by condor_submit_dag (or, in other
207 words, between condor_submit_dag and condor_dagman). WARNING!
208 This option should be used only if absolutely necessary. Al‐
209 lowing version mismatches can cause subtle problems when run‐
210 ning DAGs.
211 -no_recurse
213 This optional argument causes condor_submit_dag to not run
214 itself recursively on nested DAGs (this is now the default;
215 this flag has been kept mainly for backwards compatibility).
216 -do_recurse
218 This optional argument causes condor_submit_dag to run itself
219 recursively on nested DAGs to pre-produce their .condor.sub
220 files. DAG nodes specified with the SUBDAG EXTERNAL keyword
221 or with submit file names ending in .condor.sub are consid‐
222 ered nested DAGs. This flag is useful when the configuration
223 variable DAGMAN_GENERATE_SUBDAG_SUBMITS is False (Not de‐
224 fault).
225 -update_submit
227 This optional argument causes an existing .condor.sub file to
228 not be treated as an error; rather, the .condor.sub file will
229 be overwritten, but the existing values of -maxjobs, -maxi‐
230 dle, -maxpre, and -maxpost will be preserved.
231 -import_env
233 This optional argument causes condor_submit_dag to import the
234 current environment into the environment command of the .con‐
235 dor.sub file it generates.
236 -include_env Variables
238 This optional argument takes a comma separated list of envi‐
239 roment variables to add to .condor.sub getenv environment
240 filter which causes found matching environment variables to
241 be added to the DAGMan manager jobs environment.
242 -insert_env Key=Value
244 This optional argument takes a delimited string of Key=Value
245 pairs to explicitly set into the .condor.sub files environ‐
246 ment macro. The base delimiter is a semicolon that can be
247 overriden by setting the first character in the string to a
248 valid delimiting character. If multiple -insert_env flags
249 contain the same Key then the last occurances Value will be
250 set in the DAGMan jobs environment.
251 -DumpRescue
253 This optional argument tells condor_dagman to immediately
254 dump a rescue DAG and then exit, as opposed to actually run‐
255 ning the DAG. This feature is mainly intended for testing.
256 The Rescue DAG file is produced whether or not there are
257 parse errors reading the original DAG input file. The name of
258 the file differs if there was a parse error.
259 -valgrind
261 This optional argument causes the submit description file
262 generated for the submission of condor_dagman to be modified.
263 The executable becomes valgrind run on condor_dagman, with a
264 specific set of arguments intended for testing condor_dagman.
265 Note that this argument is intended for testing purposes
266 only. Using the -valgrind option without the necessary val‐
267 grind software installed will cause the DAG to fail. If the
268 DAG does run, it will run much more slowly than usual.
269 -DontAlwaysRunPost
271 This option causes the submit description file generated for
272 the submission of condor_dagman to be modified. It causes
273 condor_dagman to not run the POST script of a node if the PRE
274 script fails.
275 -AlwaysRunPost
277 This option causes the submit description file generated for
278 the submission of condor_dagman to be modified. It causes
279 condor_dagman to always run the POST script of a node, even
280 if the PRE script fails.
281 -priority number
283 Sets the minimum job priority of node jobs submitted and run‐
284 ning under the condor_dagman job submitted by this con‐
285 dor_submit_dag command.
286 -schedd-daemon-ad-file FileName
288 Specifies a full path to a daemon ad file dropped by a con‐
289 dor_schedd. Therefore this allows submission to a specific
290 scheduler if several are available without repeatedly query‐
291 ing the condor_collector. The value for this argument de‐
292 faults to the configuration attribute SCHEDD_DAEMON_AD_FILE.
293 -schedd-address-file FileName
295 Specifies a full path to an address file dropped by a con‐
296 dor_schedd. Therefore this allows submission to a specific
297 scheduler if several are available without repeatedly query‐
298 ing the condor_collector. The value for this argument de‐
299 faults to the configuration attribute SCHEDD_ADDRESS_FILE.
300 -suppress_notification
302 Causes jobs submitted by condor_dagman to not send email no‐
303 tification for events. The same effect can be achieved by
304 setting configuration variable DAGMAN_SUPPRESS_NOTIFICATION
305 to True. This command line option is independent of the -no‐
306 tification command line option, which controls notification
307 for the condor_dagman job itself.
308 -dont_suppress_notification
310 Causes jobs submitted by condor_dagman to defer to content
311 within the submit description file when deciding to send
312 email notification for events. The same effect can be
313 achieved by setting configuration variable
314 DAGMAN_SUPPRESS_NOTIFICATION to False. This command line flag
315 is independent of the -notification command line option,
316 which controls notification for the condor_dagman job itself.
317 If both -dont_suppress_notification and -suppress_notifica‐
318 tion are specified with the same command line, the last argu‐
319 ment is used.
320 -DoRecovery
322 Causes condor_dagman to start in recovery mode. This means
323 that DAGMan reads the relevant .nodes.log file to restore its
324 previous state of node completions and failures to continue
325 running.
326
328 condor_submit_dag will exit with a status value of 0 (zero) upon suc‐
329 cess, and it will exit with the value 1 (one) upon failure.
330
332 To run a single DAG:
333 $ condor_submit_dag diamond.dag
335 To run a DAG when it has already been run and the output files exist:
337 $ condor_submit_dag -force diamond.dag
339 To run a DAG, limiting the number of idle node jobs in the DAG to a
341 maximum of five:
342 $ condor_submit_dag -maxidle 5 diamond.dag
344 To run a DAG, limiting the number of concurrent PRE scripts to 10 and
346 the number of concurrent POST scripts to five:
347 $ condor_submit_dag -maxpre 10 -maxpost 5 diamond.dag
349 To run two DAGs, each of which is set up to run in its own directory:
351 $ condor_submit_dag -usedagdir dag1/diamond1.dag dag2/diamond2.dag
353
355 HTCondor Team
356
358 1990-2023, Center for High Throughput Computing, Computer Sciences De‐
359 partment, University of Wisconsin-Madison, Madison, WI, US. Licensed
360 under the Apache License, Version 2.0.
361 Oct 02, 2023 CONDOR_SUBMIT_DAG(1)