1ARCSUB(1)                   NorduGrid Users Manual                   ARCSUB(1)
2
3
4

NAME

6       arcsub - ARC Submission
7
8

DESCRIPTION

10       The  arcsub command is used for submitting jobs to Grid enabled comput‐
11       ing resources.
12
13

SYNOPSIS

15       arcsub [options] [filename ...]
16
17

OPTIONS

19       -c, --cluster=name
20              select one or more computing elements: name can be an alias  for
21              a single CE, a group of CEs or a URL
22
23       -g, --index=name
24              select one or more registries: name can be an alias for a single
25              registry, a group of registries or a URL
26
27       -R, --rejectdiscovery=URL
28              skip the service with the given URL during service discovery
29
30       -S, --submissioninterface=InterfaceName
31              only  use  this  interface   for   submitting   (e.g.   org.nor‐
32              dugrid.gridftpjob,          org.ogf.glue.emies.activitycreation,
33              org.ogf.bes)
34
35       -I, --infointerface=InterfaceName
36              the computing element specified  by  URL  at  the  command  line
37              should  be  queried  using  this information interface (possible
38              options: org.nordugrid.ldapng, org.nordugrid.ldapglue2, org.nor‐
39              dugrid.wsrfglue2, org.ogf.glue.emies.resourceinfo)
40
41       -e, --jobdescrstring=String
42              jobdescription string describing the job to be submitted
43
44       -f, --jobdescrfile=filename
45              jobdescription file describing the job to be submitted
46
47       -j, --joblist=filename
48              the   file   storing  information  about  active  jobs  (default
49              ~/.arc/jobs.xml)
50
51       -o, --jobids-to-file=filename
52              the IDs of the submitted jobs will be appended to this file
53
54       -D, --dryrun
55              submit jobs as dry run (no submission to batch system)
56
57           --direct
58              submit directly - no resource discovery or matchmaking
59
60       -x, --dumpdescription
61              do not submit - dump job description in the language accepted by
62              the target
63
64       -P, --listplugins
65              list the available plugins
66
67       -t, --timeout=seconds
68              timeout in seconds (default 20)
69
70       -z, --conffile=filename
71              configuration file (default ~/.arc/client.conf)
72
73       -d, --debug=debuglevel
74              FATAL, ERROR, WARNING, INFO, VERBOSE or DEBUG
75
76       -b, --broker=broker
77              selected  broker:  Random (default), FastestQueue or custom. Use
78              -P to find possible options.
79
80       -v, --version
81              print version information
82
83       -?, --help
84              print help
85

ARGUMENTS

87       filename ...
88              job description files describing the jobs to be submitted
89

EXTENDED DESCRIPTION

91       arcsub is the key command when submitting jobs to Grid enabled  comput‐
92       ing  resources with the ARC client. As default arcsub is able to submit
93       jobs to A-REX, CREAM and EMI ES enabled computing elements  (CEs),  and
94       as always for successful submission you need to be authenticated at the
95       targeted computing  services.  Since  arcsub  is  build  on  a  modular
96       library,  modules  can  be  installed which enables submission to other
97       targets, e.g. the classic ARC CE Grid-Manager.
98
99       Job submission can be accomplished by specifying a job description file
100       to  submit as an argument. arcsub will then by default perform resource
101       discovery on the Grid and then the discovered resources will be matched
102       to  the  job  description  and  ranked  according  to the chosen broker
103       (--broker option). If no Grid environment has been  configured,  please
104       contact  your system administrator, or setup one yourself in the client
105       configuration file (see files section). Another option is to explicitly
106       specify  a  registry  service (or multiple) to arcsub using the --index
107       option, which accepts an URL, alias or group.  Alternatively a specific
108       CE (or multiple) can be targeted by using the --cluster option. If such
109       a scenario is the most common, it is worthwhile to specify those CEs in
110       the  client configuration as default services, which makes it superflu‐
111       ous to specify them as argument. In the same manner aliases and groups,
112       defined  in the configuration file, can be utilized, and can be used as
113       argument to the --cluster or --index options.  In all of the above sce‐
114       narios  arcsub  obtains resource information from the services which is
115       then used for matchmaking against the  job  description,  however  that
116       step  can  be  avoided by specifying the --direct option, in which case
117       the job description is submitted directly to first specified endpoint.
118
119       The format of a classic GRIDFTP-based cluster URLs:
120       [ldap://]<hostname>[:2135/nordugrid-cluster-name=<hostname>,Mds-Vo-
121       name=local,o=grid]
122       Only  the  hostname  part  has  to be specified, the rest of the URL is
123       automatically generated.
124
125       The format of an A-REX URL is:
126       [https://]<hostname>[:<port>][/<path>]
127       Here the port is 443 by default, but the path cannot be guessed, so  if
128       it  is  not  specified, then the service is assumed to live on the root
129       path.
130
131       Job descriptions can also be specified using the --jobdescrfile  option
132       which  expect  the  file  name  of  the description as argument, or the
133       --jobdescrstring option which expect as argument the job description as
134       a string, and both options can be specified multiple times and one does
135       not exclude the other. The default supported job description  languages
136       are xRSL and EMIES ADL.
137
138       If  the  job description is successfully submitted a job-ID is returned
139       and printed. This job-ID uniquely identifies the job while it is  being
140       executed.   On  the  other hand it is also possible that no CEs matches
141       the constraints defined in the description in which case no  submission
142       will  be  done.  Upon successful submission, the job-ID along with more
143       technical job information is stored in  the  job-list  file  (described
144       below).  The  stored information enables the job management commands of
145       the ARC client to manage jobs easily, and thus the job-ID need  not  to
146       be  saved  manually. By default the job-list file is stored in the .arc
147       directory in the home directory of the user, however  another  location
148       can be specified using the --joblist option taking the location of this
149       file as argument. If the --joblist option was used  during  submission,
150       it  should  also be specified in the consecutive commands when managing
151       the job. If a Computing Element has multiple job submission  interfaces
152       (e.g.  gridftp,  EMI-ES, BES), then the brokering algorithm will choose
153       one of them.   With  the  --submissioninterface  option  the  requested
154       interface  can be specified, and in that case only those Computing Ele‐
155       ments will be considered which has that specific  interface,  and  only
156       that interface will be used to submit the jobs.
157
158       As mentioned above registry or index services can be specified with the
159       --index option. Specifying one or multiple index servers instructs  the
160       arcsub  command  to  query the servers for registered CEs, the returned
161       CEs will then be matched against the job description and those matching
162       will  be ranked by the chosen broker (see below) and submission will be
163       tried in order until successful or reaching the end.  From the returned
164       list  of  CEs  it  might happen that a troublesome or undesirable CE is
165       selected for submission, in that case it possible to reject that  clus‐
166       ter  using  the --rejectdiscovery option and providing the URL (or just
167       the hostname) of the CE, which will disregard that CE as a  target  for
168       submission.
169
170       When multiple CEs are targeted for submission, the resource broker will
171       be used to filter out CEs  which  do  not  match  the  job  description
172       requirements  and  then  rank  the  remaining  CEs.  The broker used by
173       default will rank the CEs randomly, however a different broker  can  be
174       chosen by using the --broker option, which takes the name of the broker
175       as argument. The broker type can also be specified in client.conf.  The
176       brokers  available can be seen using arcsub -P.  By default the follow‐
177       ing brokers are available:
178
179       Random (default)
180              Chooses a random CE matching the job requirements.
181
182       FastestQueue
183              Ranks matching CEs according to the length of the job  queue  at
184              the CEs, ranking those with shortest queue first/highest.
185
186       Benchmark
187              Ranks  matching  CEs  according  to a specified benchmark, which
188              should be specified by appending the broker name  with  ':'  and
189              then  the  name  of  the benchmark. If no option is given to the
190              Benchmark broker then  CEs  will  be  ranked  according  to  the
191              'specint2000' benchmark.
192
193       Data   Ranks  matching CEs according to the amount of input data cached
194              by each CE, by querying the CE. Only  CEs  with  the  A-REX  BES
195              interface support this operation.
196
197       Null   Choose a random CE with no filtering at all of CEs.
198
199       PythonBroker
200              User-defined  custom  brokers  can be created in Python. See the
201              example broker SampleBroker.py or ACIXBroker.py (like Data  bro‐
202              ker  but  uses the ARC Cache Index) that come installed with ARC
203              for more details of how to write your own broker. A PythonBroker
204              is specified by --broker PythonBroker:Filename.Class:args, where
205              Filename is the file containing the class Class which implements
206              the broker interface. The directory containing this file must be
207              in the PYTHONPATH. args is optional and allows specifying  argu‐
208              ments to the broker.
209
210       Before  submission,  arcsub performs an intelligent modification of the
211       job description (adding or modifying attributes,  even  converting  the
212       description  language  to  fit the needs of the CE) ensuring that it is
213       valid. The modified job description can be printed  by  specifying  the
214       --dumpdescription option. The format, i.e. job description language, of
215       the printed job description cannot be specified, and will be that which
216       will  be sent to and accepted by the chosen target. Further information
217       from arcsub can be obtained by increasing the verbosity, which is  done
218       with  the  --debug option where the default verbosity level is WARNING.
219       Setting the level to DEBUG will show all messages, while setting it  to
220       FATAL will only show fatal log messages.
221
222       To validate your job description without actually submitting a job, use
223       the --dryrun option: it will capture possible syntax or  other  errors,
224       but  will  instruct  the site not to submit the job for execution. Only
225       the grid-manager (ARC0) and A-REX (ARC1) CEs support this feature.
226
227

EXAMPLES

229       Submission of a job description file "helloworld.adl" to the Grid
230       arcsub helloworld.adl
231
232       A information index server (registry) can also be queried  for  CEs  to
233       submit to:
234       arcsub -g registry.example.com helloworld.adl
235
236       Submission  of  a  job  description  file  "helloworld.adl" to ce.exam‐
237       ple.com:
238       arcsub -c ce.example.com helloworld.adl
239
240       Direct submission to a CE is done as:
241       arcsub --direct -c cd.example.com helloworld.adl
242
243       The job description can also be specified directly on the command  line
244       as shown in the example, using the XRSL job description language:
245       arcsub -c example.com/arex -e \
246       ´&(executable="/bin/echo")(arguments="Hello World!")´
247
248       When submitting against CEs retrieved from information index servers it
249       might be useful to do resource brokering:
250       arcsub -g registry.example.com -b FastestQueue helloworld.adl
251
252       If the job has a large input data set, it can be useful to send it to a
253       CE  where  those  files are already cached. The ACIX broker can be used
254       for this:
255       arcsub  -g  registry.example.com  -b   PythonBroker:ACIXBroker.ACIXBro‐
256       ker:https://cacheindex.ndgf.org:6443/data/index helloworld.adl
257
258       Disregarding  a specific CE for submission submitting against an infor‐
259       mation index server:
260       arcsub -g  registry.example.com  -R  badcomputingelement.com/arex  hel‐
261       loworld.adl
262
263       Dumping the job description is done as follows:
264       arcsub -c example.com/arex -x helloworld.adl
265
266

FILES

268       ~/.arc/client.conf
269              Some  options  can be given default values by specifying them in
270              the ARC client configuration file. Registry and  computing  ele‐
271              ment  services can be specified in separate sections of the con‐
272              fig.  The  default  services  can   be   specified   by   adding
273              'default=yes' attribute to the section of the service, thus when
274              no --cluster or --index options are given these will be used for
275              submission.  Each service has an alias, and can be member of any
276              number of groups. Then specifying the alias or the name  of  the
277              group  with  the  --cluster  or  --index options will select the
278              given services.  By using the --conffile option a different con‐
279              figuration  file  can  be  used than the default. Note that some
280              installations also have a system client configuration file, how‐
281              ever  attributes  in  the  client one takes precedence, and then
282              command line options takes precedence  over  configuration  file
283              attributes.
284
285
286       ~/.arc/jobs.xml
287              This  a local list of the user's active jobs. When a job is suc‐
288              cessfully submitted it is added to this  list  and  when  it  is
289              removed  from  the  remote cluster it is removed from this list.
290              This list is used as the list of all active jobs when  the  user
291              specifies  the  --all  option  to the various NorduGrid ARC user
292              interface commands. By using the --joblist  option  a  different
293              file can be used than the default.
294
295

ENVIRONMENT VARIABLES

297       X509_USER_PROXY
298              The  location  of  the  user's Grid proxy file. Shouldn't be set
299              unless the proxy is in a non-standard location.
300
301
302       ARC_LOCATION
303              The location where ARC is installed can  be  specified  by  this
304              variable.  If  not specified the install location will be deter‐
305              mined from the path to the command being executed, and  if  this
306              fails a WARNING will be given stating the location which will be
307              used.
308
309
310       ARC_PLUGIN_PATH
311              The location of ARC plugins can be specified by  this  variable.
312              Multiple  locations  can be specified by separating them by : (;
313              in Windows). The default location is $ARC_LOCATION/lib/arc (\ in
314              Windows).
315
316
318       APACHE LICENSE Version 2.0
319
320

AUTHOR

322       ARC    software   is   developed   by   the   NorduGrid   Collaboration
323       (http://www.nordugrid.org), please consult the AUTHORS file distributed
324       with    ARC.    Please    report   bugs   and   feature   requests   to
325       http://bugzilla.nordugrid.org
326
327

SEE ALSO

329       arccat(1), arcclean(1), arccp(1),  arcget(1),  arcinfo(1),  arckill(1),
330       arcls(1),  arcmkdir(1),  arcproxy(1),  arcrenew(1), arcresub(1), arcre‐
331       sume(1), arcrm(1), arcstat(1), arcsync(1), arctest(1)
332
333
334
335
336NorduGrid ARC 6.6.0               2020-04-27                         ARCSUB(1)
Impressum