1SPANK(8) Slurm Component SPANK(8)
2
6 SPANK - Slurm Plug-in Architecture for Node and job (K)control
7
10 This manual briefly describes the capabilities of the Slurm Plug-in
11 architecture for Node and job Kontrol (SPANK) as well as the SPANK con‐
12 figuration file: (By default: plugstack.conf.)
13 SPANK provides a very generic interface for stackable plug-ins which
15 may be used to dynamically modify the job launch code in Slurm. SPANK
16 plugins may be built without access to Slurm source code. They need
17 only be compiled against Slurm's spank.h header file, added to the
18 SPANK config file plugstack.conf, and they will be loaded at runtime
19 during the next job launch. Thus, the SPANK infrastructure provides
20 administrators and other developers a low cost, low effort ability to
21 dynamically modify the runtime behavior of Slurm job launch.
22 Note: SPANK plugins using the Slurm APIs need to be recompiled when
24 upgrading Slurm to a new major release.
25
27 SPANK plugins are loaded in up to five separate contexts during a Slurm
28 job. Briefly, the five contexts are:
29 local In local context, the plugin is loaded by srun. (i.e. the
31 "local" part of a parallel job).
32 remote In remote context, the plugin is loaded by slurmstepd. (i.e.
34 the "remote" part of a parallel job).
35 allocator
37 In allocator context, the plugin is loaded in one of the job
38 allocation utilities sbatch or salloc.
39 slurmd In slurmd context, the plugin is loaded in the
41 slurmd daemon itself. Note: Plugins loaded in slurmd context
42 persist for the entire time slurmd is running, so if configura‐
43 tion is changed or plugins are updated, slurmd must be restarted
44 for the changes to take effect.
45 job_script
47 In the job_script context, plugins are loaded in the context of
48 the job prolog or epilog. Note: Plugins are loaded in job_script
49 context on each run on the job prolog or epilog, in a separate
50 address space from plugins in slurmd context. This means there
51 is no state shared between this context and other contexts, or
52 even between one call to slurm_spank_job_prolog or
53 slurm_spank_job_epilog and subsequent calls.
54 In local context, only the init, exit, init_post_opt, and
56 local_user_init functions are called. In allocator context, only the
57 init, exit, and init_post_opt functions are called. Similarly, in
58 slurmd context, only the slurmd_init and slurmd_exit callbacks are
59 active, and in the job_script context, only the job_prolog and job_epi‐
60 log callbacks are used. Plugins may query the context in which they
61 are running with the spank_context and spank_remote functions defined
62 in <slurm/spank.h>.
63 SPANK plugins may be called from multiple points during the Slurm job
65 launch. A plugin may define the following functions:
66 slurm_spank_init
68 Called just after plugins are loaded. In remote context, this is just
69 after job step is initialized. This function is called before any
70 plugin option processing. This function is not called in slurmd con‐
71 text.
72 slurm_spank_slurmd_init
74 Called in slurmd just after the daemon is started.
75 slurm_spank_job_prolog
77 Called at the same time as the job prolog. If this function returns a
78 negative value and the SPANK plugin that contains it is required in
79 the plugstack.conf, the node that this is run on will be drained.
80 slurm_spank_init_post_opt
83 Called at the same point as slurm_spank_init, but after all user
84 options to the plugin have been processed. The reason that the init
85 and init_post_opt callbacks are separated is so that plugins can
86 process system-wide options specified in plugstack.conf in the init
87 callback, then process user options, and finally take some action in
88 slurm_spank_init_post_opt if necessary. In the case of a heteroge‐
89 neous job, slurm_spank_init is invoked once per job component.
90 slurm_spank_local_user_init
92 Called in local (srun) context only after all options have been pro‐
93 cessed. This is called after the job ID and step IDs are available.
94 This happens in srun after the allocation is made, but before tasks
95 are launched.
96 slurm_spank_user_init
98 Called after privileges are temporarily dropped. (remote context
99 only)
100 slurm_spank_task_init_privileged
102 Called for each task just after fork, but before all elevated privi‐
103 leges are dropped. (remote context only)
104 slurm_spank_task_init
106 Called for each task just before execve (2). (remote context only)
107 slurm_spank_task_post_fork
109 Called for each task from parent process after fork (2) is complete.
110 Due to the fact that slurmd does not exec any tasks until all tasks
111 have completed fork (2), this call is guaranteed to run before the
112 user task is executed. (remote context only)
113 slurm_spank_task_exit
115 Called for each task as its exit status is collected by Slurm.
116 (remote context only)
117 slurm_spank_exit
119 Called once just before slurmstepd exits in remote context. In local
120 context, called before srun exits.
121 slurm_spank_job_epilog
123 Called at the same time as the job epilog. If this function returns a
124 negative value and the SPANK plugin that contains it is required in
125 the plugstack.conf, the node that this is run on will be drained.
126 slurm_spank_slurmd_exit
128 Called in slurmd when the daemon is shut down.
129 All of these functions have the same prototype, for example:
131 int slurm_spank_init (spank_t spank, int ac, char *argv[])
133 Where spank is the SPANK handle which must be passed back to Slurm when
136 the plugin calls functions like spank_get_item and spank_getenv. Con‐
137 figured arguments (See CONFIGURATION below) are passed in the argument
138 vector argv with argument count ac.
139 SPANK plugins can query the current list of supported slurm_spank sym‐
141 bols to determine if the current version supports a given plugin hook.
142 This may be useful because the list of plugin symbols may grow in the
143 future. The query is done using the spank_symbol_supported function,
144 which has the following prototype:
145 int spank_symbol_supported (const char *sym);
147 The return value is 1 if the symbol is supported, 0 if not.
150 SPANK plugins do not have direct access to internally defined Slurm
152 data structures. Instead, information about the currently executing job
153 is obtained via the spank_get_item function call.
154 spank_err_t spank_get_item (spank_t spank, spank_item_t item, ...);
156 The spank_get_item call must be passed the current SPANK handle as well
158 as the item requested, which is defined by the passed spank_item_t. A
159 variable number of pointer arguments are also passed, depending on
160 which item was requested by the plugin. A list of the valid values for
161 item is kept in the spank.h header file. Some examples are:
162 S_JOB_UID
164 User id for running job. (uid_t *) is third arg of spank_get_item
165 S_JOB_STEPID
167 Job step id for running job. (uint32_t *) is third arg of
168 spank_get_item.
169 S_TASK_EXIT_STATUS
171 Exit status for exited task. Only valid from slurm_spank_task_exit.
172 (int *) is third arg of spank_get_item.
173 S_JOB_ARGV
175 Complete job command line. Third and fourth args to spank_get_item
176 are (int *, char ***).
177 See spank.h for more details, and EXAMPLES below for an example of
179 spank_get_item usage.
180 SPANK functions in the local and allocator environment should use the
182 getenv, setenv, and unsetenv functions to view and modify the job's
183 environment. SPANK functions in the remote environment should use the
184 spank_getenv, spank_setenv, and spank_unsetenv functions to view and
185 modify the job's environment. spank_getenv searches the job's environ‐
186 ment for the environment variable var and copies the current value into
187 a buffer buf of length len. spank_setenv allows a SPANK plugin to set
188 or overwrite a variable in the job's environment, and spank_unsetenv
189 unsets an environment variable in the job's environment. The prototypes
190 are:
191 spank_err_t spank_getenv (spank_t spank, const char *var,
193 char *buf, int len);
194 spank_err_t spank_setenv (spank_t spank, const char *var,
195 const char *val, int overwrite);
196 spank_err_t spank_unsetenv (spank_t spank, const char *var);
197 These are only necessary in remote context since modifications of the
199 standard process environment using setenv (3), getenv (3), and unsetenv
200 (3) may be used in local context.
201 Functions are also available from within the SPANK plugins to establish
203 environment variables to be exported to the Slurm PrologSlurmctld, Pro‐
204 log, Epilog and EpilogSlurmctld programs (the so-called job control
205 environment). The name of environment variables established by these
206 calls will be prepended with the string SPANK_ in order to avoid any
207 security implications of arbitrary environment variable control. (After
208 all, the job control scripts do run as root or the Slurm user.).
209 These functions are available from local context only.
211 spank_err_t spank_job_control_getenv(spank_t spank, const char *var,
213 char *buf, int len);
214 spank_err_t spank_job_control_setenv(spank_t spank, const char *var,
215 const char *val, int overwrite);
216 spank_err_t spank_job_control_unsetenv(spank_t spank, const char *var);
217 See spank.h for more information, and EXAMPLES below for an example for
219 spank_getenv usage.
220 Many of the described SPANK functions available to plugins return
222 errors via the spank_err_t error type. On success, the return value
223 will be set to ESPANK_SUCCESS, while on failure, the return value will
224 be set to one of many error values defined in slurm/spank.h. The SPANK
225 interface provides a simple function
226 const char * spank_strerror(spank_err_t err);
228 which may be used to translate a spank_err_t value into its string rep‐
230 resentation.
231
234 SPANK plugins also have an interface through which they may define and
235 implement extra job options. These options are made available to the
236 user through Slurm commands such as srun(1), salloc(1), and sbatch(1).
237 If the option is specified by the user, its value is forwarded and reg‐
238 istered with the plugin in slurmd when the job is run. In this way,
239 SPANK plugins may dynamically provide new options and functionality to
240 Slurm.
241 Each option registered by a plugin to Slurm takes the form of a struct
243 spank_option which is declared in <slurm/spank.h> as
244 struct spank_option {
246 char * name;
247 char * arginfo;
248 char * usage;
249 int has_arg;
250 int val;
251 spank_opt_cb_f cb;
252 };
253 Where
256 name is the name of the option. Its length is limited to
258 SPANK_OPTION_MAXLEN defined in <slurm/spank.h>.
259 arginfo
261 is a description of the argument to the option, if the option
262 does take an argument.
263 usage is a short description of the option suitable for --help output.
265 has_arg
267 0 if option takes no argument, 1 if option takes an argument,
268 and 2 if the option takes an optional argument. (See getopt_long
269 (3)).
270 val A plugin-local value to return to the option callback function.
272 cb A callback function that is invoked when the plugin option is
274 registered with Slurm. spank_opt_cb_f is typedef'd in
275 <slurm/spank.h> as
276 typedef int (*spank_opt_cb_f) (int val, const char *optarg,
278 int remote);
279 Where val is the value of the val field in the spank_option
281 struct, optarg is the supplied argument if applicable, and
282 remote is 0 if the function is being called from the "local"
283 host (e.g. host where srun or sbatch/salloc are invoked) or 1
284 from the "remote" host (host where slurmd/slurmstepd run) but
285 only executed by slurmstepd (remote context) if the option was
286 registered for such context.
287 Plugin options may be registered with Slurm using the spank_option_reg‐
289 ister function. This function is only valid when called from the plug‐
290 in's slurm_spank_init handler, and registers one option at a time. The
291 prototype is
292 spank_err_t spank_option_register (spank_t sp,
294 struct spank_option *opt);
295 This function will return ESPANK_SUCCESS on successful registration of
297 an option, or ESPANK_BAD_ARG for errors including invalid spank_t han‐
298 dle, or when the function is not called from the slurm_spank_init func‐
299 tion. All options need to be registered from all contexts in which they
300 will be used. For instance, if an option is only used in local (srun)
301 and remote (slurmd) contexts, then spank_option_register should only be
302 called from within those contexts. For example:
303 if (spank_context() != S_CTX_ALLOCATOR)
305 spank_option_register (sp, opt);
306 If, however, the option is used in all contexts, the spank_option_reg‐
308 ister needs to be called everywhere.
309 In addition to spank_option_register, plugins may also export options
311 to Slurm by defining a table of struct spank_option with the symbol
312 name spank_options. This method, however, is not supported for use with
313 sbatch and salloc (allocator context), thus the use of
314 spank_option_register is preferred. When using the spank_options table,
315 the final element in the array must be filled with zeros. A
316 SPANK_OPTIONS_TABLE_END macro is provided in <slurm/spank.h> for this
317 purpose.
318 When an option is provided by the user on the local side, either by
320 command line options or by environment variables, Slurm will immedi‐
321 ately invoke the option's callback with remote=0. This is meant for the
322 plugin to do local sanity checking of the option before the value is
323 sent to the remote side during job launch. If the argument the user
324 specified is invalid, the plugin should issue an error and issue a
325 non-zero return code from the callback. The plugin should be able to
326 handle cases where the spank option is set multiple times through envi‐
327 ronment variables and command line options. Environment variables are
328 processed before command line options.
329 On the remote side, options and their arguments are registered just
331 after SPANK plugins are loaded and before the spank_init handler is
332 called. This allows plugins to modify behavior of all plugin function‐
333 ality based on the value of user-provided options. (See EXAMPLES below
334 for a plugin that registers an option with Slurm).
335 As an alternative to use of an option callback and global variable,
337 plugins can use the spank_option_getopt option to check for supplied
338 options after option processing. This function has the prototype:
339 spank_err_t spank_option_getopt(spank_t sp,
341 struct spank_option *opt, char **optargp);
342 This function returns ESPANK_SUCCESS if the option defined in the
344 struct spank_option opt has been used by the user. If optargp
345 is non-NULL then it is set to any option argument passed (if the option
346 takes an argument). The use of this method is required to process
347 options in job_script context (slurm_spank_job_prolog and
348 slurm_spank_job_epilog) and can not be used from other contexts.
349
352 The default SPANK plug-in stack configuration file is plugstack.conf in
353 the same directory as slurm.conf(5), though this may be changed via the
354 Slurm config parameter PlugStackConfig. Normally the plugstack.conf
355 file should be identical on all nodes of the cluster. The config file
356 lists SPANK plugins, one per line, along with whether the plugin is
357 required or optional, and any global arguments that are to be passed to
358 the plugin for runtime configuration. Comments are preceded with '#'
359 and extend to the end of the line. If the configuration file is miss‐
360 ing or empty, it will simply be ignored.
361 The format of each non-comment line in the configuration file is:
363 required/optional plugin arguments
365 For example:
367 optional /usr/lib/slurm/test.so
369 Tells slurmd to load the plugin test.so passing no arguments. If a
371 SPANK plugin is required, then failure of any of the plugin's functions
372 will cause slurmd to terminate the job, while optional plugins only
373 cause a warning.
374 If a fully-qualified path is not specified for a plugin, then the cur‐
376 rently configured PluginDir in slurm.conf(5) is searched.
377 SPANK plugins are stackable, meaning that more than one plugin may be
379 placed into the config file. The plugins will simply be called in
380 order, one after the other, and appropriate action taken on failure
381 given that state of the plugin's optional flag.
382 Additional config files or directories of config files may be included
384 in plugstack.conf with the include keyword. The include keyword must
385 appear on its own line, and takes a glob as its parameter, so multiple
386 files may be included from one include line. For example, the following
387 syntax will load all config files in the /etc/slurm/plugstack.conf.d
388 directory, in local collation order:
389 include /etc/slurm/plugstack.conf.d/*
391 which might be considered a more flexible method for building up a
393 spank plugin stack.
394 The SPANK config file is re-read on each job launch, so editing the
396 config file will not affect running jobs. However care should be taken
397 so that a partially edited config file is not read by a launching job.
398
401 Simple SPANK config file:
402 #
404 # SPANK config file
405 #
406 # required? plugin args
407 #
408 optional renice.so min_prio=-10
409 required /usr/lib/slurm/test.so
410 The following is a simple SPANK plugin to modify the nice value of job
413 tasks. This plugin adds a --renice=[prio] option to srun which users
414 can use to set the priority of all remote tasks. Priority may also be
415 specified via a SLURM_RENICE environment variable. A minimum priority
416 may be established via a "min_prio" parameter in plugstack.conf (See
417 above for example).
418 /*
420 * To compile:
421 * gcc -shared -o renice.so renice.c
422 *
423 */
424 #include <sys/types.h>
425 #include <stdio.h>
426 #include <stdlib.h>
427 #include <unistd.h>
428 #include <string.h>
429 #include <sys/resource.h>
430 #include <slurm/spank.h>
432 /*
434 * All spank plugins must define this macro for the
435 * Slurm plugin loader.
436 */
437 SPANK_PLUGIN(renice, 1);
438 #define PRIO_ENV_VAR "SLURM_RENICE"
440 #define PRIO_NOT_SET 42
441 /*
443 * Minimum allowable value for priority. May be
444 * set globally via plugin option min_prio=<prio>
445 */
446 static int min_prio = -20;
447 static int prio = PRIO_NOT_SET;
449 static int _renice_opt_process (int val,
451 const char *optarg,
452 int remote);
453 static int _str2prio (const char *str, int *p2int);
454 /*
456 * Provide a --renice=[prio] option to srun:
457 */
458 struct spank_option spank_options[] =
459 {
460 { "renice", "[prio]",
461 "Re-nice job tasks to priority [prio].", 2, 0,
462 (spank_opt_cb_f) _renice_opt_process
463 },
464 SPANK_OPTIONS_TABLE_END
465 };
466 /*
468 * Called from both srun and slurmd.
469 */
470 int slurm_spank_init (spank_t sp, int ac, char **av)
471 {
472 int i;
473 /* Don't do anything in sbatch/salloc */
475 if (spank_context () == S_CTX_ALLOCATOR)
476 return (0);
477 for (i = 0; i < ac; i++) {
479 if (strncmp ("min_prio=", av[i], 9) == 0) {
480 const char *optarg = av[i] + 9;
481 if (_str2prio (optarg, &min_prio) < 0)
482 slurm_error ("Ignoring invalid min_prio value: %s",
483 av[i]);
484 } else {
485 slurm_error ("renice: Invalid option: %s", av[i]);
486 }
487 }
488 if (!spank_remote (sp))
490 slurm_verbose ("renice: min_prio = %d", min_prio);
491 return (0);
493 }
494 int slurm_spank_task_post_fork (spank_t sp, int ac, char **av)
497 {
498 pid_t pid;
499 int taskid;
500 if (prio == PRIO_NOT_SET) {
502 /* See if SLURM_RENICE env var is set by user */
503 char val [1024];
504 if (spank_getenv (sp, PRIO_ENV_VAR, val, 1024)
506 != ESPANK_SUCCESS)
507 return (0);
508 if (_str2prio (val, &prio) < 0) {
510 slurm_error ("Bad value for %s: %s",
511 PRIO_ENV_VAR, optarg);
512 return (-1);
513 }
514 if (prio < min_prio) {
516 slurm_error ("%s=%d not allowed, using min=%d",
517 PRIO_ENV_VAR, prio, min_prio);
518 }
519 }
520 if (prio < min_prio)
522 prio = min_prio;
523 spank_get_item (sp, S_TASK_GLOBAL_ID, &taskid);
525 spank_get_item (sp, S_TASK_PID, &pid);
526 slurm_info ("re-nicing task%d pid %ld to %ld",
528 taskid, pid, prio);
529 if (setpriority (PRIO_PROCESS, (int) pid,
531 (int) prio) < 0) {
532 slurm_error ("setpriority: %m");
533 return (-1);
534 }
535 return (0);
537 }
538 static int _str2prio (const char *str, int *p2int)
540 {
541 long int l;
542 char *p;
543 l = strtol (str, &p, 10);
545 if ((*p != ' ') || (l < -20) || (l > 20))
546 return (-1);
547 *p2int = (int) l;
549 return (0);
551 }
552 static int _renice_opt_process (int val,
554 const char *optarg,
555 int remote)
556 {
557 if (optarg == NULL) {
558 slurm_error ("renice: invalid argument!");
559 return (-1);
560 }
561 if (_str2prio (optarg, &prio) < 0) {
563 slurm_error ("Bad value for --renice: %s",
564 optarg);
565 return (-1);
566 }
567 if (prio < min_prio) {
569 slurm_error ("--renice=%d not allowed, will use min=%d",
570 prio, min_prio);
571 }
572 return (0);
574 }
575
579 Portions copyright (C) 2010-2018 SchedMD LLC. Copyright (C) 2006 The
580 Regents of the University of California. Produced at Lawrence Liver‐
581 more National Laboratory (cf, DISCLAIMER). CODE-OCEC-09-009. All
582 rights reserved.
583 This file is part of Slurm, a resource management program. For
585 details, see <https://slurm.schedmd.com/>.
586 Slurm is free software; you can redistribute it and/or modify it under
588 the terms of the GNU General Public License as published by the Free
589 Software Foundation; either version 2 of the License, or (at your
590 option) any later version.
591 Slurm is distributed in the hope that it will be useful, but WITHOUT
593 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
594 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
595 for more details.
596
598 /etc/slurm/slurm.conf - Slurm configuration file.
599 /etc/slurm/plugstack.conf - SPANK configuration file.
600 /usr/include/slurm/spank.h - SPANK header file.
601
603 srun(1), slurm.conf(5)
604August 2017 Slurm Component SPANK(8)