1LIBPIPELINE(3) BSD Library Functions Manual LIBPIPELINE(3)
2
4 libpipeline — pipeline manipulation library
5
7 #include <pipeline.h>
8
10 libpipeline is a C library for setting up and running pipelines of pro‐
11 cesses, without needing to involve shell command-line parsing which is
12 often error-prone and insecure. This relieves programmers of the need to
13 laboriously construct pipelines using lower-level primitives such as fork
14 and execve.
15 The general way to use libpipeline involves constructing a pipeline
17 structure and adding one or more pipecmd structures to it. A pipecmd
18 represents a subprocess (or “command”), while a pipeline represents a se‐
19 quence of subprocesses each of whose outputs is connected to the next
20 one's input, as in the example ls | grep pattern | less. The calling
21 program may adjust certain properties of each command independently, such
22 as its environment and nice(3) priority, as well as properties of the en‐
23 tire pipeline such as its input and output and the way signals are han‐
24 dled while executing it. The calling program may then start the pipe‐
25 line, read output from it, wait for it to complete, and gather its exit
26 status.
27 Strings passed as const char * function arguments will be copied by the
29 library.
30 Functions to build individual commands
32 pipecmd *pipecmd_new(const char *name)
33 Construct a new command representing execution of a program called
35 name.
36 pipecmd *pipecmd_new_argv(const char *name, va_list argv)
38 pipecmd *pipecmd_new_args(const char *name, ...)
39 Convenience constructors wrapping pipecmd_new() and pipecmd_arg().
41 Construct a new command representing execution of a program called
42 name with arguments. Terminate arguments with NULL.
43 pipecmd *pipecmd_new_argstr(const char *argstr)
45 Split argstr on whitespace to construct a command and arguments,
47 honouring shell-style single-quoting, double-quoting, and back‐
48 slashes, but not other shell evilness like wildcards, semicolons,
49 or backquotes. This is included only to support situations where
50 command arguments are encoded into configuration files and the
51 like. While it is safer than system(3), it still involves signifi‐
52 cant string parsing which is inherently riskier than avoiding it
53 altogether. Please try to avoid using it in new code.
54 typedef void pipecmd_function_type (void *);
56 typedef void pipecmd_function_free_type (void *);
57 pipecmd *pipecmd_new_function(const char *name,
58 pipecmd_function_type *func, pipecmd_function_free_type *free_func,
59 void *data)
60 Construct a new command that calls a given function rather than ex‐
62 ecuting a process.
63 The data argument is passed as the function's only argument, and
65 will be freed before returning using free_func (if non-NULL).
66 pipecmd_* functions that deal with arguments cannot be used with
68 the command returned by this function.
69 pipecmd *pipecmd_new_sequencev(const char *name, va_list cmdv)
71 pipecmd *pipecmd_new_sequence(const char *name, ...)
72 Construct a new command that itself runs a sequence of commands,
74 supplied as command * arguments following name and terminated by
75 NULL. The commands will be executed in forked children; if any ex‐
76 its non-zero then it will terminate the sequence, as with "&&" in
77 shell.
78 pipecmd_* functions that deal with arguments cannot be used with
80 the command returned by this function.
81 pipecmd *pipecmd_new_passthrough(void)
83 Return a new command that just passes data from its input to its
85 output.
86 pipecmd *pipecmd_dup(pipecmd *cmd)
88 Return a duplicate of a command.
90 void pipecmd_arg(pipecmd *cmd, const char *arg)
92 Add an argument to a command.
94 void pipecmd_argf(pipecmd *cmd, const char *format, ...)
96 Convenience function to add an argument with printf substitutions.
98 void pipecmd_argv(pipecmd *cmd, va_list argv)
100 void pipecmd_args(pipecmd *cmd, ...)
101 Convenience functions wrapping pipecmd_arg() to add multiple argu‐
103 ments at once. Terminate arguments with NULL.
104 void pipecmd_argstr(pipecmd *cmd, const char *argstr)
106 Split argstr on whitespace to add a list of arguments, honouring
108 shell-style single-quoting, double-quoting, and backslashes, but
109 not other shell evilness like wildcards, semicolons, or backquotes.
110 This is included only to support situations where command arguments
111 are encoded into configuration files and the like. While it is
112 safer than system(3), it still involves significant string parsing
113 which is inherently riskier than avoiding it altogether. Please
114 try to avoid using it in new code.
115 void pipecmd_get_nargs(pipecmd *cmd)
117 Return the number of arguments to this command. Note that this in‐
119 cludes the command name as the first argument, so the command ‘echo
120 foo bar’ is counted as having three arguments.
121 Added in libpipeline 1.1.0.
123 void pipecmd_nice(pipecmd *cmd, int value)
125 Set the nice(3) value for this command. Defaults to 0. Errors
127 while attempting to set the nice value are ignored, aside from
128 emitting a debug message.
129 void pipecmd_discard_err(pipecmd *cmd, int discard_err)
131 If discard_err is non-zero, redirect this command's standard error
133 to /dev/null. Otherwise, and by default, pass it through. This is
134 usually a bad idea.
135 void pipecmd_chdir(pipecmd *cmd, const char *directory)
137 Change the working directory to directory while running this com‐
139 mand.
140 Added in libpipeline 1.3.0.
142 void pipecmd_fchdir(pipecmd *cmd, int directory_fd)
144 Change the working directory to the directory given by the open
146 file descriptor directory_fd while running this command.
147 Added in libpipeline 1.4.0.
149 void pipecmd_setenv(pipecmd *cmd, const char *name, const char *value)
151 Set environment variable name to value while running this command.
153 void pipecmd_unsetenv(pipecmd *cmd, const char *name)
155 Unset environment variable name while running this command.
157 void pipecmd_clearenv(pipecmd *cmd)
159 Clear the environment while running this command. (Note that envi‐
161 ronment operations work in sequence; pipecmd_clearenv followed by
162 pipecmd_setenv causes the command to have just a single environment
163 variable set.) Beware that this may cause unexpected failures, for
164 example if some of the contents of the environment are necessary to
165 execute programs at all (say, PATH).
166 Added in libpipeline 1.1.0.
168 void pipecmd_pre_exec(pipecmd *cmd, pipecmd_function_type *func,
170 pipecmd_function_free_type *free_func, void *data)
171 Install a pre-exec handler. This will be run immediately before
173 executing the command's payload (process or function). Pass NULL
174 to clear any existing pre-exec handler. The data argument is
175 passed as the function's only argument, and will be freed before
176 returning using free_func (if non-NULL).
177 This is similar to pipeline_install_post_fork, except that is spe‐
179 cific to a single command rather than installing a global handler,
180 and it runs slightly later (immediately before exec rather than im‐
181 mediately after fork).
182 Added in libpipeline 1.5.0.
184 void pipecmd_sequence_command(pipecmd *cmd, pipecmd *child)
186 Add a command to a sequence created using pipecmd_new_sequence().
188 void pipecmd_dump(pipecmd *cmd, FILE *stream)
190 Dump a string representation of a command to stream.
192 char *pipecmd_tostring(pipecmd *cmd)
194 Return a string representation of a command. The caller should
196 free the result.
197 void pipecmd_exec(pipecmd *cmd)
199 Execute a single command, replacing the current process. Never re‐
201 turns, instead exiting non-zero on failure.
202 Added in libpipeline 1.1.0.
204 void pipecmd_free(pipecmd *cmd)
206 Destroy a command. Safely does nothing if cmd is NULL.
208 Functions to build pipelines
210 pipeline *pipeline_new(void)
211 Construct a new pipeline.
213 pipeline *pipeline_new_commandv(pipecmd *cmd1, va_list cmdv)
215 pipeline *pipeline_new_commands(pipecmd *cmd1, ...)
216 Convenience constructors wrapping pipeline_new() and
218 pipeline_command(). Construct a new pipeline consisting of the
219 given list of commands. Terminate commands with NULL.
220 pipeline *pipeline_new_command_argv(const char *name, va_list argv)
222 pipeline *pipeline_new_command_args(const char *name, ...)
223 Construct a new pipeline and add a single command to it.
225 pipeline *pipeline_join(pipeline *p1, pipeline *p2)
227 Joins two pipelines, neither of which are allowed to be started.
229 Discards want_out, want_outfile, and outfd from p1, and want_in,
230 want_infile, and infd from p2.
231 void pipeline_connect(pipeline *source, pipeline *sink, ...)
233 Connect the input of one or more sink pipelines to the output of a
235 source pipeline. The source pipeline may be started, but in that
236 case pipeline_want_out() must have been called with a negative fd;
237 otherwise, calls pipeline_want_out(source, -1). In any event,
238 calls pipeline_want_in(sink, -1) on all sinks, none of which are
239 allowed to be started. Terminate arguments with NULL.
240 This is an application-level connection; data may be intercepted
242 between the pipelines by the program before calling
243 pipeline_pump(), which sets data flowing from the source to the
244 sinks. It is primarily useful when more than one sink pipeline is
245 involved, in which case the pipelines cannot simply be concatenated
246 into one.
247 The result is similar to tee(1), except that output can be sent to
249 more than two places and can easily be sent to multiple processes.
250 void pipeline_command(pipeline *p, pipecmd *cmd)
252 Add a command to a pipeline.
254 void pipeline_command_argv(pipeline *p, const char *name, va_list argv)
256 void pipeline_command_args(pipeline *p, const char *name, ...)
257 Construct a new command and add it to a pipeline in one go.
259 void pipeline_command_argstr(pipeline *p, const char *argstr)
261 Construct a new command from a shell-quoted string and add it to a
263 pipeline in one go. See the comment against pipecmd_new_argstr()
264 above if you're tempted to use this function.
265 void pipeline_commandv(pipeline *p, va_list cmdv)
267 void pipeline_commands(pipeline *p, ...)
268 Convenience functions wrapping pipeline_command() to add multiple
270 commands at once. Terminate arguments with NULL.
271 void pipeline_want_in(pipeline *p, int fd)
273 void pipeline_want_out(pipeline *p, int fd)
274 Set file descriptors to use as the input and output of the whole
276 pipeline. If non-negative, fd is used directly as a file descrip‐
277 tor. If negative, pipeline_start() will create pipes and store the
278 input writing half and the output reading half in the pipeline's
279 infd or outfd field as appropriate. The default is to leave input
280 and output as stdin and stdout unless pipeline_want_infile() or
281 pipeline_want_outfile() respectively has been called.
282 Calling these functions supersedes any previous call to
284 pipeline_want_infile() or pipeline_want_outfile() respectively.
285 void pipeline_want_infile(pipeline *p, const char *file)
287 void pipeline_want_outfile(pipeline *p, const char *file)
288 Set file names to open and use as the input and output of the whole
290 pipeline. This may be more convenient than supplying file descrip‐
291 tors, and guarantees that the files are opened with the same privi‐
292 leges under which the pipeline is run.
293 Calling these functions (even with NULL, which returns to the de‐
295 fault of leaving input and output as stdin and stdout) supersedes
296 any previous call to pipeline_want_in() or pipeline_want_outfile()
297 respectively.
298 The given files will be opened when the pipeline is started. If an
300 output file does not already exist, it is created (with mode 0666
301 modified in the usual way by umask); if it does exist, then it is
302 truncated.
303 void pipeline_ignore_signals(pipeline *p, int ignore_signals)
305 If ignore_signals is non-zero, ignore SIGINT and SIGQUIT in the
307 calling process while the pipeline is running, like system(3).
308 Otherwise, and by default, leave their dispositions unchanged.
309 int pipeline_get_ncommands(pipeline *p)
311 Return the number of commands in this pipeline.
313 pipecmd *pipeline_get_command(pipeline *p, int n)
315 Return command number n from this pipeline, counting from zero, or
317 NULL if n is out of range.
318 pipecmd *pipeline_set_command(pipeline *p, int n, pipecmd *cmd)
320 Set command number n in this pipeline, counting from zero, to cmd,
322 and return the previous command in that position. Do nothing and
323 return NULL if n is out of range.
324 pid_t pipeline_get_pid(pipeline *p, int n)
326 Return the process ID of command number n from this pipeline,
328 counting from zero. The pipeline must be started. Return -1 if n
329 is out of range or if the command has already exited and been
330 reaped.
331 Added in libpipeline 1.2.0.
333 FILE *pipeline_get_infile(pipeline *p)
335 FILE *pipeline_get_outfile(pipeline *p)
336 Get streams corresponding to infd and outfd respectively. The
338 pipeline must be started.
339 void pipeline_dump(pipeline *p, FILE *stream)
341 Dump a string representation of p to stream.
343 char *pipeline_tostring(pipeline *p)
345 Return a string representation of p. The caller should free the
347 result.
348 void pipeline_free(pipeline *p)
350 Destroy a pipeline and all its commands. Safely does nothing if p
352 is NULL. May wait for the pipeline to complete if it has not al‐
353 ready done so.
354 Functions to run pipelines and handle signals
356 typedef void pipeline_post_fork_fn (void);
357 void pipeline_install_post_fork(pipeline_post_fork_fn *fn)
358 Install a post-fork handler. This will be run in any child process
360 immediately after it is forked. For instance, this may be used for
361 cleaning up application-specific signal handlers. Pass NULL to
362 clear any existing post-fork handler.
363 See pipecmd_pre_exec for a similar facility limited to a single
365 command rather than global to the calling process.
366 void pipeline_start(pipeline *p)
368 Start the processes in a pipeline. Installs this library's SIGCHLD
370 handler if not already installed. Calls error (FATAL) on error.
371 The standard file descriptors (0, 1, and 2) must be open before
373 calling this function.
374 int pipeline_wait_all(pipeline *p, int **statuses, int *n_statuses)
376 Wait for a pipeline to complete. Set *statuses to a newly-allo‐
378 cated array of wait statuses, as returned by waitpid(2), and
379 *n_statuses to the length of that array. The return value is simi‐
380 lar to the exit status that a shell would return, with some modifi‐
381 cations. If the last command exits with a signal (other than
382 SIGPIPE, which is considered equivalent to exiting zero), then the
383 return value is 128 plus the signal number; if the last command ex‐
384 its normally but non-zero, then the return value is its exit sta‐
385 tus; if any other command exits non-zero, then the return value is
386 127; otherwise, the return value is 0. This means that the return
387 value is only 0 if all commands in the pipeline exit successfully.
388 int pipeline_wait(pipeline *p)
390 Wait for a pipeline to complete and return its combined exit sta‐
392 tus, calculated as for pipeline_wait_all().
393 int pipeline_run(pipeline *p)
395 Start a pipeline, wait for it to complete, and free it, all in one
397 go.
398 void pipeline_pump(pipeline *p, ...)
400 Pump data among one or more pipelines connected using
402 pipeline_connect() until all source pipelines have reached end-of-
403 file and all data has been written to all sinks (or failed). All
404 relevant pipelines must be supplied: that is, no pipeline that has
405 been connected to a source pipeline may be supplied unless that
406 source pipeline is also supplied. Automatically starts all pipe‐
407 lines if they are not already started, but does not wait for them.
408 Terminate arguments with NULL.
409 Functions to read output from pipelines
411 In general, output is returned as a pointer into a buffer owned by the
412 pipeline, which is automatically freed when pipeline_free() is called.
413 This saves the caller from having to explicitly free individual blocks of
414 output data.
415 const char *pipeline_read(pipeline *p, size_t *len)
417 Read len bytes of data from the pipeline, returning the data block.
419 len is updated with the number of bytes read.
420 const char *pipeline_peek(pipeline *p, size_t *len)
422 Look ahead in the pipeline's output for len bytes of data, return‐
424 ing the data block. len is updated with the number of bytes read.
425 The starting position of the next read or peek is not affected by
426 this call.
427 size_t pipeline_peek_size(pipeline *p)
429 Return the number of bytes of data that can be read using
431 pipeline_read() or pipeline_peek() solely from the peek cache,
432 without having to read from the pipeline itself (and thus poten‐
433 tially block).
434 void pipeline_peek_skip(pipeline *p, size_t len)
436 Skip over and discard len bytes of data from the peek cache. As‐
438 serts that enough data is available to skip, so you may want to
439 check using pipeline_peek_size() first.
440 const char *pipeline_readline(pipeline *p)
442 Read a line of data from the pipeline, returning it.
444 const char *pipeline_peekline(pipeline *p)
446 Look ahead in the pipeline's output for a line of data, returning
448 it. The starting position of the next read or peek is not affected
449 by this call.
450 Signal handling
452 libpipeline installs a signal handler for SIGCHLD, and collects the exit
453 status of child processes in pipeline_wait(). Applications using this
454 library must either refrain from changing the disposition of SIGCHLD (in
455 other words, must rely on libpipeline for all child process handling) or
456 else must make sure to restore libpipeline's SIGCHLD handler before call‐
457 ing any of its functions.
458 If the ignore_signals flag is set in a pipeline (which is the default),
460 then the SIGINT and SIGQUIT signals will be ignored in the parent process
461 while child processes are running. This mirrors the behaviour of
462 system(3).
463 libpipeline leaves child processes with the default disposition of
465 SIGPIPE, namely to terminate the process. It ignores SIGPIPE in the par‐
466 ent process while running pipeline_pump().
467 Reaping of child processes
469 libpipeline installs a SIGCHLD handler that will attempt to reap child
470 processes which have exited. This calls waitpid(2) with -1, so it will
471 reap any child process, not merely those created by way of this library.
472 At present, this means that if the calling program forks other child pro‐
473 cesses which may exit while a pipeline is running, the program is not
474 guaranteed to be able to collect exit statuses of those processes.
475 You should not rely on this behaviour, and in future it may be modified
477 either to reap only child processes created by this library or to provide
478 a way to return foreign statuses to the application. Please contact the
479 author if you have an example application and would like to help design
480 such an interface.
481
483 If the PIPELINE_DEBUG environment variable is set to “1”, then
484 libpipeline will emit debugging messages on standard error.
485 If the PIPELINE_QUIET environment variable is set to any value, then
487 libpipeline will refrain from printing an error message when a subprocess
488 is terminated by a signal. Added in libpipeline 1.4.0.
489
491 In the following examples, function names starting with pipecmd_ or
492 pipeline_ are real libpipeline functions, while any other function names
493 are pseudocode.
494 The simplest case is simple. To run a single command, such as mv source
496 dest:
497 pipeline *p = pipeline_new_command_args ("mv", source, dest, NULL);
499 int status = pipeline_run (p);
500 libpipeline is often used to mimic shell pipelines, such as the following
502 example:
503 zsoelim < input-file | tbl | nroff -mandoc -Tutf8
505 The code to construct this would be:
507 pipeline *p;
509 int status;
510 p = pipeline_new ();
512 pipeline_want_infile (p, "input-file");
513 pipeline_command_args (p, "zsoelim", NULL);
514 pipeline_command_args (p, "tbl", NULL);
515 pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
516 status = pipeline_run (p);
517 You might want to construct a command more dynamically:
519 pipecmd *manconv = pipecmd_new_args ("manconv", "-f", from_code,
521 "-t", "UTF-8", NULL);
522 if (quiet)
523 pipecmd_arg (manconv, "-q");
524 pipeline_command (p, manconv);
525 Perhaps you want an environment variable set only while running a certain
527 command:
528 pipecmd *less = pipecmd_new ("less");
530 pipecmd_setenv (less, "LESSCHARSET", lesscharset);
531 You might find yourself needing to pass the output of one pipeline to
533 several other pipelines, in a “tee” arrangement:
534 pipeline *source, *sink1, *sink2;
536 source = make_source ();
538 sink1 = make_sink1 ();
539 sink2 = make_sink2 ();
540 pipeline_connect (source, sink1, sink2, NULL);
541 /* Pump data among these pipelines until there's nothing left. */
542 pipeline_pump (source, sink1, sink2, NULL);
543 pipeline_free (sink2);
544 pipeline_free (sink1);
545 pipeline_free (source);
546 Maybe one of your commands is actually an in-process function, rather
548 than an external program:
549 pipecmd *inproc = pipecmd_new_function ("in-process", &func,
551 NULL, NULL);
552 pipeline_command (p, inproc);
553 Sometimes your program needs to consume the output of a pipeline, rather
555 than sending it all to some other subprocess:
556 pipeline *p = make_pipeline ();
558 const char *line;
559 pipeline_want_out (p, -1);
561 pipeline_start (p);
562 line = pipeline_peekline (p);
563 if (!strstr (line, "coding: UTF-8"))
564 printf ("Unicode text follows:0);
565 while (line = pipeline_readline (p))
566 printf (" %s", line);
567 pipeline_free (p);
568
570 fork(2), execve(2), system(3), popen(3).
571
573 Most of libpipeline was written by Colin Watson <cjwatson@debian.org>,
574 originally for use in man-db. The initial version was based very loosely
575 on the run_pipeline() function in GNU groff, written by James Clark
576 <jjc@jclark.com>. It also contains library code by Markus Armbruster,
577 and by various contributors to Gnulib.
578 libpipeline is licensed under the GNU General Public License, version 3
580 or later. See the README file for full details.
581
583 Using this library in a program which runs any other child processes
584 and/or installs its own SIGCHLD handler is unlikely to work.
585GNU April 28, 2022 GNU