1OSRF_PYCOMMON(1) osrf_pycommon OSRF_PYCOMMON(1)
2
3
4
6 osrf_pycommon - osrf_pycommon Documentation
7
8 osrf_pycommon is a python package which contains commonly used Python
9 boilerplate code and patterns. Things like ansi terminal coloring,
10 capturing colored output from programs using subprocess, or even a sim‐
11 ple logging system which provides some nice functionality over the
12 built-in Python logging system.
13
14 The functionality provided here should be generic enough to be reused
15 in arbitrary scenarios and should avoid bringing in dependencies which
16 are not part of the standard Python library. Where possible Windows
17 and Linux/OS X should be supported, and where it cannot it should be
18 gracefully degrading. Code should be pure Python as well as Python 2
19 and Python 3 bilingual.
20
21 Contents:
22
24 This module provides functions and patterns for creating Command Line
25 Interface (CLI) tools.
26
27 Common CLI Functions
28 Commonly used, CLI related functions.
29
30 osrf_pycommon.cli_utils.common.extract_argument_group(args, delimit‐
31 ing_option)
32 Extract a group of arguments from a list of arguments using a
33 delimiter.
34
35 Here is an example:
36
37 >>> extract_argument_group(['foo', '--args', 'bar', '--baz'], '--args')
38 (['foo'], ['bar', '--baz'])
39
40 The group can always be ended using the double hyphen --. In
41 order to pass a double hyphen as arguments, use three hyphens
42 ---. Any set of hyphens encountered after the delimiter, and up
43 to --, which have three or more hyphens and are isolated, will
44 be captured and reduced by one hyphen.
45
46 For example:
47
48 >> extract_argument_group(['foo',
49 '--args', 'bar', '--baz', '---', '--',
50 '--foo-option'], '--args')
51 (['foo', '--foo-option'], ['bar', '--baz', '--'])
52
53 In the result the -- comes from the --- in the input. The
54 --args and the corresponding -- are removed entirely.
55
56 The delimiter and -- terminator combination can also happen mul‐
57 tiple times, in which case the bodies of arguments are combined
58 and returned in the order they appeared.
59
60 For example:
61
62 >> extract_argument_group(['foo',
63 '--args', 'ping', '--',
64 'bar',
65 '--args', 'pong', '--',
66 'baz',
67 '--args', '--'], '--args')
68 (['foo', 'bar', 'baz'], ['ping', 'pong'])
69
70 Note: -- cannot be used as the delimiting_option.
71
72 Parameters
73
74 • args (list) – list of strings which are ordered argu‐
75 ments.
76
77 • delimiting_option (str) – option which denotes where to
78 split the args.
79
80 Returns
81 tuple of arguments before and after the delimiter.
82
83 Return type
84 tuple
85
86 Raises ValueError if the delimiting_option is --.
87
88 osrf_pycommon.cli_utils.common.extract_jobs_flags(arguments)
89 Extracts make job flags from a list of other make flags, i.e.
90 -j8 -l8
91
92 The input arguments are given as a string separated by white‐
93 space. Make job flags are matched and removed from the argu‐
94 ments, and the Make job flags and what is left over from the in‐
95 put arguments are returned.
96
97 If no job flags are encountered, then an empty string is re‐
98 turned as the first element of the returned tuple.
99
100 Examples:
101
102 >> extract_jobs_flags('-j8 -l8')
103 ('-j8 -l8', '')
104 >> extract_jobs_flags('-j8 ')
105 ('-j8', ' ')
106 >> extract_jobs_flags('target -j8 -l8 --some-option')
107 ('-j8 -l8', 'target --some-option')
108 >> extract_jobs_flags('target --some-option')
109 ('', 'target --some-option')
110
111 Parameters
112 arguments (str) – string of space separated arguments
113 which may or may not contain make job flags
114
115 Returns
116 tuple of make jobs flags as a space separated string and
117 leftover arguments as a space separated string
118
119 Return type
120 tuple
121
122 The Verb Pattern
123 The verb pattern is a pattern where a single command aggregates multi‐
124 ple related commands by taking a required positional argument which is
125 the “verb” for the action you want to perform. For example, catkin
126 build is an example of a command and verb pair, where catkin is the
127 command and build is the verb. In this example, the catkin command
128 groups “actions” which are related to catkin together using verbs like
129 build which will build a workspace of catkin packages.
130
131 Command Boilerplate
132 This is an example boilerplate of a command which will use verbs:
133
134 from __future__ import print_function
135
136 import argparse
137 import sys
138
139 from osrf_pycommon.cli_utils.verb_pattern import create_subparsers
140 from osrf_pycommon.cli_utils.verb_pattern import list_verbs
141 from osrf_pycommon.cli_utils.verb_pattern import split_arguments_by_verb
142
143 COMMAND_NAME = '<INSERT COMMAND NAME HERE>'
144
145 VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
146
147
148 def main(sysargs=None):
149 # Assign sysargs if not set
150 sysargs = sys.argv[1:] if sysargs is None else sysargs
151
152 # Create a top level parser
153 parser = argparse.ArgumentParser(
154 description="{0} command".format(COMMAND_NAME)
155 )
156
157 # Generate a list of verbs available
158 verbs = list_verbs(VERBS_ENTRY_POINT)
159
160 # Create the subparsers for each verb and collect the arg preprocessors
161 argument_preprocessors, verb_subparsers = create_subparsers(
162 parser,
163 COMMAND_NAME,
164 verbs,
165 VERBS_ENTRY_POINT,
166 sysargs,
167 )
168
169 # Determine the verb, splitting arguments into pre and post verb
170 verb, pre_verb_args, post_verb_args = split_arguments_by_verb(sysargs)
171
172 # Short circuit -h and --help
173 if '-h' in pre_verb_args or '--help' in pre_verb_args:
174 parser.print_help()
175 sys.exit(0)
176
177 # Error on no verb provided
178 if verb is None:
179 print(parser.format_usage())
180 sys.exit("Error: No verb provided.")
181 # Error on unknown verb provided
182 if verb not in verbs:
183 print(parser.format_usage())
184 sys.exit("Error: Unknown verb '{0}' provided.".format(verb))
185
186 # Short circuit -h and --help for verbs
187 if '-h' in post_verb_args or '--help' in post_verb_args:
188 verb_subparsers[verb].print_help()
189 sys.exit(0)
190
191 # First allow the verb's argument preprocessor to strip any args
192 # and return any "extra" information it wants as a dict
193 processed_post_verb_args, extras = \
194 argument_preprocessors[verb](post_verb_args)
195 # Then allow argparse to process the left over post-verb arguments along
196 # with the pre-verb arguments and the verb itself
197 args = parser.parse_args(pre_verb_args + [verb] + processed_post_verb_args)
198 # Extend the argparse result with the extras from the preprocessor
199 for key, value in extras.items():
200 setattr(args, key, value)
201
202 # Finally call the subparser's main function with the processed args
203 # and the extras which the preprocessor may have returned
204 sys.exit(args.main(args) or 0)
205
206 This function is mostly boilerplate in that it will likely not change
207 much between commands of different types, but it would also be less
208 transparent to have this function created for you. If you are using
209 this boilerplate to implement your command, then you should be careful
210 to update COMMAND_NAME to reflect your command’s name.
211
212 This line defines the entry_point group for your command’s verbs:
213
214 VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
215
216 In the case that your command is called foo then this would become
217 foo.verbs. This name is important because it is how verbs for this
218 command can be provided by your Python package or others. For example,
219 each verb for your command foo will need entry in the setup.py of its
220 containing package, like this:
221
222 setup(
223 ...
224 entry_points={
225 ...
226 'foo.verbs': [
227 'bar = foo.verbs.bar:entry_point_data',
228 ],
229 }
230 )
231
232 You can see here that you are defining bar to be a an entry_point of
233 type foo.verbs which in turn points to a module and reference
234 foo.verbs.bar and entry_point_data. At run time this verb pattern will
235 let your command lookup all things defined as foo.verbs and load up the
236 reference to which they point.
237
238 Adding Verbs
239 In order to add a verb to your command, a few things must happen.
240
241 First you must have an entry in the setup.py as described above. This
242 allows the command to find the entry_point for your verb at run time.
243 The entry_point for these verbs should point to a dictionary which de‐
244 scribes the verb being added.
245
246 This is an example of an entry_point_data dictionary for a verb:
247
248 entry_point_data = dict(
249 verb='build',
250 description='Builds a workspace of packages',
251 # Called for execution, given parsed arguments object
252 main=main,
253 # Called first to setup argparse, given argparse parser
254 prepare_arguments=prepare_arguments,
255 # Called after prepare_arguments, but before argparse.parse_args
256 argument_preprocessor=argument_preprocessor,
257 )
258
259 As you can see this dictionary describes the verb and gives references
260 to functions which allow the command to describe the verb, hook into
261 argparse parameter creation for the verb, and to execute the verb. The
262 verb, description, main, and prepare_arguments keys of the dictionary
263 are required, but the argument_preprocessor key is optional.
264
265 • verb: This is the name of the verb, and is how the command knows
266 which verb implementation to match to a verb on the command line.
267
268 • description: This is used by the argument parsing to describe the
269 verb in --help.
270
271 • prepare_arguments: This function gets called to allow the verb to
272 setup it’s own argparse options. This function should always take one
273 parameter which is the argparse.ArgumentParser for this verb, to
274 which arguments can be added. It can optionally take a second parame‐
275 ter which are the current command line arguments. This is not always
276 needed, but can be useful in some cases. This function should always
277 return the parser.
278
279 • argument_preprocessor: This function is optional, but allows your
280 verb an opportunity to process the raw arguments before they are
281 passed to argparse’s parse_args function. This can be useful when
282 argparse is not capable of processing the options correctly.
283
284 • main: This is the implementation of the verb, it gets called last and
285 is passed the parsed arguments. The return type of this function is
286 used for sys.exit, a return type of None is interpreted as 0.
287
288 Here is an invented example of main, prepare_arguments, and argu‐
289 ment_preprocessor:
290
291 def prepare_arguments(parser):
292 parser.add_argument('--some-argument', action='store_true', default=False)
293 return parser
294
295 def argument_preprocessor(args):
296 extras = {}
297
298 if '-strange-argument' in args:
299 args.remove('-strange-argument')
300 extras['strange_argument'] = True
301
302 return args, extras
303
304 def main(options):
305 print('--some-argument:', options.some_argument)
306 print('-strange-argument:', options.strange_argument)
307 if options.strange_argument:
308 return 1
309 return 0
310
311 The above example is simply to illustrate the signature of these func‐
312 tions and how they might be used.
313
314 Verb Pattern API
315 API for implementing commands and verbs which used the verb pattern.
316
317 osrf_pycommon.cli_utils.verb_pattern.call_prepare_arguments(func,
318 parser, sysargs=None)
319 Call a prepare_arguments function with the correct number of pa‐
320 rameters.
321
322 The prepare_arguments function of a verb can either take one pa‐
323 rameter, parser, or two parameters parser and args, where args
324 are the current arguments being processed.
325
326 Parameters
327
328 • func (Callable) – Callable prepare_arguments function.
329
330 • parser (argparse.ArgumentParser) – parser which is al‐
331 ways passed to the function
332
333 • sysargs (list) – arguments to optionally pass to the
334 function, if needed
335
336 Returns
337 return value of function or the parser if the function
338 returns None.
339
340 Return type
341 argparse.ArgumentParser
342
343 Raises ValueError if a function with the wrong number of parame‐
344 ters is given
345
346 osrf_pycommon.cli_utils.verb_pattern.create_subparsers(parser,
347 cmd_name, verbs, group, sysargs, title=None)
348 Creates argparse subparsers for each verb which can be discov‐
349 ered.
350
351 Using the verbs parameter, the available verbs are iterated
352 through. For each verb a subparser is created for it using the
353 parser parameter. The cmd_name is used to fill the title and
354 description of the add_subparsers function call. The group pa‐
355 rameter is used with each verb to load the verb’s description,
356 prepare_arguments function, and the verb’s argument_preproces‐
357 sors if available. Each verb’s prepare_arguments function is
358 called, allowing them to add arguments. Finally a list of argu‐
359 ment_preprocessors functions and verb subparsers are returned,
360 one for each verb.
361
362 Parameters
363
364 • parser (argparse.ArgumentParser) – parser for this com‐
365 mand
366
367 • cmd_name (str) – name of the command to which the verbs
368 are being added
369
370 • verbs (list) – list of verbs (by name as a string)
371
372 • group (str) – name of the entry_point group for the
373 verbs
374
375 • sysargs (list) – list of system arguments
376
377 • title (str) – optional custom title for the command
378
379 Returns
380 tuple of argument_preprocessors and verb subparsers
381
382 Return type
383 tuple
384
385 osrf_pycommon.cli_utils.verb_pattern.default_argument_preproces‐
386 sor(args)
387 Return unmodified args and an empty dict for extras
388
389 osrf_pycommon.cli_utils.verb_pattern.list_verbs(group)
390 List verbs available for a given entry_point group.
391
392 Parameters
393 group (str) – entry_point group name for the verbs to
394 list
395
396 Returns
397 list of verb names for the given entry_point group
398
399 Return type
400 list of str
401
402 osrf_pycommon.cli_utils.verb_pattern.load_verb_description(verb_name,
403 group)
404 Load description of a verb in a given group by name.
405
406 Parameters
407
408 • verb_name (str) – name of the verb to load, as a string
409
410 • group (str) – entry_point group name which the verb is
411 in
412
413 Returns
414 verb description
415
416 Return type
417 dict
418
419 osrf_pycommon.cli_utils.verb_pattern.split_arguments_by_verb(arguments)
420 Split arguments by verb.
421
422 Given a list of arguments (list of strings), the verb, the pre
423 verb arguments, and the post verb arguments are returned.
424
425 For example:
426
427 >>> args = ['--command-arg1', 'verb', '--verb-arg1', '--verb-arg2']
428 >>> split_arguments_by_verb(args)
429 ('verb', ['--command-arg1'], ['--verb-arg1', '--verb-arg2'])
430
431 Parameters
432 arguments (list) – list of system arguments
433
434 Returns
435 the verb (str), pre verb args (list), and post verb args
436 (list)
437
438 Return type
439 tuple
440
442 This module provides functions for doing process management.
443
444 These are the main sections of this module:
445
446 • Asynchronous Process Utilities
447
448 • Synchronous Process Utilities
449
450 • Utility Functions
451
452 Asynchronous Process Utilities
453 There is a function and class which can be used together with your cus‐
454 tom Tollius or asyncio run loop.
455
456 The osrf_pycommon.process_utils.async_execute_process() function is a
457 coroutine which allows you to run a process and get the output back bit
458 by bit in real-time, either with stdout and stderr separated or com‐
459 bined. This function also allows you to emulate the terminal using a
460 pty simply by toggling a flag in the parameters.
461
462 Along side this coroutine is a Protocol class,
463 osrf_pycommon.process_utils.AsyncSubprocessProtocol, from which you can
464 inherit in order to customize how the yielded output is handled.
465
466 Because this coroutine is built on the trollius/asyncio framework’s
467 subprocess functions, it is portable and should behave the same on all
468 major OS’s. (including on Windows where an IOCP implementation is used)
469
470 async osrf_pycommon.process_utils.async_execute_process(protocol_class,
471 cmd=None, cwd=None, env=None, shell=False, emulate_tty=False,
472 stderr_to_stdout=True)
473 Coroutine to execute a subprocess and yield the output back
474 asynchronously.
475
476 This function is meant to be used with the Python asyncio mod‐
477 ule, which is available via pip with Python 3.3 and built-in to
478 Python 3.4. On Python >= 2.6 you can use the trollius module to
479 get the same functionality, but without using the new yield from
480 syntax.
481
482 Here is an example of how to use this function:
483
484 import asyncio
485 from osrf_pycommon.process_utils import async_execute_process
486 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
487 from osrf_pycommon.process_utils import get_loop
488
489
490 @asyncio.coroutine
491 def setup():
492 transport, protocol = yield from async_execute_process(
493 AsyncSubprocessProtocol, ['ls', '/usr'])
494 returncode = yield from protocol.complete
495 return returncode
496
497 retcode = get_loop().run_until_complete(setup())
498 get_loop().close()
499
500 That same example using trollius would look like this:
501
502 import trollius as asyncio
503 from osrf_pycommon.process_utils import async_execute_process
504 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
505 from osrf_pycommon.process_utils import get_loop
506
507
508 @asyncio.coroutine
509 def setup():
510 transport, protocol = yield asyncio.From(async_execute_process(
511 AsyncSubprocessProtocol, ['ls', '/usr']))
512 returncode = yield asyncio.From(protocol.complete)
513 raise asyncio.Return(returncode)
514
515 retcode = get_loop().run_until_complete(setup())
516 get_loop().close()
517
518 This difference is required because in Python < 3.3 the yield
519 from syntax is not valid.
520
521 In both examples, the first argument is the default
522 AsyncSubprocessProtocol protocol class, which simply prints out‐
523 put from stdout to stdout and output from stderr to stderr.
524
525 If you want to capture and do something with the output or write
526 to the stdin, then you need to subclass from the
527 AsyncSubprocessProtocol class, and override the on_stdout_re‐
528 ceived, on_stderr_received, and on_process_exited functions.
529
530 See the documentation for the AsyncSubprocessProtocol class for
531 more details, but here is an example which uses asyncio from
532 Python 3.4:
533
534 import asyncio
535 from osrf_pycommon.process_utils import async_execute_process
536 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
537 from osrf_pycommon.process_utils import get_loop
538
539
540 class MyProtocol(AsyncSubprocessProtocol):
541 def __init__(self, file_name, **kwargs):
542 self.fh = open(file_name, 'w')
543 AsyncSubprocessProtocol.__init__(self, **kwargs)
544
545 def on_stdout_received(self, data):
546 # Data has line endings intact, but is bytes in Python 3
547 self.fh.write(data.decode('utf-8'))
548
549 def on_stderr_received(self, data):
550 self.fh.write(data.decode('utf-8'))
551
552 def on_process_exited(self, returncode):
553 self.fh.write("Exited with return code: {0}".format(returncode))
554 self.fh.close()
555
556
557 @asyncio.coroutine
558 def log_command_to_file(cmd, file_name):
559
560 def create_protocol(**kwargs):
561 return MyProtocol(file_name, **kwargs)
562
563 transport, protocol = yield from async_execute_process(
564 create_protocol, cmd)
565 returncode = yield from protocol.complete
566 return returncode
567
568 get_loop().run_until_complete(
569 log_command_to_file(['ls', '/'], '/tmp/out.txt'))
570 get_loop().close()
571
572 See the subprocess.Popen class for more details on some of the
573 parameters to this function like cwd, env, and shell.
574
575 See the osrf_pycommon.process_utils.execute_process() function
576 for more details on the emulate_tty parameter.
577
578 Parameters
579
580 • protocol_class (AsyncSubprocessProtocol or a subclass)
581 – Protocol class which handles subprocess callbacks
582
583 • cmd (list) – list of arguments where the executable is
584 the first item
585
586 • cwd (str) – directory in which to run the command
587
588 • env (dict) – a dictionary of environment variable names
589 to values
590
591 • shell (bool) – if True, the cmd variable is interpreted
592 by a the shell
593
594 • emulate_tty (bool) – if True, pty’s are passed to the
595 subprocess for stdout and stderr, see
596 osrf_pycommon.process_utils.execute_process().
597
598 • stderr_to_stdout (bool) – if True, stderr is directed
599 to stdout, so they are not captured separately.
600
601 class osrf_pycommon.process_utils.AsyncSubprocessProtocol(stdin=None,
602 stdout=None, stderr=None)
603 Protocol to subclass to get events from async_execute_process().
604
605 When subclassing this Protocol class, you should override these
606 functions:
607
608 def on_stdout_received(self, data):
609 # ...
610
611 def on_stderr_received(self, data):
612 # ...
613
614 def on_process_exited(self, returncode):
615 # ...
616
617 By default these functions just print the data received from
618 stdout and stderr and does nothing when the process exits.
619
620 Data received by the on_stdout_received and on_stderr_received
621 functions is always in bytes (str in Python2 and bytes in
622 Python3). Therefore, it may be necessary to call .decode() on
623 the data before printing to the screen.
624
625 Additionally, the data received will not be stripped of new
626 lines, so take that into consideration when printing the result.
627
628 You can also override these less commonly used functions:
629
630 def on_stdout_open(self):
631 # ...
632
633 def on_stdout_close(self, exc):
634 # ...
635
636 def on_stderr_open(self):
637 # ...
638
639 def on_stderr_close(self, exc):
640 # ...
641
642 These functions are called when stdout/stderr are opened and
643 closed, and can be useful when using pty’s for example. The exc
644 parameter of the *_close functions is None unless there was an
645 exception.
646
647 In addition to the overridable functions this class has a few
648 useful public attributes. The stdin attribute is a reference to
649 the PipeProto which follows the asyncio.WriteTransport inter‐
650 face. The stdout and stderr attributes also reference their
651 PipeProto. The complete attribute is a asyncio.Future which is
652 set to complete when the process exits and its result is the re‐
653 turn code.
654
655 The complete attribute can be used like this:
656
657 import asyncio
658 from osrf_pycommon.process_utils import async_execute_process
659 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
660 from osrf_pycommon.process_utils import get_loop
661
662
663 @asyncio.coroutine
664 def setup():
665 transport, protocol = yield from async_execute_process(
666 AsyncSubprocessProtocol, ['ls', '-G', '/usr'])
667 retcode = yield from protocol.complete
668 print("Exited with", retcode)
669
670 # This will block until the protocol.complete Future is done.
671 get_loop().run_until_complete(setup())
672 get_loop().close()
673
674 connection_made(transport)
675 Called when a connection is made.
676
677 The argument is the transport representing the pipe con‐
678 nection. To receive data, wait for data_received()
679 calls. When the connection is closed, connection_lost()
680 is called.
681
682 pipe_data_received(fd, data)
683 Called when the subprocess writes data into stdout/stderr
684 pipe.
685
686 fd is int file descriptor. data is bytes object.
687
688 process_exited()
689 Called when subprocess has exited.
690
691 In addtion to these functions, there is a utility function for getting
692 the correct asyncio event loop:
693
694 osrf_pycommon.process_utils.get_loop()
695 This function will return the proper event loop for the subpro‐
696 cess async calls.
697
698 On Unix this just returns asyncio.get_event_loop(), but on Win‐
699 dows it will set and return a asyncio.ProactorEventLoop instead.
700
701 Treatment of File Descriptors
702 Unlike subprocess.Popen, all of the process_utils functions behave the
703 same way on Python versions 2.7 through 3.4, and they do not close in‐
704 heritable <https://docs.python.org/3.4/library/os.html#fd-inheritance>.
705 file descriptors before starting subprocesses. This is equivalent to
706 passing close_fds=False to subprocess.Popen on all Python versions.
707
708 In Python 3.2, the subprocess.Popen default for the close_fds option
709 changed from False to True so that file descriptors opened by the par‐
710 ent process were closed before spawning the child process. In Python
711 3.4, PEP 0446 additionally made it so even when close_fds=False file
712 descriptors which are non-inheritable are still closed before spawning
713 the subprocess.
714
715 If you want to be able to pass file descriptors to subprocesses in
716 Python 3.4 or higher, you will need to make sure they are inheritable
717 <https://docs.python.org/3.4/library/os.html#fd-inheritance>.
718
719 Synchronous Process Utilities
720 For synchronous execution and output capture of subprocess, there are
721 two functions:
722
723 • osrf_pycommon.process_utils.execute_process()
724
725 • osrf_pycommon.process_utils.execute_process_split()
726
727 These functions are not yet using the trollius/asyncio framework as a
728 back-end and therefore on Windows will not stream the data from the
729 subprocess as it does on Unix machines. Instead data will not be
730 yielded until the subprocess is finished and all output is buffered
731 (the normal warnings about long running programs with lots of output
732 apply).
733
734 The streaming of output does not work on Windows because on Windows the
735 select.select() method only works on sockets and not file-like objects
736 which are used with subprocess pipes. asyncio implements Windows sub‐
737 process support by implementing a Proactor event loop based on Window’s
738 IOCP API. One future option will be to implement this synchronous
739 style method using IOCP in this module, but another option is to just
740 make synchronous the asynchronous calls, but there are issues with that
741 as well. In the mean time, if you need streaming of output in both
742 Windows and Unix, use the asynchronous calls.
743
744 osrf_pycommon.process_utils.execute_process(cmd, cwd=None, env=None,
745 shell=False, emulate_tty=False)
746 Executes a command with arguments and returns output line by
747 line.
748
749 All arguments, except emulate_tty, are passed directly to sub‐
750 process.Popen.
751
752 execute_process returns a generator which yields the output,
753 line by line, until the subprocess finishes at which point the
754 return code is yielded.
755
756 This is an example of how this function should be used:
757
758 from __future__ import print_function
759 from osrf_pycommon.process_utils import execute_process
760
761 cmd = ['ls', '-G']
762 for line in execute_process(cmd, cwd='/usr'):
763 if isinstance(line, int):
764 # This is a return code, the command has exited
765 print("'{0}' exited with: {1}".format(' '.join(cmd), line))
766 continue # break would also be appropriate here
767 # In Python 3, it will be a bytes array which needs to be decoded
768 if not isinstance(line, str):
769 line = line.decode('utf-8')
770 # Then print it to the screen
771 print(line, end='')
772
773 stdout and stderr are always captured together and returned line
774 by line through the returned generator. New line characters are
775 preserved in the output, so if re-printing the data take care to
776 use end='' or first rstrip the output lines.
777
778 When emulate_tty is used on Unix systems, commands will identify
779 that they are on a tty and should output color to the screen as
780 if you were running it on the terminal, and therefore there
781 should not be any need to pass arguments like -c color.ui=always
782 to commands like git. Additionally, programs might also behave
783 differently in when emulate_tty is being used, for example,
784 Python will default to unbuffered output when it detects a tty.
785
786 emulate_tty works by using psuedo-terminals on Unix machines,
787 and so if you are running this command many times in parallel
788 (like hundreds of times) then you may get one of a few different
789 OSError’s. For example, “OSError: [Errno 24] Too many open
790 files: ‘/dev/ttyp0’” or “OSError: out of pty devices”. You
791 should also be aware that you share pty devices with the rest of
792 the system, so even if you are not using a lot, it is possible
793 to get this error. You can catch this error before getting data
794 from the generator, so when using emulate_tty you might want to
795 do something like this:
796
797 from __future__ import print_function
798 from osrf_pycommon.process_utils import execute_process
799
800 cmd = ['ls', '-G', '/usr']
801 try:
802 output = execute_process(cmd, emulate_tty=True)
803 except OSError:
804 output = execute_process(cmd, emulate_tty=False)
805 for line in output:
806 if isinstance(line, int):
807 print("'{0}' exited with: {1}".format(' '.join(cmd), line))
808 continue
809 # In Python 3, it will be a bytes array which needs to be decoded
810 if not isinstance(line, str):
811 line = line.decode('utf-8')
812 print(line, end='')
813
814 This way if a pty cannot be opened in order to emulate the tty
815 then you can try again without emulation, and any other OSError
816 should raise again with emulate_tty set to False. Obviously,
817 you only want to do this if emulating the tty is non-critical to
818 your processing, like when you are using it to capture color.
819
820 Any color information that the command outputs as ANSI escape
821 sequences is captured by this command. That way you can print
822 the output to the screen and preserve the color formatting.
823
824 If you do not want color to be in the output, then try setting
825 emulate_tty to False, but that does not guarantee that there is
826 no color in the output, instead it only will cause called pro‐
827 cesses to identify that they are not being run in a terminal.
828 Most well behaved programs will not output color if they detect
829 that they are not being executed in a terminal, but you
830 shouldn’t rely on that.
831
832 If you want to ensure there is no color in the output from an
833 executed process, then use this function:
834
835 osrf_pycommon.terminal_color.remove_ansi_escape_sequences()
836
837 Exceptions can be raised by functions called by the implementa‐
838 tion, for example, subprocess.Popen can raise an OSError when
839 the given command is not found. If you want to check for the
840 existence of an executable on the path, see: which(). However,
841 this function itself does not raise any special exceptions.
842
843 Parameters
844
845 • cmd (list) – list of strings with the first item being
846 a command and subsequent items being any arguments to
847 that command; passed directly to subprocess.Popen.
848
849 • cwd (str) – path in which to run the command, defaults
850 to None which means os.getcwd() is used; passed di‐
851 rectly to subprocess.Popen.
852
853 • env (dict) – environment dictionary to use for execut‐
854 ing the command, default is None which uses the os.env‐
855 iron environment; passed directly to subprocess.Popen.
856
857 • shell (bool) – If True the system shell is used to
858 evaluate the command, default is False; passed directly
859 to subprocess.Popen.
860
861 • emulate_tty (bool) – If True attempts to use a pty to
862 convince subprocess’s that they are being run in a ter‐
863 minal. Typically this is useful for capturing colorized
864 output from commands. This does not work on Windows (no
865 pty’s), so it is considered False even when True. De‐
866 faults to False.
867
868 Returns
869 a generator which yields output from the command line by
870 line
871
872 Return type
873 generator which yields strings
874
875 Availability: Unix (streaming), Windows (blocking)
876
877 osrf_pycommon.process_utils.execute_process_split(cmd, cwd=None,
878 env=None, shell=False, emulate_tty=False)
879 execute_process(), except stderr is returned separately.
880
881 Instead of yielding output line by line until yielding a return
882 code, this function always a triplet of stdout, stderr, and re‐
883 turn code. Each time only one of the three will not be None.
884 Once you receive a non-None return code (type will be int) there
885 will be no more stdout or stderr. Therefore you can use the
886 command like this:
887
888 from __future__ import print_function
889 import sys
890 from osrf_pycommon.process_utils import execute_process_split
891
892 cmd = ['time', 'ls', '-G']
893 for out, err, ret in execute_process_split(cmd, cwd='/usr'):
894 # In Python 3, it will be a bytes array which needs to be decoded
895 out = out.decode('utf-8') if out is not None else None
896 err = err.decode('utf-8') if err is not None else None
897 if ret is not None:
898 # This is a return code, the command has exited
899 print("'{0}' exited with: {1}".format(' '.join(cmd), ret))
900 break
901 if out is not None:
902 print(out, end='')
903 if err is not None:
904 print(err, end='', file=sys.stderr)
905
906 When using this, it is possible that the stdout and stderr data
907 can be returned in a different order than what would happen on
908 the terminal. This is due to the fact that the subprocess is
909 given different buffers for stdout and stderr and so there is a
910 race condition on the subprocess writing to the different buf‐
911 fers and this command reading the buffers. This can be avoided
912 in most scenarios by using emulate_tty, because of the use of
913 pty’s, though the ordering can still not be guaranteed and the
914 number of pty’s is finite as explained in the documentation for
915 execute_process(). For situations where output ordering between
916 stdout and stderr are critical, they should not be returned sep‐
917 arately and instead should share one buffer, and so
918 execute_process() should be used.
919
920 For all other parameters and documentation see:
921 execute_process()
922
923 Availability: Unix (streaming), Windows (blocking)
924
925 Utility Functions
926 Currently there is only one utility function, a Python implementation
927 of the which shell command.
928
929 osrf_pycommon.process_utils.which(cmd, mode=1, path=None, **kwargs)
930 Given a command, mode, and a PATH string, return the path which
931 conforms to the given mode on the PATH, or None if there is no
932 such file.
933
934 mode defaults to os.F_OK | os.X_OK. path defaults to the result
935 of os.environ.get("PATH"), or can be overridden with a custom
936 search path.
937
938 Backported from shutil.which() (‐
939 https://docs.python.org/3.3/library/shutil.html#shutil.which),
940 available in Python 3.3.
941
943 This module provides tools for colorizing terminal output.
944
945 This module defines the ansi escape sequences used for colorizing the
946 output from terminal programs in Linux. You can access the ansi escape
947 sequences using the ansi() function:
948
949 >>> from osrf_pycommon.terminal_color import ansi
950 >>> print(["This is ", ansi('red'), "red", ansi('reset'), "."])
951 ['This is ', '\x1b[31m', 'red', '\x1b[0m', '.']
952
953 You can also use format_color() to do in-line substitution of keys
954 wrapped in @{} markers for their ansi escape sequences:
955
956 >>> from osrf_pycommon.terminal_color import format_color
957 >>> print(format_color("This is @{bf}blue@{reset}.").split())
958 ['This', 'is', '\x1b[34mblue\x1b[0m.']
959
960 This is a list of all of the available substitutions:
961
962 ┌──────────────┬─────────┬──────────┐
963 │Long Form │ Shorter │ Value │
964 ├──────────────┼─────────┼──────────┤
965 │@{blackf} │ @{kf} │ \033[30m │
966 ├──────────────┼─────────┼──────────┤
967 │@{redf} │ @{rf} │ \033[31m │
968 ├──────────────┼─────────┼──────────┤
969 │@{greenf} │ @{gf} │ \033[32m │
970 ├──────────────┼─────────┼──────────┤
971 │@{yellowf} │ @{yf} │ \033[33m │
972 ├──────────────┼─────────┼──────────┤
973 │@{bluef} │ @{bf} │ \033[34m │
974 ├──────────────┼─────────┼──────────┤
975 │@{purplef} │ @{pf} │ \033[35m │
976 ├──────────────┼─────────┼──────────┤
977 │@{cyanf} │ @{cf} │ \033[36m │
978 ├──────────────┼─────────┼──────────┤
979 │@{whitef} │ @{wf} │ \033[37m │
980 ├──────────────┼─────────┼──────────┤
981 │@{blackb} │ @{kb} │ \033[40m │
982 ├──────────────┼─────────┼──────────┤
983 │@{redb} │ @{rb} │ \033[41m │
984 ├──────────────┼─────────┼──────────┤
985 │@{greenb} │ @{gb} │ \033[42m │
986 ├──────────────┼─────────┼──────────┤
987 │@{yellowb} │ @{yb} │ \033[43m │
988 ├──────────────┼─────────┼──────────┤
989 │@{blueb} │ @{bb} │ \033[44m │
990 ├──────────────┼─────────┼──────────┤
991 │@{purpleb} │ @{pb} │ \033[45m │
992 ├──────────────┼─────────┼──────────┤
993 │@{cyanb} │ @{cb} │ \033[46m │
994 ├──────────────┼─────────┼──────────┤
995 │@{whiteb} │ @{wb} │ \033[47m │
996 ├──────────────┼─────────┼──────────┤
997 │@{escape} │ │ \033 │
998 ├──────────────┼─────────┼──────────┤
999 │@{reset} │ @| │ \033[0m │
1000 ├──────────────┼─────────┼──────────┤
1001 │@{boldon} │ @! │ \033[1m │
1002 ├──────────────┼─────────┼──────────┤
1003 │@{italicson} │ @/ │ \033[3m │
1004 ├──────────────┼─────────┼──────────┤
1005 │@{ulon} │ @_ │ \033[4m │
1006 ├──────────────┼─────────┼──────────┤
1007 │@{invon} │ │ \033[7m │
1008 ├──────────────┼─────────┼──────────┤
1009 │@{boldoff} │ │ \033[22m │
1010 ├──────────────┼─────────┼──────────┤
1011 │@{italicsoff} │ │ \033[23m │
1012 └──────────────┴─────────┴──────────┘
1013
1014 │@{uloff} │ │ \033[24m │
1015 ├──────────────┼─────────┼──────────┤
1016 │@{invoff} │ │ \033[27m │
1017 └──────────────┴─────────┴──────────┘
1018
1019 These substitution's values come from the ANSI color escape sequences,
1020 see: http://en.wikipedia.org/wiki/ANSI_escape_code
1021
1022 Also for any of the keys which have a trailing f, you can safely drop
1023 the trailing f and get the same thing.
1024
1025 For example, format_color("@{redf}") and format_color("@{red}") are
1026 functionally equivalent.
1027
1028 Also, many of the substitutions have shorten forms for convenience,
1029 such that @{redf}, @{rf}, @{red}, and @{r} are all the same.
1030
1031 Note that a trailing b is always required when specifying a background.
1032
1033 Some of the most common non-color sequences have {}'less versions.
1034
1035 For example, @{boldon}'s shorter form is @!.
1036
1037 By default, the substitutions (and calls to ansi()) resolve to escape
1038 sequences, but if you call disable_ansi_color_substitution_globally()
1039 then they will resolve to empty strings.
1040
1041 This allows you to always use the substitution strings and disable them
1042 globally when desired.
1043
1044 On Windows the substitutions are always resolved to empty strings as
1045 the ansi escape sequences do not work on Windows. Instead strings an‐
1046 notated with @{} style substitutions or raw \x1b[xxm style ansi escape
1047 sequences must be passed to print_color() in order for colors to be
1048 displayed on windows. Also the print_ansi_color_win32() function can
1049 be used on strings which only contain ansi escape sequences.
1050
1051 NOTE:
1052 There are existing Python modules like colorama which provide ansi
1053 colorization on multiple platforms, so a valid question is: "why
1054 write this module?". The reason for writing this module is to pro‐
1055 vide the color annotation of strings and functions for removing or
1056 replacing ansi escape sequences which are not provided by modules
1057 like colorama. This module could have depended on colorama for col‐
1058 orization on Windows, but colorama works by replacing the built-in
1059 sys.stdout and sys.stderr, which we did not want and it has extra
1060 functionality that we do not need. So, instead of depending on col‐
1061 orama, the Windows color printing code was used as the inspiration
1062 for the Windows color printing in the windows.py module in this ter‐
1063 minal_color package. The colorama license was placed in the header
1064 of that file and the colorama license is compatible with this pack‐
1065 age's license.
1066
1067 osrf_pycommon.terminal_color.ansi(key)
1068 Returns the escape sequence for a given ansi color key.
1069
1070 osrf_pycommon.terminal_color.disable_ansi_color_substitution_globally()
1071 Causes format_color() to replace color annotations with empty
1072 strings.
1073
1074 It also affects ansi().
1075
1076 This is not the case by default, so if you want to make all sub‐
1077 stitutions given to either function mentioned above return empty
1078 strings then call this function.
1079
1080 The default behavior can be restored by calling
1081 enable_ansi_color_substitution_globally().
1082
1083 osrf_pycommon.terminal_color.enable_ansi_color_substitution_globally()
1084 Causes format_color() to replace color annotations with ansi
1085 esacpe sequences.
1086
1087 It also affects ansi().
1088
1089 This is the case by default, so there is no need to call this
1090 everytime.
1091
1092 If you have previously caused all substitutions to evaluate to
1093 an empty string by calling
1094 disable_ansi_color_substitution_globally(), then you can restore
1095 the escape sequences for substitutions by calling this function.
1096
1097 osrf_pycommon.terminal_color.format_color(msg)
1098 Replaces color annotations with ansi escape sequences.
1099
1100 See this module's documentation for the list of available sub‐
1101 stitutions.
1102
1103 If disable_ansi_color_substitution_globally() has been called
1104 then all color annotations will be replaced by empty strings.
1105
1106 Also, on Windows all color annotations will be replaced with
1107 empty strings. If you want colorization on Windows, you must
1108 pass annotated strings to print_color().
1109
1110 Parameters
1111 msg (str) -- string message to be colorized
1112
1113 Returns
1114 colorized string
1115
1116 Return type
1117 str
1118
1119 osrf_pycommon.terminal_color.get_ansi_dict()
1120 Returns a copy of the dictionary of keys and ansi escape se‐
1121 quences.
1122
1123 osrf_pycommon.terminal_color.print_ansi_color_win32(*args, **kwargs)
1124 Prints color string containing ansi escape sequences to console
1125 in Windows.
1126
1127 If called on a non-Windows system, a NotImplementedError occurs.
1128
1129 Does not respect disable_ansi_color_substitution_globally().
1130
1131 Does not substitute color annotations like @{r} or @!, the
1132 string must already contain the \033[1m style ansi escape se‐
1133 quences.
1134
1135 Works by splitting each argument up by ansi escape sequence,
1136 printing the text between the sequences, and doing the corre‐
1137 sponding win32 action for each ansi sequence encountered.
1138
1139 osrf_pycommon.terminal_color.print_color(*args, **kwargs)
1140 Colorizes and prints with an implicit ansi reset at the end
1141
1142 Calls format_color() on each positional argument and then sends
1143 all positional and keyword arguments to print.
1144
1145 If the end keyword argument is not present then the default end
1146 value ansi('reset') + '\n' is used and passed to print.
1147
1148 os.linesep is used to determine the actual value for \n.
1149
1150 Therefore, if you use the end keyword argument be sure to in‐
1151 clude an ansi reset escape sequence if necessary.
1152
1153 On Windows the substituted arguments and keyword arguments are
1154 passed to print_ansi_color_win32() instead of just print.
1155
1156 osrf_pycommon.terminal_color.remove_ansi_escape_senquences(string)
1157 Removes any ansi escape sequences found in the given string and
1158 returns it.
1159
1160 osrf_pycommon.terminal_color.remove_ansi_escape_sequences(string)
1161 Removes any ansi escape sequences found in the given string and
1162 returns it.
1163
1164 osrf_pycommon.terminal_color.sanitize(msg)
1165 Sanitizes the given string to prevent format_color() from sub‐
1166 stituting content.
1167
1168 For example, when the string 'Email: {user}@{org}' is passed to
1169 format_color() the @{org} will be incorrectly recognized as a
1170 colorization annotation and it will fail to substitute with a
1171 KeyError: org.
1172
1173 In order to prevent this, you can first "sanitize" the string,
1174 add color annotations, and then pass the whole string to
1175 format_color().
1176
1177 If you give this function the string 'Email: {user}@{org}', then
1178 it will return 'Email: {{user}}@@{{org}}'. Then if you pass that
1179 to format_color() it will return 'Email: {user}@{org}'. In this
1180 way format_color() is the reverse of this function and so it is
1181 safe to call this function on any incoming data if it will even‐
1182 tually be passed to format_color().
1183
1184 In addition to expanding { => {{, } => }}, and @ => @@, this
1185 function will also replace any instances of @!, @/, @_, and @|
1186 with @{atexclamation}, @{atfwdslash}, @{atunderscore}, and @{at‐
1187 bar} respectively. And then there are corresponding keys in the
1188 ansi dict to convert them back.
1189
1190 For example, if you pass the string '|@ Notice @|' to this func‐
1191 tion it will return '|@@ Notice @{atbar}'. And since ansi('at‐
1192 bar') always returns @|, even when
1193 disable_ansi_color_substitution_globally() has been called, the
1194 result of passing that string to format_color() will be '|@ No‐
1195 tice @|' again.
1196
1197 There are two main strategies for constructing strings which use
1198 both the Python str.format() function and the colorization anno‐
1199 tations.
1200
1201 One way is to just build each piece and concatenate the result:
1202
1203 print_color("@{r}", "{error}".format(error=error_str))
1204 # Or using print (remember to include an ansi reset)
1205 print(format_color("@{r}" + "{error}".format(error=error_str) + "@|"))
1206
1207 Another way is to use this function on the format string, con‐
1208 catenate to the annotations, pass the whole string to
1209 format_color(), and then format the whole thing:
1210
1211 print(format_color("@{r}" + sanitize("{error}") + "@|")
1212 .format(error=error_str))
1213
1214 However, the most common use for this function is to sanitize
1215 incoming strings which may have unknown content:
1216
1217 def my_func(user_content):
1218 print_color("@{y}" + sanitize(user_content))
1219
1220 This function is not intended to be used on strings with color
1221 annotations.
1222
1223 Parameters
1224 msg (str) -- string message to be sanitized
1225
1226 Returns
1227 sanitized string
1228
1229 Return type
1230 str
1231
1232 osrf_pycommon.terminal_color.split_by_ansi_escape_sequence(string, in‐
1233 clude_delimiters=False)
1234 Splits a string into a list using any ansi escape sequence as a
1235 delimiter.
1236
1237 Parameters
1238
1239 • string (str) -- string to be split
1240
1241 • include_delimiters (bool) -- If True include matched
1242 escape sequences in the list (default: False)
1243
1244 Returns
1245 list of strings, split from original string by escape se‐
1246 quences
1247
1248 Return type
1249 list
1250
1251 osrf_pycommon.terminal_color.test_colors(file=None)
1252 Prints a color testing block using print_color()
1253
1255 This module has a miscellaneous set of functions for working with ter‐
1256 minals.
1257
1258 You can use the get_terminal_dimensions() to get the width and height
1259 of the terminal as a tuple.
1260
1261 You can also use the is_tty() function to determine if a given object
1262 is a tty.
1263
1264 exception osrf_pycommon.terminal_utils.GetTerminalDimensionsError
1265 Raised when the terminal dimensions cannot be determined.
1266
1267 osrf_pycommon.terminal_utils.get_terminal_dimensions()
1268 Returns the width and height of the terminal.
1269
1270 Returns
1271 width and height in that order as a tuple
1272
1273 Return type
1274 tuple
1275
1276 Raises GetTerminalDimensionsError when the terminal dimensions
1277 cannot be determined
1278
1279 osrf_pycommon.terminal_utils.is_tty(stream)
1280 Returns True if the given stream is a tty, else False
1281
1282 Parameters
1283 stream -- object to be checked for being a tty
1284
1285 Returns
1286 True if the given object is a tty, otherwise False
1287
1288 Return type
1289 bool
1290
1292 Given that you have a copy of the source code, you can install osrf_py‐
1293 common like this:
1294
1295 $ python setup.py install
1296
1297 NOTE:
1298 If you are installing to a system Python you may need to use sudo.
1299
1300 If you do not want to install osrf_pycommon into your system Python, or
1301 you don’t have access to sudo, then you can use a virtualenv.
1302
1304 Because osrf_pycommon uses setuptools you can (and should) use the
1305 develop feature:
1306
1307 $ python setup.py develop
1308
1309 NOTE:
1310 If you are developing against the system Python, you may need sudo.
1311
1312 This will “install” osrf_pycommon to your Python path, but rather than
1313 copying the source files, it will instead place a marker file in the
1314 PYTHONPATH redirecting Python to your source directory. This allows
1315 you to use it as if it were installed but where changes to the source
1316 code take immediate affect.
1317
1318 When you are done with develop mode you can (and should) undo it like
1319 this:
1320
1321 $ python setup.py develop -u
1322
1323 NOTE:
1324 If you are developing against the system Python, you may need sudo.
1325
1326 That will “uninstall” the hooks into the PYTHONPATH which point to your
1327 source directory, but you should be wary that sometimes console scripts
1328 do not get removed from the bin folder.
1329
1331 In order to run the tests you will need to install nosetests, flake8,
1332 and Mock.
1333
1334 Once you have installed those, then run nosetest in the root of the
1335 osrf_pycommon source directory:
1336
1337 $ nosetests
1338
1340 In order to build the docs you will need to first install Sphinx.
1341
1342 You can build the documentation by invoking the Sphinx provided make
1343 target in the docs folder:
1344
1345 $ # In the docs folder
1346 $ make html
1347 $ open _build/html/index.html
1348
1349 Sometimes Sphinx does not pickup on changes to modules in packages
1350 which utilize the __all__ mechanism, so on repeat builds you may need
1351 to clean the docs first:
1352
1353 $ # In the docs folder
1354 $ make clean
1355 $ make html
1356 $ open _build/html/index.html
1357
1359 William Woodall
1360
1362 2021, Open Source Robotics Foundation
1363
1364
1365
1366
13670.0 Jul 23, 2021 OSRF_PYCOMMON(1)