1OSRF_PYCOMMON(1) osrf_pycommon OSRF_PYCOMMON(1)
2
6 osrf_pycommon - osrf_pycommon Documentation
7 osrf_pycommon is a python package which contains commonly used Python
9 boilerplate code and patterns. Things like ansi terminal coloring,
10 capturing colored output from programs using subprocess, or even a sim‐
11 ple logging system which provides some nice functionality over the
12 built-in Python logging system.
13 The functionality provided here should be generic enough to be reused
15 in arbitrary scenarios and should avoid bringing in dependencies which
16 are not part of the standard Python library. Where possible Windows,
17 Linux, and macOS should be supported, and where it cannot it should be
18 gracefully degrading. Code should be pure Python 3.
19 Contents:
21
23 This module provides functions and patterns for creating Command Line
24 Interface (CLI) tools.
25 Common CLI Functions
27 Commonly used, CLI related functions.
28 osrf_pycommon.cli_utils.common.extract_argument_group(args, delimit‐
30 ing_option)
31 Extract a group of arguments from a list of arguments using a
32 delimiter.
33 Here is an example:
35 >>> extract_argument_group(['foo', '--args', 'bar', '--baz'], '--args')
37 (['foo'], ['bar', '--baz'])
38 The group can always be ended using the double hyphen --. In
40 order to pass a double hyphen as arguments, use three hyphens
41 ---. Any set of hyphens encountered after the delimiter, and up
42 to --, which have three or more hyphens and are isolated, will
43 be captured and reduced by one hyphen.
44 For example:
46 >> extract_argument_group(['foo',
48 '--args', 'bar', '--baz', '---', '--',
49 '--foo-option'], '--args')
50 (['foo', '--foo-option'], ['bar', '--baz', '--'])
51 In the result the -- comes from the --- in the input. The
53 --args and the corresponding -- are removed entirely.
54 The delimiter and -- terminator combination can also happen mul‐
56 tiple times, in which case the bodies of arguments are combined
57 and returned in the order they appeared.
58 For example:
60 >> extract_argument_group(['foo',
62 '--args', 'ping', '--',
63 'bar',
64 '--args', 'pong', '--',
65 'baz',
66 '--args', '--'], '--args')
67 (['foo', 'bar', 'baz'], ['ping', 'pong'])
68 Note: -- cannot be used as the delimiting_option.
70 Parameters
72 • args (list) – list of strings which are ordered argu‐
74 ments.
75 • delimiting_option (str) – option which denotes where to
77 split the args.
78 Returns
80 tuple of arguments before and after the delimiter.
81 Return type
83 tuple
84 Raises ValueError if the delimiting_option is --.
86 osrf_pycommon.cli_utils.common.extract_jobs_flags(arguments)
88 Extracts make job flags from a list of other make flags, i.e.
89 -j8 -l8
90 The input arguments are given as a string separated by white‐
92 space. Make job flags are matched and removed from the argu‐
93 ments, and the Make job flags and what is left over from the in‐
94 put arguments are returned.
95 If no job flags are encountered, then an empty string is re‐
97 turned as the first element of the returned tuple.
98 Examples:
100 >> extract_jobs_flags('-j8 -l8')
102 ('-j8 -l8', '')
103 >> extract_jobs_flags('-j8 ')
104 ('-j8', ' ')
105 >> extract_jobs_flags('target -j8 -l8 --some-option')
106 ('-j8 -l8', 'target --some-option')
107 >> extract_jobs_flags('target --some-option')
108 ('', 'target --some-option')
109 Parameters
111 arguments (str) – string of space separated arguments
112 which may or may not contain make job flags
113 Returns
115 tuple of make jobs flags as a space separated string and
116 leftover arguments as a space separated string
117 Return type
119 tuple
120 The Verb Pattern
122 The verb pattern is a pattern where a single command aggregates multi‐
123 ple related commands by taking a required positional argument which is
124 the “verb” for the action you want to perform. For example, catkin
125 build is an example of a command and verb pair, where catkin is the
126 command and build is the verb. In this example, the catkin command
127 groups “actions” which are related to catkin together using verbs like
128 build which will build a workspace of catkin packages.
129 Command Boilerplate
131 This is an example boilerplate of a command which will use verbs:
132 from __future__ import print_function
134 import argparse
136 import sys
137 from osrf_pycommon.cli_utils.verb_pattern import create_subparsers
139 from osrf_pycommon.cli_utils.verb_pattern import list_verbs
140 from osrf_pycommon.cli_utils.verb_pattern import split_arguments_by_verb
141 COMMAND_NAME = '<INSERT COMMAND NAME HERE>'
143 VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
145 def main(sysargs=None):
148 # Assign sysargs if not set
149 sysargs = sys.argv[1:] if sysargs is None else sysargs
150 # Create a top level parser
152 parser = argparse.ArgumentParser(
153 description="{0} command".format(COMMAND_NAME)
154 )
155 # Generate a list of verbs available
157 verbs = list_verbs(VERBS_ENTRY_POINT)
158 # Create the subparsers for each verb and collect the arg preprocessors
160 argument_preprocessors, verb_subparsers = create_subparsers(
161 parser,
162 COMMAND_NAME,
163 verbs,
164 VERBS_ENTRY_POINT,
165 sysargs,
166 )
167 # Determine the verb, splitting arguments into pre and post verb
169 verb, pre_verb_args, post_verb_args = split_arguments_by_verb(sysargs)
170 # Short circuit -h and --help
172 if '-h' in pre_verb_args or '--help' in pre_verb_args:
173 parser.print_help()
174 sys.exit(0)
175 # Error on no verb provided
177 if verb is None:
178 print(parser.format_usage())
179 sys.exit("Error: No verb provided.")
180 # Error on unknown verb provided
181 if verb not in verbs:
182 print(parser.format_usage())
183 sys.exit("Error: Unknown verb '{0}' provided.".format(verb))
184 # Short circuit -h and --help for verbs
186 if '-h' in post_verb_args or '--help' in post_verb_args:
187 verb_subparsers[verb].print_help()
188 sys.exit(0)
189 # First allow the verb's argument preprocessor to strip any args
191 # and return any "extra" information it wants as a dict
192 processed_post_verb_args, extras = \
193 argument_preprocessors[verb](post_verb_args)
194 # Then allow argparse to process the left over post-verb arguments along
195 # with the pre-verb arguments and the verb itself
196 args = parser.parse_args(pre_verb_args + [verb] + processed_post_verb_args)
197 # Extend the argparse result with the extras from the preprocessor
198 for key, value in extras.items():
199 setattr(args, key, value)
200 # Finally call the subparser's main function with the processed args
202 # and the extras which the preprocessor may have returned
203 sys.exit(args.main(args) or 0)
204 This function is mostly boilerplate in that it will likely not change
206 much between commands of different types, but it would also be less
207 transparent to have this function created for you. If you are using
208 this boilerplate to implement your command, then you should be careful
209 to update COMMAND_NAME to reflect your command’s name.
210 This line defines the entry_point group for your command’s verbs:
212 VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
214 In the case that your command is called foo then this would become
216 foo.verbs. This name is important because it is how verbs for this
217 command can be provided by your Python package or others. For example,
218 each verb for your command foo will need entry in the setup.py of its
219 containing package, like this:
220 setup(
222 ...
223 entry_points={
224 ...
225 'foo.verbs': [
226 'bar = foo.verbs.bar:entry_point_data',
227 ],
228 }
229 )
230 You can see here that you are defining bar to be a an entry_point of
232 type foo.verbs which in turn points to a module and reference
233 foo.verbs.bar and entry_point_data. At run time this verb pattern will
234 let your command lookup all things defined as foo.verbs and load up the
235 reference to which they point.
236 Adding Verbs
238 In order to add a verb to your command, a few things must happen.
239 First you must have an entry in the setup.py as described above. This
241 allows the command to find the entry_point for your verb at run time.
242 The entry_point for these verbs should point to a dictionary which de‐
243 scribes the verb being added.
244 This is an example of an entry_point_data dictionary for a verb:
246 entry_point_data = dict(
248 verb='build',
249 description='Builds a workspace of packages',
250 # Called for execution, given parsed arguments object
251 main=main,
252 # Called first to setup argparse, given argparse parser
253 prepare_arguments=prepare_arguments,
254 # Called after prepare_arguments, but before argparse.parse_args
255 argument_preprocessor=argument_preprocessor,
256 )
257 As you can see this dictionary describes the verb and gives references
259 to functions which allow the command to describe the verb, hook into
260 argparse parameter creation for the verb, and to execute the verb. The
261 verb, description, main, and prepare_arguments keys of the dictionary
262 are required, but the argument_preprocessor key is optional.
263 • verb: This is the name of the verb, and is how the command knows
265 which verb implementation to match to a verb on the command line.
266 • description: This is used by the argument parsing to describe the
268 verb in --help.
269 • prepare_arguments: This function gets called to allow the verb to
271 setup it’s own argparse options. This function should always take one
272 parameter which is the argparse.ArgumentParser for this verb, to
273 which arguments can be added. It can optionally take a second parame‐
274 ter which are the current command line arguments. This is not always
275 needed, but can be useful in some cases. This function should always
276 return the parser.
277 • argument_preprocessor: This function is optional, but allows your
279 verb an opportunity to process the raw arguments before they are
280 passed to argparse’s parse_args function. This can be useful when
281 argparse is not capable of processing the options correctly.
282 • main: This is the implementation of the verb, it gets called last and
284 is passed the parsed arguments. The return type of this function is
285 used for sys.exit, a return type of None is interpreted as 0.
286 Here is an invented example of main, prepare_arguments, and argu‐
288 ment_preprocessor:
289 def prepare_arguments(parser):
291 parser.add_argument('--some-argument', action='store_true', default=False)
292 return parser
293 def argument_preprocessor(args):
295 extras = {}
296 if '-strange-argument' in args:
298 args.remove('-strange-argument')
299 extras['strange_argument'] = True
300 return args, extras
302 def main(options):
304 print('--some-argument:', options.some_argument)
305 print('-strange-argument:', options.strange_argument)
306 if options.strange_argument:
307 return 1
308 return 0
309 The above example is simply to illustrate the signature of these func‐
311 tions and how they might be used.
312 Verb Pattern API
314 API for implementing commands and verbs which used the verb pattern.
315 osrf_pycommon.cli_utils.verb_pattern.call_prepare_arguments(func,
317 parser, sysargs=None)
318 Call a prepare_arguments function with the correct number of pa‐
319 rameters.
320 The prepare_arguments function of a verb can either take one pa‐
322 rameter, parser, or two parameters parser and args, where args
323 are the current arguments being processed.
324 Parameters
326 • func (Callable) – Callable prepare_arguments function.
328 • parser (argparse.ArgumentParser) – parser which is al‐
330 ways passed to the function
331 • sysargs (list) – arguments to optionally pass to the
333 function, if needed
334 Returns
336 return value of function or the parser if the function
337 returns None.
338 Return type
340 argparse.ArgumentParser
341 Raises ValueError if a function with the wrong number of parame‐
343 ters is given
344 osrf_pycommon.cli_utils.verb_pattern.create_subparsers(parser,
346 cmd_name, verbs, group, sysargs, title=None)
347 Creates argparse subparsers for each verb which can be discov‐
348 ered.
349 Using the verbs parameter, the available verbs are iterated
351 through. For each verb a subparser is created for it using the
352 parser parameter. The cmd_name is used to fill the title and
353 description of the add_subparsers function call. The group pa‐
354 rameter is used with each verb to load the verb’s description,
355 prepare_arguments function, and the verb’s argument_preproces‐
356 sors if available. Each verb’s prepare_arguments function is
357 called, allowing them to add arguments. Finally a list of argu‐
358 ment_preprocessors functions and verb subparsers are returned,
359 one for each verb.
360 Parameters
362 • parser (argparse.ArgumentParser) – parser for this com‐
364 mand
365 • cmd_name (str) – name of the command to which the verbs
367 are being added
368 • verbs (list) – list of verbs (by name as a string)
370 • group (str) – name of the entry_point group for the
372 verbs
373 • sysargs (list) – list of system arguments
375 • title (str) – optional custom title for the command
377 Returns
379 tuple of argument_preprocessors and verb subparsers
380 Return type
382 tuple
383 osrf_pycommon.cli_utils.verb_pattern.default_argument_preproces‐
385 sor(args)
386 Return unmodified args and an empty dict for extras
387 osrf_pycommon.cli_utils.verb_pattern.list_verbs(group)
389 List verbs available for a given entry_point group.
390 Parameters
392 group (str) – entry_point group name for the verbs to
393 list
394 Returns
396 list of verb names for the given entry_point group
397 Return type
399 list of str
400 osrf_pycommon.cli_utils.verb_pattern.load_verb_description(verb_name,
402 group)
403 Load description of a verb in a given group by name.
404 Parameters
406 • verb_name (str) – name of the verb to load, as a string
408 • group (str) – entry_point group name which the verb is
410 in
411 Returns
413 verb description
414 Return type
416 dict
417 osrf_pycommon.cli_utils.verb_pattern.split_arguments_by_verb(arguments)
419 Split arguments by verb.
420 Given a list of arguments (list of strings), the verb, the pre
422 verb arguments, and the post verb arguments are returned.
423 For example:
425 >>> args = ['--command-arg1', 'verb', '--verb-arg1', '--verb-arg2']
427 >>> split_arguments_by_verb(args)
428 ('verb', ['--command-arg1'], ['--verb-arg1', '--verb-arg2'])
429 Parameters
431 arguments (list) – list of system arguments
432 Returns
434 the verb (str), pre verb args (list), and post verb args
435 (list)
436 Return type
438 tuple
439
441 This module provides functions for doing process management.
442 These are the main sections of this module:
444 • Asynchronous Process Utilities
446 • Synchronous Process Utilities
448 • Utility Functions
450 Asynchronous Process Utilities
452 There is a function and class which can be used together with your cus‐
453 tom asyncio run loop.
454 The osrf_pycommon.process_utils.async_execute_process() function is a
456 coroutine which allows you to run a process and get the output back bit
457 by bit in real-time, either with stdout and stderr separated or com‐
458 bined. This function also allows you to emulate the terminal using a
459 pty simply by toggling a flag in the parameters.
460 Along side this coroutine is a Protocol class,
462 osrf_pycommon.process_utils.AsyncSubprocessProtocol, from which you can
463 inherit in order to customize how the yielded output is handled.
464 Because this coroutine is built on the asyncio framework’s subprocess
466 functions, it is portable and should behave the same on all major OS’s.
467 (including on Windows where an IOCP implementation is used)
468 async osrf_pycommon.process_utils.async_execute_process(protocol_class,
470 cmd=None, cwd=None, env=None, shell=False, emulate_tty=False,
471 stderr_to_stdout=True)
472 Coroutine to execute a subprocess and yield the output back
473 asynchronously.
474 This function is meant to be used with the Python asyncio mod‐
476 ule, which is available in Python 3.5 or greater.
477 Here is an example of how to use this function:
479 import asyncio
481 from osrf_pycommon.process_utils import async_execute_process
482 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
483 from osrf_pycommon.process_utils import get_loop
484 async def setup():
487 transport, protocol = await async_execute_process(
488 AsyncSubprocessProtocol, ['ls', '/usr'])
489 returncode = await protocol.complete
490 return returncode
491 retcode = get_loop().run_until_complete(setup())
493 get_loop().close()
494 Tthe first argument is the default AsyncSubprocessProtocol pro‐
496 tocol class, which simply prints output from stdout to stdout
497 and output from stderr to stderr.
498 If you want to capture and do something with the output or write
500 to the stdin, then you need to subclass from the
501 AsyncSubprocessProtocol class, and override the on_stdout_re‐
502 ceived, on_stderr_received, and on_process_exited functions.
503 See the documentation for the AsyncSubprocessProtocol class for
505 more details, but here is an example which uses asyncio from
506 Python 3.5:
507 import asyncio
509 from osrf_pycommon.process_utils import async_execute_process
510 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
511 from osrf_pycommon.process_utils import get_loop
512 class MyProtocol(AsyncSubprocessProtocol):
515 def __init__(self, file_name, **kwargs):
516 self.fh = open(file_name, 'w')
517 AsyncSubprocessProtocol.__init__(self, **kwargs)
518 def on_stdout_received(self, data):
520 # Data has line endings intact, but is bytes in Python 3
521 self.fh.write(data.decode('utf-8'))
522 def on_stderr_received(self, data):
524 self.fh.write(data.decode('utf-8'))
525 def on_process_exited(self, returncode):
527 self.fh.write("Exited with return code: {0}".format(returncode))
528 self.fh.close()
529 async def log_command_to_file(cmd, file_name):
532 def create_protocol(**kwargs):
534 return MyProtocol(file_name, **kwargs)
535 transport, protocol = await async_execute_process(
537 create_protocol, cmd)
538 returncode = await protocol.complete
539 return returncode
540 get_loop().run_until_complete(
542 log_command_to_file(['ls', '/'], '/tmp/out.txt'))
543 get_loop().close()
544 See the subprocess.Popen class for more details on some of the
546 parameters to this function like cwd, env, and shell.
547 See the osrf_pycommon.process_utils.execute_process() function
549 for more details on the emulate_tty parameter.
550 Parameters
552 • protocol_class (AsyncSubprocessProtocol or a subclass)
554 – Protocol class which handles subprocess callbacks
555 • cmd (list) – list of arguments where the executable is
557 the first item
558 • cwd (str) – directory in which to run the command
560 • env (dict) – a dictionary of environment variable names
562 to values
563 • shell (bool) – if True, the cmd variable is interpreted
565 by a the shell
566 • emulate_tty (bool) – if True, pty’s are passed to the
568 subprocess for stdout and stderr, see
569 osrf_pycommon.process_utils.execute_process().
570 • stderr_to_stdout (bool) – if True, stderr is directed
572 to stdout, so they are not captured separately.
573 class osrf_pycommon.process_utils.AsyncSubprocessProtocol(stdin=None,
575 stdout=None, stderr=None)
576 Protocol to subclass to get events from async_execute_process().
577 When subclassing this Protocol class, you should override these
579 functions:
580 def on_stdout_received(self, data):
582 # ...
583 def on_stderr_received(self, data):
585 # ...
586 def on_process_exited(self, returncode):
588 # ...
589 By default these functions just print the data received from
591 stdout and stderr and does nothing when the process exits.
592 Data received by the on_stdout_received and on_stderr_received
594 functions is always in bytes. Therefore, it may be necessary to
595 call .decode() on the data before printing to the screen.
596 Additionally, the data received will not be stripped of new
598 lines, so take that into consideration when printing the result.
599 You can also override these less commonly used functions:
601 def on_stdout_open(self):
603 # ...
604 def on_stdout_close(self, exc):
606 # ...
607 def on_stderr_open(self):
609 # ...
610 def on_stderr_close(self, exc):
612 # ...
613 These functions are called when stdout/stderr are opened and
615 closed, and can be useful when using pty’s for example. The exc
616 parameter of the *_close functions is None unless there was an
617 exception.
618 In addition to the overridable functions this class has a few
620 useful public attributes. The stdin attribute is a reference to
621 the PipeProto which follows the asyncio.WriteTransport inter‐
622 face. The stdout and stderr attributes also reference their
623 PipeProto. The complete attribute is a asyncio.Future which is
624 set to complete when the process exits and its result is the re‐
625 turn code.
626 The complete attribute can be used like this:
628 import asyncio
630 from osrf_pycommon.process_utils import async_execute_process
631 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
632 from osrf_pycommon.process_utils import get_loop
633 async def setup():
636 transport, protocol = await async_execute_process(
637 AsyncSubprocessProtocol, ['ls', '-G', '/usr'])
638 retcode = await protocol.complete
639 print("Exited with", retcode)
640 # This will block until the protocol.complete Future is done.
642 get_loop().run_until_complete(setup())
643 get_loop().close()
644 connection_made(transport)
646 Called when a connection is made.
647 The argument is the transport representing the pipe con‐
649 nection. To receive data, wait for data_received()
650 calls. When the connection is closed, connection_lost()
651 is called.
652 pipe_data_received(fd, data)
654 Called when the subprocess writes data into stdout/stderr
655 pipe.
656 fd is int file descriptor. data is bytes object.
658 process_exited()
660 Called when subprocess has exited.
661 In addtion to these functions, there is a utility function for getting
663 the correct asyncio event loop:
664 osrf_pycommon.process_utils.get_loop()
666 This function will return the proper event loop for the subpro‐
667 cess async calls.
668 On Unix this just returns asyncio.get_event_loop(), but on Win‐
670 dows it will set and return a asyncio.ProactorEventLoop instead.
671 Treatment of File Descriptors
673 Like Python 3.4’s subprocess.Popen (and newer versions), all of the
674 process_utils functions do not close inheritable
675 <https://docs.python.org/3.4/library/os.html#fd-inheritance> file de‐
676 scriptors before starting subprocesses. This is equivalent to passing
677 close_fds=False to subprocess.Popen on all Python versions.
678 For historical context, in Python 3.2, the subprocess.Popen default for
680 the close_fds option changed from False to True so that file descrip‐
681 tors opened by the parent process were closed before spawning the child
682 process. In Python 3.4, PEP 0446 additionally made it so even when
683 close_fds=False file descriptors which are non-inheritable are still
684 closed before spawning the subprocess.
685 If you want to be able to pass file descriptors to subprocesses in
687 Python 3.4 or higher, you will need to make sure they are inheritable
688 <https://docs.python.org/3.4/library/os.html#fd-inheritance>.
689 Synchronous Process Utilities
691 For synchronous execution and output capture of subprocess, there are
692 two functions:
693 • osrf_pycommon.process_utils.execute_process()
695 • osrf_pycommon.process_utils.execute_process_split()
697 These functions are not yet using the asyncio framework as a back-end
699 and therefore on Windows will not stream the data from the subprocess
700 as it does on Unix machines. Instead data will not be yielded until
701 the subprocess is finished and all output is buffered (the normal warn‐
702 ings about long running programs with lots of output apply).
703 The streaming of output does not work on Windows because on Windows the
705 select.select() method only works on sockets and not file-like objects
706 which are used with subprocess pipes. asyncio implements Windows sub‐
707 process support by implementing a Proactor event loop based on Window’s
708 IOCP API. One future option will be to implement this synchronous
709 style method using IOCP in this module, but another option is to just
710 make synchronous the asynchronous calls, but there are issues with that
711 as well. In the mean time, if you need streaming of output in both
712 Windows and Unix, use the asynchronous calls.
713 osrf_pycommon.process_utils.execute_process(cmd, cwd=None, env=None,
715 shell=False, emulate_tty=False)
716 Executes a command with arguments and returns output line by
717 line.
718 All arguments, except emulate_tty, are passed directly to sub‐
720 process.Popen.
721 execute_process returns a generator which yields the output,
723 line by line, until the subprocess finishes at which point the
724 return code is yielded.
725 This is an example of how this function should be used:
727 from __future__ import print_function
729 from osrf_pycommon.process_utils import execute_process
730 cmd = ['ls', '-G']
732 for line in execute_process(cmd, cwd='/usr'):
733 if isinstance(line, int):
734 # This is a return code, the command has exited
735 print("'{0}' exited with: {1}".format(' '.join(cmd), line))
736 continue # break would also be appropriate here
737 # In Python 3, it will be a bytes array which needs to be decoded
738 if not isinstance(line, str):
739 line = line.decode('utf-8')
740 # Then print it to the screen
741 print(line, end='')
742 stdout and stderr are always captured together and returned line
744 by line through the returned generator. New line characters are
745 preserved in the output, so if re-printing the data take care to
746 use end='' or first rstrip the output lines.
747 When emulate_tty is used on Unix systems, commands will identify
749 that they are on a tty and should output color to the screen as
750 if you were running it on the terminal, and therefore there
751 should not be any need to pass arguments like -c color.ui=always
752 to commands like git. Additionally, programs might also behave
753 differently in when emulate_tty is being used, for example,
754 Python will default to unbuffered output when it detects a tty.
755 emulate_tty works by using psuedo-terminals on Unix machines,
757 and so if you are running this command many times in parallel
758 (like hundreds of times) then you may get one of a few different
759 OSError’s. For example, “OSError: [Errno 24] Too many open
760 files: ‘/dev/ttyp0’” or “OSError: out of pty devices”. You
761 should also be aware that you share pty devices with the rest of
762 the system, so even if you are not using a lot, it is possible
763 to get this error. You can catch this error before getting data
764 from the generator, so when using emulate_tty you might want to
765 do something like this:
766 from __future__ import print_function
768 from osrf_pycommon.process_utils import execute_process
769 cmd = ['ls', '-G', '/usr']
771 try:
772 output = execute_process(cmd, emulate_tty=True)
773 except OSError:
774 output = execute_process(cmd, emulate_tty=False)
775 for line in output:
776 if isinstance(line, int):
777 print("'{0}' exited with: {1}".format(' '.join(cmd), line))
778 continue
779 # In Python 3, it will be a bytes array which needs to be decoded
780 if not isinstance(line, str):
781 line = line.decode('utf-8')
782 print(line, end='')
783 This way if a pty cannot be opened in order to emulate the tty
785 then you can try again without emulation, and any other OSError
786 should raise again with emulate_tty set to False. Obviously,
787 you only want to do this if emulating the tty is non-critical to
788 your processing, like when you are using it to capture color.
789 Any color information that the command outputs as ANSI escape
791 sequences is captured by this command. That way you can print
792 the output to the screen and preserve the color formatting.
793 If you do not want color to be in the output, then try setting
795 emulate_tty to False, but that does not guarantee that there is
796 no color in the output, instead it only will cause called pro‐
797 cesses to identify that they are not being run in a terminal.
798 Most well behaved programs will not output color if they detect
799 that they are not being executed in a terminal, but you
800 shouldn’t rely on that.
801 If you want to ensure there is no color in the output from an
803 executed process, then use this function:
804 osrf_pycommon.terminal_color.remove_ansi_escape_sequences()
806 Exceptions can be raised by functions called by the implementa‐
808 tion, for example, subprocess.Popen can raise an OSError when
809 the given command is not found. If you want to check for the
810 existence of an executable on the path, see: which(). However,
811 this function itself does not raise any special exceptions.
812 Parameters
814 • cmd (list) – list of strings with the first item being
816 a command and subsequent items being any arguments to
817 that command; passed directly to subprocess.Popen.
818 • cwd (str) – path in which to run the command, defaults
820 to None which means os.getcwd() is used; passed di‐
821 rectly to subprocess.Popen.
822 • env (dict) – environment dictionary to use for execut‐
824 ing the command, default is None which uses the os.env‐
825 iron environment; passed directly to subprocess.Popen.
826 • shell (bool) – If True the system shell is used to
828 evaluate the command, default is False; passed directly
829 to subprocess.Popen.
830 • emulate_tty (bool) – If True attempts to use a pty to
832 convince subprocess’s that they are being run in a ter‐
833 minal. Typically this is useful for capturing colorized
834 output from commands. This does not work on Windows (no
835 pty’s), so it is considered False even when True. De‐
836 faults to False.
837 Returns
839 a generator which yields output from the command line by
840 line
841 Return type
843 generator which yields strings
844 Availability: Unix (streaming), Windows (blocking)
846 osrf_pycommon.process_utils.execute_process_split(cmd, cwd=None,
848 env=None, shell=False, emulate_tty=False)
849 execute_process(), except stderr is returned separately.
850 Instead of yielding output line by line until yielding a return
852 code, this function always a triplet of stdout, stderr, and re‐
853 turn code. Each time only one of the three will not be None.
854 Once you receive a non-None return code (type will be int) there
855 will be no more stdout or stderr. Therefore you can use the
856 command like this:
857 from __future__ import print_function
859 import sys
860 from osrf_pycommon.process_utils import execute_process_split
861 cmd = ['time', 'ls', '-G']
863 for out, err, ret in execute_process_split(cmd, cwd='/usr'):
864 # In Python 3, it will be a bytes array which needs to be decoded
865 out = out.decode('utf-8') if out is not None else None
866 err = err.decode('utf-8') if err is not None else None
867 if ret is not None:
868 # This is a return code, the command has exited
869 print("'{0}' exited with: {1}".format(' '.join(cmd), ret))
870 break
871 if out is not None:
872 print(out, end='')
873 if err is not None:
874 print(err, end='', file=sys.stderr)
875 When using this, it is possible that the stdout and stderr data
877 can be returned in a different order than what would happen on
878 the terminal. This is due to the fact that the subprocess is
879 given different buffers for stdout and stderr and so there is a
880 race condition on the subprocess writing to the different buf‐
881 fers and this command reading the buffers. This can be avoided
882 in most scenarios by using emulate_tty, because of the use of
883 pty’s, though the ordering can still not be guaranteed and the
884 number of pty’s is finite as explained in the documentation for
885 execute_process(). For situations where output ordering between
886 stdout and stderr are critical, they should not be returned sep‐
887 arately and instead should share one buffer, and so
888 execute_process() should be used.
889 For all other parameters and documentation see:
891 execute_process()
892 Availability: Unix (streaming), Windows (blocking)
894 Utility Functions
896 Currently there is only one utility function, a Python implementation
897 of the which shell command.
898 osrf_pycommon.process_utils.which(cmd, mode=1, path=None, **kwargs)
900 Given a command, mode, and a PATH string, return the path which
901 conforms to the given mode on the PATH, or None if there is no
902 such file.
903 mode defaults to os.F_OK | os.X_OK. path defaults to the result
905 of os.environ.get("PATH"), or can be overridden with a custom
906 search path.
907 Backported from shutil.which() (‐
909 https://docs.python.org/3.3/library/shutil.html#shutil.which),
910 available in Python 3.3.
911
913 This module provides tools for colorizing terminal output.
914 This module defines the ansi escape sequences used for colorizing the
916 output from terminal programs in Linux. You can access the ansi escape
917 sequences using the ansi() function:
918 >>> from osrf_pycommon.terminal_color import ansi
920 >>> print(["This is ", ansi('red'), "red", ansi('reset'), "."])
921 ['This is ', '\x1b[31m', 'red', '\x1b[0m', '.']
922 You can also use format_color() to do in-line substitution of keys
924 wrapped in @{} markers for their ansi escape sequences:
925 >>> from osrf_pycommon.terminal_color import format_color
927 >>> print(format_color("This is @{bf}blue@{reset}.").split())
928 ['This', 'is', '\x1b[34mblue\x1b[0m.']
929 This is a list of all of the available substitutions:
931 ┌──────────────┬─────────┬──────────┐
933 │Long Form │ Shorter │ Value │
934 ├──────────────┼─────────┼──────────┤
935 │@{blackf} │ @{kf} │ \033[30m │
936 ├──────────────┼─────────┼──────────┤
937 │@{redf} │ @{rf} │ \033[31m │
938 ├──────────────┼─────────┼──────────┤
939 │@{greenf} │ @{gf} │ \033[32m │
940 ├──────────────┼─────────┼──────────┤
941 │@{yellowf} │ @{yf} │ \033[33m │
942 ├──────────────┼─────────┼──────────┤
943 │@{bluef} │ @{bf} │ \033[34m │
944 ├──────────────┼─────────┼──────────┤
945 │@{purplef} │ @{pf} │ \033[35m │
946 ├──────────────┼─────────┼──────────┤
947 │@{cyanf} │ @{cf} │ \033[36m │
948 ├──────────────┼─────────┼──────────┤
949 │@{whitef} │ @{wf} │ \033[37m │
950 ├──────────────┼─────────┼──────────┤
951 │@{blackb} │ @{kb} │ \033[40m │
952 ├──────────────┼─────────┼──────────┤
953 │@{redb} │ @{rb} │ \033[41m │
954 ├──────────────┼─────────┼──────────┤
955 │@{greenb} │ @{gb} │ \033[42m │
956 ├──────────────┼─────────┼──────────┤
957 │@{yellowb} │ @{yb} │ \033[43m │
958 ├──────────────┼─────────┼──────────┤
959 │@{blueb} │ @{bb} │ \033[44m │
960 ├──────────────┼─────────┼──────────┤
961 │@{purpleb} │ @{pb} │ \033[45m │
962 ├──────────────┼─────────┼──────────┤
963 │@{cyanb} │ @{cb} │ \033[46m │
964 ├──────────────┼─────────┼──────────┤
965 │@{whiteb} │ @{wb} │ \033[47m │
966 ├──────────────┼─────────┼──────────┤
967 │@{escape} │ │ \033 │
968 └──────────────┴─────────┴──────────┘
969 │@{reset} │ @| │ \033[0m │
972 ├──────────────┼─────────┼──────────┤
973 │@{boldon} │ @! │ \033[1m │
974 ├──────────────┼─────────┼──────────┤
975 │@{italicson} │ @/ │ \033[3m │
976 ├──────────────┼─────────┼──────────┤
977 │@{ulon} │ @_ │ \033[4m │
978 ├──────────────┼─────────┼──────────┤
979 │@{invon} │ │ \033[7m │
980 ├──────────────┼─────────┼──────────┤
981 │@{boldoff} │ │ \033[22m │
982 ├──────────────┼─────────┼──────────┤
983 │@{italicsoff} │ │ \033[23m │
984 ├──────────────┼─────────┼──────────┤
985 │@{uloff} │ │ \033[24m │
986 ├──────────────┼─────────┼──────────┤
987 │@{invoff} │ │ \033[27m │
988 └──────────────┴─────────┴──────────┘
989 These substitution’s values come from the ANSI color escape sequences,
991 see: http://en.wikipedia.org/wiki/ANSI_escape_code
992 Also for any of the keys which have a trailing f, you can safely drop
994 the trailing f and get the same thing.
995 For example, format_color("@{redf}") and format_color("@{red}") are
997 functionally equivalent.
998 Also, many of the substitutions have shorten forms for convenience,
1000 such that @{redf}, @{rf}, @{red}, and @{r} are all the same.
1001 Note that a trailing b is always required when specifying a background.
1003 Some of the most common non-color sequences have {}’less versions.
1005 For example, @{boldon}’s shorter form is @!.
1007 By default, the substitutions (and calls to ansi()) resolve to escape
1009 sequences, but if you call disable_ansi_color_substitution_globally()
1010 then they will resolve to empty strings.
1011 This allows you to always use the substitution strings and disable them
1013 globally when desired.
1014 On Windows the substitutions are always resolved to empty strings as
1016 the ansi escape sequences do not work on Windows. Instead strings an‐
1017 notated with @{} style substitutions or raw \x1b[xxm style ansi escape
1018 sequences must be passed to print_color() in order for colors to be
1019 displayed on windows. Also the print_ansi_color_win32() function can
1020 be used on strings which only contain ansi escape sequences.
1021 NOTE:
1023 There are existing Python modules like colorama which provide ansi
1024 colorization on multiple platforms, so a valid question is: “why
1025 write this module?”. The reason for writing this module is to pro‐
1026 vide the color annotation of strings and functions for removing or
1027 replacing ansi escape sequences which are not provided by modules
1028 like colorama. This module could have depended on colorama for col‐
1029 orization on Windows, but colorama works by replacing the built-in
1030 sys.stdout and sys.stderr, which we did not want and it has extra
1031 functionality that we do not need. So, instead of depending on col‐
1032 orama, the Windows color printing code was used as the inspiration
1033 for the Windows color printing in the windows.py module in this ter‐
1034 minal_color package. The colorama license was placed in the header
1035 of that file and the colorama license is compatible with this pack‐
1036 age’s license.
1037 osrf_pycommon.terminal_color.ansi(key)
1039 Returns the escape sequence for a given ansi color key.
1040 osrf_pycommon.terminal_color.disable_ansi_color_substitution_globally()
1042 Causes format_color() to replace color annotations with empty
1043 strings.
1044 It also affects ansi().
1046 This is not the case by default, so if you want to make all sub‐
1048 stitutions given to either function mentioned above return empty
1049 strings then call this function.
1050 The default behavior can be restored by calling
1052 enable_ansi_color_substitution_globally().
1053 osrf_pycommon.terminal_color.enable_ansi_color_substitution_globally()
1055 Causes format_color() to replace color annotations with ansi
1056 esacpe sequences.
1057 It also affects ansi().
1059 This is the case by default, so there is no need to call this
1061 everytime.
1062 If you have previously caused all substitutions to evaluate to
1064 an empty string by calling
1065 disable_ansi_color_substitution_globally(), then you can restore
1066 the escape sequences for substitutions by calling this function.
1067 osrf_pycommon.terminal_color.format_color(msg)
1069 Replaces color annotations with ansi escape sequences.
1070 See this module’s documentation for the list of available sub‐
1072 stitutions.
1073 If disable_ansi_color_substitution_globally() has been called
1075 then all color annotations will be replaced by empty strings.
1076 Also, on Windows all color annotations will be replaced with
1078 empty strings. If you want colorization on Windows, you must
1079 pass annotated strings to print_color().
1080 Parameters
1082 msg (str) – string message to be colorized
1083 Returns
1085 colorized string
1086 Return type
1088 str
1089 osrf_pycommon.terminal_color.get_ansi_dict()
1091 Returns a copy of the dictionary of keys and ansi escape se‐
1092 quences.
1093 osrf_pycommon.terminal_color.print_ansi_color_win32(*args, **kwargs)
1095 Prints color string containing ansi escape sequences to console
1096 in Windows.
1097 If called on a non-Windows system, a NotImplementedError occurs.
1099 Does not respect disable_ansi_color_substitution_globally().
1101 Does not substitute color annotations like @{r} or @!, the
1103 string must already contain the \033[1m style ansi escape se‐
1104 quences.
1105 Works by splitting each argument up by ansi escape sequence,
1107 printing the text between the sequences, and doing the corre‐
1108 sponding win32 action for each ansi sequence encountered.
1109 osrf_pycommon.terminal_color.print_color(*args, **kwargs)
1111 Colorizes and prints with an implicit ansi reset at the end
1112 Calls format_color() on each positional argument and then sends
1114 all positional and keyword arguments to print.
1115 If the end keyword argument is not present then the default end
1117 value ansi('reset') + '\n' is used and passed to print.
1118 os.linesep is used to determine the actual value for \n.
1120 Therefore, if you use the end keyword argument be sure to in‐
1122 clude an ansi reset escape sequence if necessary.
1123 On Windows the substituted arguments and keyword arguments are
1125 passed to print_ansi_color_win32() instead of just print.
1126 osrf_pycommon.terminal_color.remove_ansi_escape_senquences(string)
1128 Removes any ansi escape sequences found in the given string and
1129 returns it.
1130 osrf_pycommon.terminal_color.remove_ansi_escape_sequences(string)
1132 Removes any ansi escape sequences found in the given string and
1133 returns it.
1134 osrf_pycommon.terminal_color.sanitize(msg)
1136 Sanitizes the given string to prevent format_color() from sub‐
1137 stituting content.
1138 For example, when the string 'Email: {user}@{org}' is passed to
1140 format_color() the @{org} will be incorrectly recognized as a
1141 colorization annotation and it will fail to substitute with a
1142 KeyError: org.
1143 In order to prevent this, you can first “sanitize” the string,
1145 add color annotations, and then pass the whole string to
1146 format_color().
1147 If you give this function the string 'Email: {user}@{org}', then
1149 it will return 'Email: {{user}}@@{{org}}'. Then if you pass that
1150 to format_color() it will return 'Email: {user}@{org}'. In this
1151 way format_color() is the reverse of this function and so it is
1152 safe to call this function on any incoming data if it will even‐
1153 tually be passed to format_color().
1154 In addition to expanding { => {{, } => }}, and @ => @@, this
1156 function will also replace any instances of @!, @/, @_, and @|
1157 with @{atexclamation}, @{atfwdslash}, @{atunderscore}, and @{at‐
1158 bar} respectively. And then there are corresponding keys in the
1159 ansi dict to convert them back.
1160 For example, if you pass the string '|@ Notice @|' to this func‐
1162 tion it will return '|@@ Notice @{atbar}'. And since ansi('at‐
1163 bar') always returns @|, even when
1164 disable_ansi_color_substitution_globally() has been called, the
1165 result of passing that string to format_color() will be '|@ No‐
1166 tice @|' again.
1167 There are two main strategies for constructing strings which use
1169 both the Python str.format() function and the colorization anno‐
1170 tations.
1171 One way is to just build each piece and concatenate the result:
1173 print_color("@{r}", "{error}".format(error=error_str))
1175 # Or using print (remember to include an ansi reset)
1176 print(format_color("@{r}" + "{error}".format(error=error_str) + "@|"))
1177 Another way is to use this function on the format string, con‐
1179 catenate to the annotations, pass the whole string to
1180 format_color(), and then format the whole thing:
1181 print(format_color("@{r}" + sanitize("{error}") + "@|")
1183 .format(error=error_str))
1184 However, the most common use for this function is to sanitize
1186 incoming strings which may have unknown content:
1187 def my_func(user_content):
1189 print_color("@{y}" + sanitize(user_content))
1190 This function is not intended to be used on strings with color
1192 annotations.
1193 Parameters
1195 msg (str) – string message to be sanitized
1196 Returns
1198 sanitized string
1199 Return type
1201 str
1202 osrf_pycommon.terminal_color.split_by_ansi_escape_sequence(string, in‐
1204 clude_delimiters=False)
1205 Splits a string into a list using any ansi escape sequence as a
1206 delimiter.
1207 Parameters
1209 • string (str) – string to be split
1211 • include_delimiters (bool) – If True include matched es‐
1213 cape sequences in the list (default: False)
1214 Returns
1216 list of strings, split from original string by escape se‐
1217 quences
1218 Return type
1220 list
1221 osrf_pycommon.terminal_color.test_colors(file=None)
1223 Prints a color testing block using print_color()
1224
1226 This module has a miscellaneous set of functions for working with ter‐
1227 minals.
1228 You can use the get_terminal_dimensions() to get the width and height
1230 of the terminal as a tuple.
1231 You can also use the is_tty() function to determine if a given object
1233 is a tty.
1234 exception osrf_pycommon.terminal_utils.GetTerminalDimensionsError
1236 Raised when the terminal dimensions cannot be determined.
1237 osrf_pycommon.terminal_utils.get_terminal_dimensions()
1239 Returns the width and height of the terminal.
1240 Returns
1242 width and height in that order as a tuple
1243 Return type
1245 tuple
1246 Raises GetTerminalDimensionsError when the terminal dimensions
1248 cannot be determined
1249 osrf_pycommon.terminal_utils.is_tty(stream)
1251 Returns True if the given stream is a tty, else False
1252 Parameters
1254 stream – object to be checked for being a tty
1255 Returns
1257 True if the given object is a tty, otherwise False
1258 Return type
1260 bool
1261
1263 Given that you have a copy of the source code, you can install osrf_py‐
1264 common like this:
1265 $ python setup.py install
1267 NOTE:
1269 If you are installing to a system Python you may need to use sudo.
1270 If you do not want to install osrf_pycommon into your system Python, or
1272 you don’t have access to sudo, then you can use a virtualenv.
1273
1275 Because osrf_pycommon uses setuptools you can (and should) use the
1276 develop feature:
1277 $ python setup.py develop
1279 NOTE:
1281 If you are developing against the system Python, you may need sudo.
1282 This will “install” osrf_pycommon to your Python path, but rather than
1284 copying the source files, it will instead place a marker file in the
1285 PYTHONPATH redirecting Python to your source directory. This allows
1286 you to use it as if it were installed but where changes to the source
1287 code take immediate affect.
1288 When you are done with develop mode you can (and should) undo it like
1290 this:
1291 $ python setup.py develop -u
1293 NOTE:
1295 If you are developing against the system Python, you may need sudo.
1296 That will “uninstall” the hooks into the PYTHONPATH which point to your
1298 source directory, but you should be wary that sometimes console scripts
1299 do not get removed from the bin folder.
1300
1302 In order to run the tests you will need to install flake8.
1303 Once you have installed those, then run unittest:
1305 $ python3 -m unittest discover -v tests
1307
1309 In order to build the docs you will need to first install Sphinx.
1310 You can build the documentation by invoking the Sphinx provided make
1312 target in the docs folder:
1313 $ # In the docs folder
1315 $ make html
1316 $ open _build/html/index.html
1317 Sometimes Sphinx does not pickup on changes to modules in packages
1319 which utilize the __all__ mechanism, so on repeat builds you may need
1320 to clean the docs first:
1321 $ # In the docs folder
1323 $ make clean
1324 $ make html
1325 $ open _build/html/index.html
1326
1328 William Woodall
1329
1331 2023, Open Source Robotics Foundation
13320.0 Aug 23, 2023 OSRF_PYCOMMON(1)