1OSRF_PYCOMMON(1) osrf_pycommon OSRF_PYCOMMON(1)
2
3
4
6 osrf_pycommon - osrf_pycommon Documentation
7
8 osrf_pycommon is a python package which contains commonly used Python
9 boilerplate code and patterns. Things like ansi terminal coloring,
10 capturing colored output from programs using subprocess, or even a sim‐
11 ple logging system which provides some nice functionality over the
12 built-in Python logging system.
13
14 The functionality provided here should be generic enough to be reused
15 in arbitrary scenarios and should avoid bringing in dependencies which
16 are not part of the standard Python library. Where possible Windows,
17 Linux, and macOS should be supported, and where it cannot it should be
18 gracefully degrading. Code should be pure Python 3.
19
20 Contents:
21
23 This module provides functions and patterns for creating Command Line
24 Interface (CLI) tools.
25
26 Common CLI Functions
27 Commonly used, CLI related functions.
28
29 osrf_pycommon.cli_utils.common.extract_argument_group(args, delimit‐
30 ing_option)
31 Extract a group of arguments from a list of arguments using a
32 delimiter.
33
34 Here is an example:
35
36 >>> extract_argument_group(['foo', '--args', 'bar', '--baz'], '--args')
37 (['foo'], ['bar', '--baz'])
38
39 The group can always be ended using the double hyphen --. In
40 order to pass a double hyphen as arguments, use three hyphens
41 ---. Any set of hyphens encountered after the delimiter, and up
42 to --, which have three or more hyphens and are isolated, will
43 be captured and reduced by one hyphen.
44
45 For example:
46
47 >> extract_argument_group(['foo',
48 '--args', 'bar', '--baz', '---', '--',
49 '--foo-option'], '--args')
50 (['foo', '--foo-option'], ['bar', '--baz', '--'])
51
52 In the result the -- comes from the --- in the input. The
53 --args and the corresponding -- are removed entirely.
54
55 The delimiter and -- terminator combination can also happen mul‐
56 tiple times, in which case the bodies of arguments are combined
57 and returned in the order they appeared.
58
59 For example:
60
61 >> extract_argument_group(['foo',
62 '--args', 'ping', '--',
63 'bar',
64 '--args', 'pong', '--',
65 'baz',
66 '--args', '--'], '--args')
67 (['foo', 'bar', 'baz'], ['ping', 'pong'])
68
69 Note: -- cannot be used as the delimiting_option.
70
71 Parameters
72
73 • args (list) -- list of strings which are ordered argu‐
74 ments.
75
76 • delimiting_option (str) -- option which denotes where
77 to split the args.
78
79 Returns
80 tuple of arguments before and after the delimiter.
81
82 Return type
83 tuple
84
85 Raises ValueError if the delimiting_option is --.
86
87 osrf_pycommon.cli_utils.common.extract_jobs_flags(arguments)
88 Extracts make job flags from a list of other make flags, i.e.
89 -j8 -l8
90
91 The input arguments are given as a string separated by white‐
92 space. Make job flags are matched and removed from the argu‐
93 ments, and the Make job flags and what is left over from the in‐
94 put arguments are returned.
95
96 If no job flags are encountered, then an empty string is re‐
97 turned as the first element of the returned tuple.
98
99 Examples:
100
101 >> extract_jobs_flags('-j8 -l8')
102 ('-j8 -l8', '')
103 >> extract_jobs_flags('-j8 ')
104 ('-j8', ' ')
105 >> extract_jobs_flags('target -j8 -l8 --some-option')
106 ('-j8 -l8', 'target --some-option')
107 >> extract_jobs_flags('target --some-option')
108 ('', 'target --some-option')
109
110 Parameters
111 arguments (str) -- string of space separated arguments
112 which may or may not contain make job flags
113
114 Returns
115 tuple of make jobs flags as a space separated string and
116 leftover arguments as a space separated string
117
118 Return type
119 tuple
120
121 The Verb Pattern
122 The verb pattern is a pattern where a single command aggregates multi‐
123 ple related commands by taking a required positional argument which is
124 the "verb" for the action you want to perform. For example, catkin
125 build is an example of a command and verb pair, where catkin is the
126 command and build is the verb. In this example, the catkin command
127 groups "actions" which are related to catkin together using verbs like
128 build which will build a workspace of catkin packages.
129
130 Command Boilerplate
131 This is an example boilerplate of a command which will use verbs:
132
133 from __future__ import print_function
134
135 import argparse
136 import sys
137
138 from osrf_pycommon.cli_utils.verb_pattern import create_subparsers
139 from osrf_pycommon.cli_utils.verb_pattern import list_verbs
140 from osrf_pycommon.cli_utils.verb_pattern import split_arguments_by_verb
141
142 COMMAND_NAME = '<INSERT COMMAND NAME HERE>'
143
144 VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
145
146
147 def main(sysargs=None):
148 # Assign sysargs if not set
149 sysargs = sys.argv[1:] if sysargs is None else sysargs
150
151 # Create a top level parser
152 parser = argparse.ArgumentParser(
153 description="{0} command".format(COMMAND_NAME)
154 )
155
156 # Generate a list of verbs available
157 verbs = list_verbs(VERBS_ENTRY_POINT)
158
159 # Create the subparsers for each verb and collect the arg preprocessors
160 argument_preprocessors, verb_subparsers = create_subparsers(
161 parser,
162 COMMAND_NAME,
163 verbs,
164 VERBS_ENTRY_POINT,
165 sysargs,
166 )
167
168 # Determine the verb, splitting arguments into pre and post verb
169 verb, pre_verb_args, post_verb_args = split_arguments_by_verb(sysargs)
170
171 # Short circuit -h and --help
172 if '-h' in pre_verb_args or '--help' in pre_verb_args:
173 parser.print_help()
174 sys.exit(0)
175
176 # Error on no verb provided
177 if verb is None:
178 print(parser.format_usage())
179 sys.exit("Error: No verb provided.")
180 # Error on unknown verb provided
181 if verb not in verbs:
182 print(parser.format_usage())
183 sys.exit("Error: Unknown verb '{0}' provided.".format(verb))
184
185 # Short circuit -h and --help for verbs
186 if '-h' in post_verb_args or '--help' in post_verb_args:
187 verb_subparsers[verb].print_help()
188 sys.exit(0)
189
190 # First allow the verb's argument preprocessor to strip any args
191 # and return any "extra" information it wants as a dict
192 processed_post_verb_args, extras = \
193 argument_preprocessors[verb](post_verb_args)
194 # Then allow argparse to process the left over post-verb arguments along
195 # with the pre-verb arguments and the verb itself
196 args = parser.parse_args(pre_verb_args + [verb] + processed_post_verb_args)
197 # Extend the argparse result with the extras from the preprocessor
198 for key, value in extras.items():
199 setattr(args, key, value)
200
201 # Finally call the subparser's main function with the processed args
202 # and the extras which the preprocessor may have returned
203 sys.exit(args.main(args) or 0)
204
205 This function is mostly boilerplate in that it will likely not change
206 much between commands of different types, but it would also be less
207 transparent to have this function created for you. If you are using
208 this boilerplate to implement your command, then you should be careful
209 to update COMMAND_NAME to reflect your command's name.
210
211 This line defines the entry_point group for your command's verbs:
212
213 VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
214
215 In the case that your command is called foo then this would become
216 foo.verbs. This name is important because it is how verbs for this
217 command can be provided by your Python package or others. For example,
218 each verb for your command foo will need entry in the setup.py of its
219 containing package, like this:
220
221 setup(
222 ...
223 entry_points={
224 ...
225 'foo.verbs': [
226 'bar = foo.verbs.bar:entry_point_data',
227 ],
228 }
229 )
230
231 You can see here that you are defining bar to be a an entry_point of
232 type foo.verbs which in turn points to a module and reference
233 foo.verbs.bar and entry_point_data. At run time this verb pattern will
234 let your command lookup all things defined as foo.verbs and load up the
235 reference to which they point.
236
237 Adding Verbs
238 In order to add a verb to your command, a few things must happen.
239
240 First you must have an entry in the setup.py as described above. This
241 allows the command to find the entry_point for your verb at run time.
242 The entry_point for these verbs should point to a dictionary which de‐
243 scribes the verb being added.
244
245 This is an example of an entry_point_data dictionary for a verb:
246
247 entry_point_data = dict(
248 verb='build',
249 description='Builds a workspace of packages',
250 # Called for execution, given parsed arguments object
251 main=main,
252 # Called first to setup argparse, given argparse parser
253 prepare_arguments=prepare_arguments,
254 # Called after prepare_arguments, but before argparse.parse_args
255 argument_preprocessor=argument_preprocessor,
256 )
257
258 As you can see this dictionary describes the verb and gives references
259 to functions which allow the command to describe the verb, hook into
260 argparse parameter creation for the verb, and to execute the verb. The
261 verb, description, main, and prepare_arguments keys of the dictionary
262 are required, but the argument_preprocessor key is optional.
263
264 • verb: This is the name of the verb, and is how the command knows
265 which verb implementation to match to a verb on the command line.
266
267 • description: This is used by the argument parsing to describe the
268 verb in --help.
269
270 • prepare_arguments: This function gets called to allow the verb to
271 setup it's own argparse options. This function should always take one
272 parameter which is the argparse.ArgumentParser for this verb, to
273 which arguments can be added. It can optionally take a second parame‐
274 ter which are the current command line arguments. This is not always
275 needed, but can be useful in some cases. This function should always
276 return the parser.
277
278 • argument_preprocessor: This function is optional, but allows your
279 verb an opportunity to process the raw arguments before they are
280 passed to argparse's parse_args function. This can be useful when
281 argparse is not capable of processing the options correctly.
282
283 • main: This is the implementation of the verb, it gets called last and
284 is passed the parsed arguments. The return type of this function is
285 used for sys.exit, a return type of None is interpreted as 0.
286
287 Here is an invented example of main, prepare_arguments, and argu‐
288 ment_preprocessor:
289
290 def prepare_arguments(parser):
291 parser.add_argument('--some-argument', action='store_true', default=False)
292 return parser
293
294 def argument_preprocessor(args):
295 extras = {}
296
297 if '-strange-argument' in args:
298 args.remove('-strange-argument')
299 extras['strange_argument'] = True
300
301 return args, extras
302
303 def main(options):
304 print('--some-argument:', options.some_argument)
305 print('-strange-argument:', options.strange_argument)
306 if options.strange_argument:
307 return 1
308 return 0
309
310 The above example is simply to illustrate the signature of these func‐
311 tions and how they might be used.
312
313 Verb Pattern API
314 API for implementing commands and verbs which used the verb pattern.
315
316 osrf_pycommon.cli_utils.verb_pattern.call_prepare_arguments(func,
317 parser, sysargs=None)
318 Call a prepare_arguments function with the correct number of pa‐
319 rameters.
320
321 The prepare_arguments function of a verb can either take one pa‐
322 rameter, parser, or two parameters parser and args, where args
323 are the current arguments being processed.
324
325 Parameters
326
327 • func (Callable) -- Callable prepare_arguments function.
328
329 • parser (argparse.ArgumentParser) -- parser which is al‐
330 ways passed to the function
331
332 • sysargs (list) -- arguments to optionally pass to the
333 function, if needed
334
335 Returns
336 return value of function or the parser if the function
337 returns None.
338
339 Return type
340 argparse.ArgumentParser
341
342 Raises ValueError if a function with the wrong number of parame‐
343 ters is given
344
345 osrf_pycommon.cli_utils.verb_pattern.create_subparsers(parser,
346 cmd_name, verbs, group, sysargs, title=None)
347 Creates argparse subparsers for each verb which can be discov‐
348 ered.
349
350 Using the verbs parameter, the available verbs are iterated
351 through. For each verb a subparser is created for it using the
352 parser parameter. The cmd_name is used to fill the title and
353 description of the add_subparsers function call. The group pa‐
354 rameter is used with each verb to load the verb's description,
355 prepare_arguments function, and the verb's argument_preproces‐
356 sors if available. Each verb's prepare_arguments function is
357 called, allowing them to add arguments. Finally a list of argu‐
358 ment_preprocessors functions and verb subparsers are returned,
359 one for each verb.
360
361 Parameters
362
363 • parser (argparse.ArgumentParser) -- parser for this
364 command
365
366 • cmd_name (str) -- name of the command to which the
367 verbs are being added
368
369 • verbs (list) -- list of verbs (by name as a string)
370
371 • group (str) -- name of the entry_point group for the
372 verbs
373
374 • sysargs (list) -- list of system arguments
375
376 • title (str) -- optional custom title for the command
377
378 Returns
379 tuple of argument_preprocessors and verb subparsers
380
381 Return type
382 tuple
383
384 osrf_pycommon.cli_utils.verb_pattern.default_argument_preproces‐
385 sor(args)
386 Return unmodified args and an empty dict for extras
387
388 osrf_pycommon.cli_utils.verb_pattern.list_verbs(group)
389 List verbs available for a given entry_point group.
390
391 Parameters
392 group (str) -- entry_point group name for the verbs to
393 list
394
395 Returns
396 list of verb names for the given entry_point group
397
398 Return type
399 list of str
400
401 osrf_pycommon.cli_utils.verb_pattern.load_verb_description(verb_name,
402 group)
403 Load description of a verb in a given group by name.
404
405 Parameters
406
407 • verb_name (str) -- name of the verb to load, as a
408 string
409
410 • group (str) -- entry_point group name which the verb is
411 in
412
413 Returns
414 verb description
415
416 Return type
417 dict
418
419 osrf_pycommon.cli_utils.verb_pattern.split_arguments_by_verb(arguments)
420 Split arguments by verb.
421
422 Given a list of arguments (list of strings), the verb, the pre
423 verb arguments, and the post verb arguments are returned.
424
425 For example:
426
427 >>> args = ['--command-arg1', 'verb', '--verb-arg1', '--verb-arg2']
428 >>> split_arguments_by_verb(args)
429 ('verb', ['--command-arg1'], ['--verb-arg1', '--verb-arg2'])
430
431 Parameters
432 arguments (list) -- list of system arguments
433
434 Returns
435 the verb (str), pre verb args (list), and post verb args
436 (list)
437
438 Return type
439 tuple
440
442 This module provides functions for doing process management.
443
444 These are the main sections of this module:
445
446 • Asynchronous Process Utilities
447
448 • Synchronous Process Utilities
449
450 • Utility Functions
451
452 Asynchronous Process Utilities
453 There is a function and class which can be used together with your cus‐
454 tom asyncio run loop.
455
456 The osrf_pycommon.process_utils.async_execute_process() function is a
457 coroutine which allows you to run a process and get the output back bit
458 by bit in real-time, either with stdout and stderr separated or com‐
459 bined. This function also allows you to emulate the terminal using a
460 pty simply by toggling a flag in the parameters.
461
462 Along side this coroutine is a Protocol class,
463 osrf_pycommon.process_utils.AsyncSubprocessProtocol, from which you can
464 inherit in order to customize how the yielded output is handled.
465
466 Because this coroutine is built on the asyncio framework’s subprocess
467 functions, it is portable and should behave the same on all major OS’s.
468 (including on Windows where an IOCP implementation is used)
469
470 async osrf_pycommon.process_utils.async_execute_process(protocol_class,
471 cmd=None, cwd=None, env=None, shell=False, emulate_tty=False,
472 stderr_to_stdout=True)
473 Coroutine to execute a subprocess and yield the output back
474 asynchronously.
475
476 This function is meant to be used with the Python asyncio mod‐
477 ule, which is available in Python 3.5 or greater.
478
479 Here is an example of how to use this function:
480
481 import asyncio
482 from osrf_pycommon.process_utils import async_execute_process
483 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
484 from osrf_pycommon.process_utils import get_loop
485
486
487 async def setup():
488 transport, protocol = await async_execute_process(
489 AsyncSubprocessProtocol, ['ls', '/usr'])
490 returncode = await protocol.complete
491 return returncode
492
493 retcode = get_loop().run_until_complete(setup())
494 get_loop().close()
495
496 Tthe first argument is the default AsyncSubprocessProtocol pro‐
497 tocol class, which simply prints output from stdout to stdout
498 and output from stderr to stderr.
499
500 If you want to capture and do something with the output or write
501 to the stdin, then you need to subclass from the
502 AsyncSubprocessProtocol class, and override the on_stdout_re‐
503 ceived, on_stderr_received, and on_process_exited functions.
504
505 See the documentation for the AsyncSubprocessProtocol class for
506 more details, but here is an example which uses asyncio from
507 Python 3.5:
508
509 import asyncio
510 from osrf_pycommon.process_utils import async_execute_process
511 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
512 from osrf_pycommon.process_utils import get_loop
513
514
515 class MyProtocol(AsyncSubprocessProtocol):
516 def __init__(self, file_name, **kwargs):
517 self.fh = open(file_name, 'w')
518 AsyncSubprocessProtocol.__init__(self, **kwargs)
519
520 def on_stdout_received(self, data):
521 # Data has line endings intact, but is bytes in Python 3
522 self.fh.write(data.decode('utf-8'))
523
524 def on_stderr_received(self, data):
525 self.fh.write(data.decode('utf-8'))
526
527 def on_process_exited(self, returncode):
528 self.fh.write("Exited with return code: {0}".format(returncode))
529 self.fh.close()
530
531
532 async def log_command_to_file(cmd, file_name):
533
534 def create_protocol(**kwargs):
535 return MyProtocol(file_name, **kwargs)
536
537 transport, protocol = await async_execute_process(
538 create_protocol, cmd)
539 returncode = await protocol.complete
540 return returncode
541
542 get_loop().run_until_complete(
543 log_command_to_file(['ls', '/'], '/tmp/out.txt'))
544 get_loop().close()
545
546 See the subprocess.Popen class for more details on some of the
547 parameters to this function like cwd, env, and shell.
548
549 See the osrf_pycommon.process_utils.execute_process() function
550 for more details on the emulate_tty parameter.
551
552 Parameters
553
554 • protocol_class (AsyncSubprocessProtocol or a subclass)
555 – Protocol class which handles subprocess callbacks
556
557 • cmd (list) – list of arguments where the executable is
558 the first item
559
560 • cwd (str) – directory in which to run the command
561
562 • env (dict) – a dictionary of environment variable names
563 to values
564
565 • shell (bool) – if True, the cmd variable is interpreted
566 by a the shell
567
568 • emulate_tty (bool) – if True, pty’s are passed to the
569 subprocess for stdout and stderr, see
570 osrf_pycommon.process_utils.execute_process().
571
572 • stderr_to_stdout (bool) – if True, stderr is directed
573 to stdout, so they are not captured separately.
574
575 class osrf_pycommon.process_utils.AsyncSubprocessProtocol(stdin=None,
576 stdout=None, stderr=None)
577 Protocol to subclass to get events from async_execute_process().
578
579 When subclassing this Protocol class, you should override these
580 functions:
581
582 def on_stdout_received(self, data):
583 # ...
584
585 def on_stderr_received(self, data):
586 # ...
587
588 def on_process_exited(self, returncode):
589 # ...
590
591 By default these functions just print the data received from
592 stdout and stderr and does nothing when the process exits.
593
594 Data received by the on_stdout_received and on_stderr_received
595 functions is always in bytes. Therefore, it may be necessary to
596 call .decode() on the data before printing to the screen.
597
598 Additionally, the data received will not be stripped of new
599 lines, so take that into consideration when printing the result.
600
601 You can also override these less commonly used functions:
602
603 def on_stdout_open(self):
604 # ...
605
606 def on_stdout_close(self, exc):
607 # ...
608
609 def on_stderr_open(self):
610 # ...
611
612 def on_stderr_close(self, exc):
613 # ...
614
615 These functions are called when stdout/stderr are opened and
616 closed, and can be useful when using pty’s for example. The exc
617 parameter of the *_close functions is None unless there was an
618 exception.
619
620 In addition to the overridable functions this class has a few
621 useful public attributes. The stdin attribute is a reference to
622 the PipeProto which follows the asyncio.WriteTransport inter‐
623 face. The stdout and stderr attributes also reference their
624 PipeProto. The complete attribute is a asyncio.Future which is
625 set to complete when the process exits and its result is the re‐
626 turn code.
627
628 The complete attribute can be used like this:
629
630 import asyncio
631 from osrf_pycommon.process_utils import async_execute_process
632 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
633 from osrf_pycommon.process_utils import get_loop
634
635
636 async def setup():
637 transport, protocol = await async_execute_process(
638 AsyncSubprocessProtocol, ['ls', '-G', '/usr'])
639 retcode = await protocol.complete
640 print("Exited with", retcode)
641
642 # This will block until the protocol.complete Future is done.
643 get_loop().run_until_complete(setup())
644 get_loop().close()
645
646 connection_made(transport)
647 Called when a connection is made.
648
649 The argument is the transport representing the pipe con‐
650 nection. To receive data, wait for data_received()
651 calls. When the connection is closed, connection_lost()
652 is called.
653
654 pipe_data_received(fd, data)
655 Called when the subprocess writes data into stdout/stderr
656 pipe.
657
658 fd is int file descriptor. data is bytes object.
659
660 process_exited()
661 Called when subprocess has exited.
662
663 In addtion to these functions, there is a utility function for getting
664 the correct asyncio event loop:
665
666 osrf_pycommon.process_utils.get_loop()
667 This function will return the proper event loop for the subpro‐
668 cess async calls.
669
670 On Unix this just returns asyncio.get_event_loop(), but on Win‐
671 dows it will set and return a asyncio.ProactorEventLoop instead.
672
673 Treatment of File Descriptors
674 Like Python 3.4’s subprocess.Popen (and newer versions), all of the
675 process_utils functions do not close inheritable
676 <https://docs.python.org/3.4/library/os.html#fd-inheritance> file de‐
677 scriptors before starting subprocesses. This is equivalent to passing
678 close_fds=False to subprocess.Popen on all Python versions.
679
680 For historical context, in Python 3.2, the subprocess.Popen default for
681 the close_fds option changed from False to True so that file descrip‐
682 tors opened by the parent process were closed before spawning the child
683 process. In Python 3.4, PEP 0446 additionally made it so even when
684 close_fds=False file descriptors which are non-inheritable are still
685 closed before spawning the subprocess.
686
687 If you want to be able to pass file descriptors to subprocesses in
688 Python 3.4 or higher, you will need to make sure they are inheritable
689 <https://docs.python.org/3.4/library/os.html#fd-inheritance>.
690
691 Synchronous Process Utilities
692 For synchronous execution and output capture of subprocess, there are
693 two functions:
694
695 • osrf_pycommon.process_utils.execute_process()
696
697 • osrf_pycommon.process_utils.execute_process_split()
698
699 These functions are not yet using the asyncio framework as a back-end
700 and therefore on Windows will not stream the data from the subprocess
701 as it does on Unix machines. Instead data will not be yielded until
702 the subprocess is finished and all output is buffered (the normal warn‐
703 ings about long running programs with lots of output apply).
704
705 The streaming of output does not work on Windows because on Windows the
706 select.select() method only works on sockets and not file-like objects
707 which are used with subprocess pipes. asyncio implements Windows sub‐
708 process support by implementing a Proactor event loop based on Window’s
709 IOCP API. One future option will be to implement this synchronous
710 style method using IOCP in this module, but another option is to just
711 make synchronous the asynchronous calls, but there are issues with that
712 as well. In the mean time, if you need streaming of output in both
713 Windows and Unix, use the asynchronous calls.
714
715 osrf_pycommon.process_utils.execute_process(cmd, cwd=None, env=None,
716 shell=False, emulate_tty=False)
717 Executes a command with arguments and returns output line by
718 line.
719
720 All arguments, except emulate_tty, are passed directly to sub‐
721 process.Popen.
722
723 execute_process returns a generator which yields the output,
724 line by line, until the subprocess finishes at which point the
725 return code is yielded.
726
727 This is an example of how this function should be used:
728
729 from __future__ import print_function
730 from osrf_pycommon.process_utils import execute_process
731
732 cmd = ['ls', '-G']
733 for line in execute_process(cmd, cwd='/usr'):
734 if isinstance(line, int):
735 # This is a return code, the command has exited
736 print("'{0}' exited with: {1}".format(' '.join(cmd), line))
737 continue # break would also be appropriate here
738 # In Python 3, it will be a bytes array which needs to be decoded
739 if not isinstance(line, str):
740 line = line.decode('utf-8')
741 # Then print it to the screen
742 print(line, end='')
743
744 stdout and stderr are always captured together and returned line
745 by line through the returned generator. New line characters are
746 preserved in the output, so if re-printing the data take care to
747 use end='' or first rstrip the output lines.
748
749 When emulate_tty is used on Unix systems, commands will identify
750 that they are on a tty and should output color to the screen as
751 if you were running it on the terminal, and therefore there
752 should not be any need to pass arguments like -c color.ui=always
753 to commands like git. Additionally, programs might also behave
754 differently in when emulate_tty is being used, for example,
755 Python will default to unbuffered output when it detects a tty.
756
757 emulate_tty works by using psuedo-terminals on Unix machines,
758 and so if you are running this command many times in parallel
759 (like hundreds of times) then you may get one of a few different
760 OSError’s. For example, “OSError: [Errno 24] Too many open
761 files: ‘/dev/ttyp0’” or “OSError: out of pty devices”. You
762 should also be aware that you share pty devices with the rest of
763 the system, so even if you are not using a lot, it is possible
764 to get this error. You can catch this error before getting data
765 from the generator, so when using emulate_tty you might want to
766 do something like this:
767
768 from __future__ import print_function
769 from osrf_pycommon.process_utils import execute_process
770
771 cmd = ['ls', '-G', '/usr']
772 try:
773 output = execute_process(cmd, emulate_tty=True)
774 except OSError:
775 output = execute_process(cmd, emulate_tty=False)
776 for line in output:
777 if isinstance(line, int):
778 print("'{0}' exited with: {1}".format(' '.join(cmd), line))
779 continue
780 # In Python 3, it will be a bytes array which needs to be decoded
781 if not isinstance(line, str):
782 line = line.decode('utf-8')
783 print(line, end='')
784
785 This way if a pty cannot be opened in order to emulate the tty
786 then you can try again without emulation, and any other OSError
787 should raise again with emulate_tty set to False. Obviously,
788 you only want to do this if emulating the tty is non-critical to
789 your processing, like when you are using it to capture color.
790
791 Any color information that the command outputs as ANSI escape
792 sequences is captured by this command. That way you can print
793 the output to the screen and preserve the color formatting.
794
795 If you do not want color to be in the output, then try setting
796 emulate_tty to False, but that does not guarantee that there is
797 no color in the output, instead it only will cause called pro‐
798 cesses to identify that they are not being run in a terminal.
799 Most well behaved programs will not output color if they detect
800 that they are not being executed in a terminal, but you
801 shouldn’t rely on that.
802
803 If you want to ensure there is no color in the output from an
804 executed process, then use this function:
805
806 osrf_pycommon.terminal_color.remove_ansi_escape_sequences()
807
808 Exceptions can be raised by functions called by the implementa‐
809 tion, for example, subprocess.Popen can raise an OSError when
810 the given command is not found. If you want to check for the
811 existence of an executable on the path, see: which(). However,
812 this function itself does not raise any special exceptions.
813
814 Parameters
815
816 • cmd (list) – list of strings with the first item being
817 a command and subsequent items being any arguments to
818 that command; passed directly to subprocess.Popen.
819
820 • cwd (str) – path in which to run the command, defaults
821 to None which means os.getcwd() is used; passed di‐
822 rectly to subprocess.Popen.
823
824 • env (dict) – environment dictionary to use for execut‐
825 ing the command, default is None which uses the os.env‐
826 iron environment; passed directly to subprocess.Popen.
827
828 • shell (bool) – If True the system shell is used to
829 evaluate the command, default is False; passed directly
830 to subprocess.Popen.
831
832 • emulate_tty (bool) – If True attempts to use a pty to
833 convince subprocess’s that they are being run in a ter‐
834 minal. Typically this is useful for capturing colorized
835 output from commands. This does not work on Windows (no
836 pty’s), so it is considered False even when True. De‐
837 faults to False.
838
839 Returns
840 a generator which yields output from the command line by
841 line
842
843 Return type
844 generator which yields strings
845
846 Availability: Unix (streaming), Windows (blocking)
847
848 osrf_pycommon.process_utils.execute_process_split(cmd, cwd=None,
849 env=None, shell=False, emulate_tty=False)
850 execute_process(), except stderr is returned separately.
851
852 Instead of yielding output line by line until yielding a return
853 code, this function always a triplet of stdout, stderr, and re‐
854 turn code. Each time only one of the three will not be None.
855 Once you receive a non-None return code (type will be int) there
856 will be no more stdout or stderr. Therefore you can use the
857 command like this:
858
859 from __future__ import print_function
860 import sys
861 from osrf_pycommon.process_utils import execute_process_split
862
863 cmd = ['time', 'ls', '-G']
864 for out, err, ret in execute_process_split(cmd, cwd='/usr'):
865 # In Python 3, it will be a bytes array which needs to be decoded
866 out = out.decode('utf-8') if out is not None else None
867 err = err.decode('utf-8') if err is not None else None
868 if ret is not None:
869 # This is a return code, the command has exited
870 print("'{0}' exited with: {1}".format(' '.join(cmd), ret))
871 break
872 if out is not None:
873 print(out, end='')
874 if err is not None:
875 print(err, end='', file=sys.stderr)
876
877 When using this, it is possible that the stdout and stderr data
878 can be returned in a different order than what would happen on
879 the terminal. This is due to the fact that the subprocess is
880 given different buffers for stdout and stderr and so there is a
881 race condition on the subprocess writing to the different buf‐
882 fers and this command reading the buffers. This can be avoided
883 in most scenarios by using emulate_tty, because of the use of
884 pty’s, though the ordering can still not be guaranteed and the
885 number of pty’s is finite as explained in the documentation for
886 execute_process(). For situations where output ordering between
887 stdout and stderr are critical, they should not be returned sep‐
888 arately and instead should share one buffer, and so
889 execute_process() should be used.
890
891 For all other parameters and documentation see:
892 execute_process()
893
894 Availability: Unix (streaming), Windows (blocking)
895
896 Utility Functions
897 Currently there is only one utility function, a Python implementation
898 of the which shell command.
899
900 osrf_pycommon.process_utils.which(cmd, mode=1, path=None, **kwargs)
901 Given a command, mode, and a PATH string, return the path which
902 conforms to the given mode on the PATH, or None if there is no
903 such file.
904
905 mode defaults to os.F_OK | os.X_OK. path defaults to the result
906 of os.environ.get("PATH"), or can be overridden with a custom
907 search path.
908
909 Backported from shutil.which() (‐
910 https://docs.python.org/3.3/library/shutil.html#shutil.which),
911 available in Python 3.3.
912
914 This module provides tools for colorizing terminal output.
915
916 This module defines the ansi escape sequences used for colorizing the
917 output from terminal programs in Linux. You can access the ansi escape
918 sequences using the ansi() function:
919
920 >>> from osrf_pycommon.terminal_color import ansi
921 >>> print(["This is ", ansi('red'), "red", ansi('reset'), "."])
922 ['This is ', '\x1b[31m', 'red', '\x1b[0m', '.']
923
924 You can also use format_color() to do in-line substitution of keys
925 wrapped in @{} markers for their ansi escape sequences:
926
927 >>> from osrf_pycommon.terminal_color import format_color
928 >>> print(format_color("This is @{bf}blue@{reset}.").split())
929 ['This', 'is', '\x1b[34mblue\x1b[0m.']
930
931 This is a list of all of the available substitutions:
932
933 ┌──────────────┬─────────┬──────────┐
934 │Long Form │ Shorter │ Value │
935 ├──────────────┼─────────┼──────────┤
936 │@{blackf} │ @{kf} │ \033[30m │
937 ├──────────────┼─────────┼──────────┤
938 │@{redf} │ @{rf} │ \033[31m │
939 ├──────────────┼─────────┼──────────┤
940 │@{greenf} │ @{gf} │ \033[32m │
941 ├──────────────┼─────────┼──────────┤
942 │@{yellowf} │ @{yf} │ \033[33m │
943 ├──────────────┼─────────┼──────────┤
944 │@{bluef} │ @{bf} │ \033[34m │
945 ├──────────────┼─────────┼──────────┤
946 │@{purplef} │ @{pf} │ \033[35m │
947 ├──────────────┼─────────┼──────────┤
948 │@{cyanf} │ @{cf} │ \033[36m │
949 ├──────────────┼─────────┼──────────┤
950 │@{whitef} │ @{wf} │ \033[37m │
951 ├──────────────┼─────────┼──────────┤
952 │@{blackb} │ @{kb} │ \033[40m │
953 ├──────────────┼─────────┼──────────┤
954 │@{redb} │ @{rb} │ \033[41m │
955 ├──────────────┼─────────┼──────────┤
956 │@{greenb} │ @{gb} │ \033[42m │
957 ├──────────────┼─────────┼──────────┤
958 │@{yellowb} │ @{yb} │ \033[43m │
959 ├──────────────┼─────────┼──────────┤
960 │@{blueb} │ @{bb} │ \033[44m │
961 └──────────────┴─────────┴──────────┘
962
963
964 │@{purpleb} │ @{pb} │ \033[45m │
965 ├──────────────┼─────────┼──────────┤
966 │@{cyanb} │ @{cb} │ \033[46m │
967 ├──────────────┼─────────┼──────────┤
968 │@{whiteb} │ @{wb} │ \033[47m │
969 ├──────────────┼─────────┼──────────┤
970 │@{escape} │ │ \033 │
971 ├──────────────┼─────────┼──────────┤
972 │@{reset} │ @| │ \033[0m │
973 ├──────────────┼─────────┼──────────┤
974 │@{boldon} │ @! │ \033[1m │
975 ├──────────────┼─────────┼──────────┤
976 │@{italicson} │ @/ │ \033[3m │
977 ├──────────────┼─────────┼──────────┤
978 │@{ulon} │ @_ │ \033[4m │
979 ├──────────────┼─────────┼──────────┤
980 │@{invon} │ │ \033[7m │
981 ├──────────────┼─────────┼──────────┤
982 │@{boldoff} │ │ \033[22m │
983 ├──────────────┼─────────┼──────────┤
984 │@{italicsoff} │ │ \033[23m │
985 ├──────────────┼─────────┼──────────┤
986 │@{uloff} │ │ \033[24m │
987 ├──────────────┼─────────┼──────────┤
988 │@{invoff} │ │ \033[27m │
989 └──────────────┴─────────┴──────────┘
990
991 These substitution’s values come from the ANSI color escape sequences,
992 see: http://en.wikipedia.org/wiki/ANSI_escape_code
993
994 Also for any of the keys which have a trailing f, you can safely drop
995 the trailing f and get the same thing.
996
997 For example, format_color("@{redf}") and format_color("@{red}") are
998 functionally equivalent.
999
1000 Also, many of the substitutions have shorten forms for convenience,
1001 such that @{redf}, @{rf}, @{red}, and @{r} are all the same.
1002
1003 Note that a trailing b is always required when specifying a background.
1004
1005 Some of the most common non-color sequences have {}’less versions.
1006
1007 For example, @{boldon}’s shorter form is @!.
1008
1009 By default, the substitutions (and calls to ansi()) resolve to escape
1010 sequences, but if you call disable_ansi_color_substitution_globally()
1011 then they will resolve to empty strings.
1012
1013 This allows you to always use the substitution strings and disable them
1014 globally when desired.
1015
1016 On Windows the substitutions are always resolved to empty strings as
1017 the ansi escape sequences do not work on Windows. Instead strings an‐
1018 notated with @{} style substitutions or raw \x1b[xxm style ansi escape
1019 sequences must be passed to print_color() in order for colors to be
1020 displayed on windows. Also the print_ansi_color_win32() function can
1021 be used on strings which only contain ansi escape sequences.
1022
1023 NOTE:
1024 There are existing Python modules like colorama which provide ansi
1025 colorization on multiple platforms, so a valid question is: “why
1026 write this module?”. The reason for writing this module is to pro‐
1027 vide the color annotation of strings and functions for removing or
1028 replacing ansi escape sequences which are not provided by modules
1029 like colorama. This module could have depended on colorama for col‐
1030 orization on Windows, but colorama works by replacing the built-in
1031 sys.stdout and sys.stderr, which we did not want and it has extra
1032 functionality that we do not need. So, instead of depending on col‐
1033 orama, the Windows color printing code was used as the inspiration
1034 for the Windows color printing in the windows.py module in this ter‐
1035 minal_color package. The colorama license was placed in the header
1036 of that file and the colorama license is compatible with this pack‐
1037 age’s license.
1038
1039 osrf_pycommon.terminal_color.ansi(key)
1040 Returns the escape sequence for a given ansi color key.
1041
1042 osrf_pycommon.terminal_color.disable_ansi_color_substitution_globally()
1043 Causes format_color() to replace color annotations with empty
1044 strings.
1045
1046 It also affects ansi().
1047
1048 This is not the case by default, so if you want to make all sub‐
1049 stitutions given to either function mentioned above return empty
1050 strings then call this function.
1051
1052 The default behavior can be restored by calling
1053 enable_ansi_color_substitution_globally().
1054
1055 osrf_pycommon.terminal_color.enable_ansi_color_substitution_globally()
1056 Causes format_color() to replace color annotations with ansi
1057 esacpe sequences.
1058
1059 It also affects ansi().
1060
1061 This is the case by default, so there is no need to call this
1062 everytime.
1063
1064 If you have previously caused all substitutions to evaluate to
1065 an empty string by calling
1066 disable_ansi_color_substitution_globally(), then you can restore
1067 the escape sequences for substitutions by calling this function.
1068
1069 osrf_pycommon.terminal_color.format_color(msg)
1070 Replaces color annotations with ansi escape sequences.
1071
1072 See this module’s documentation for the list of available sub‐
1073 stitutions.
1074
1075 If disable_ansi_color_substitution_globally() has been called
1076 then all color annotations will be replaced by empty strings.
1077
1078 Also, on Windows all color annotations will be replaced with
1079 empty strings. If you want colorization on Windows, you must
1080 pass annotated strings to print_color().
1081
1082 Parameters
1083 msg (str) – string message to be colorized
1084
1085 Returns
1086 colorized string
1087
1088 Return type
1089 str
1090
1091 osrf_pycommon.terminal_color.get_ansi_dict()
1092 Returns a copy of the dictionary of keys and ansi escape se‐
1093 quences.
1094
1095 osrf_pycommon.terminal_color.print_ansi_color_win32(*args, **kwargs)
1096 Prints color string containing ansi escape sequences to console
1097 in Windows.
1098
1099 If called on a non-Windows system, a NotImplementedError occurs.
1100
1101 Does not respect disable_ansi_color_substitution_globally().
1102
1103 Does not substitute color annotations like @{r} or @!, the
1104 string must already contain the \033[1m style ansi escape se‐
1105 quences.
1106
1107 Works by splitting each argument up by ansi escape sequence,
1108 printing the text between the sequences, and doing the corre‐
1109 sponding win32 action for each ansi sequence encountered.
1110
1111 osrf_pycommon.terminal_color.print_color(*args, **kwargs)
1112 Colorizes and prints with an implicit ansi reset at the end
1113
1114 Calls format_color() on each positional argument and then sends
1115 all positional and keyword arguments to print.
1116
1117 If the end keyword argument is not present then the default end
1118 value ansi('reset') + '\n' is used and passed to print.
1119
1120 os.linesep is used to determine the actual value for \n.
1121
1122 Therefore, if you use the end keyword argument be sure to in‐
1123 clude an ansi reset escape sequence if necessary.
1124
1125 On Windows the substituted arguments and keyword arguments are
1126 passed to print_ansi_color_win32() instead of just print.
1127
1128 osrf_pycommon.terminal_color.remove_ansi_escape_senquences(string)
1129 Removes any ansi escape sequences found in the given string and
1130 returns it.
1131
1132 osrf_pycommon.terminal_color.remove_ansi_escape_sequences(string)
1133 Removes any ansi escape sequences found in the given string and
1134 returns it.
1135
1136 osrf_pycommon.terminal_color.sanitize(msg)
1137 Sanitizes the given string to prevent format_color() from sub‐
1138 stituting content.
1139
1140 For example, when the string 'Email: {user}@{org}' is passed to
1141 format_color() the @{org} will be incorrectly recognized as a
1142 colorization annotation and it will fail to substitute with a
1143 KeyError: org.
1144
1145 In order to prevent this, you can first “sanitize” the string,
1146 add color annotations, and then pass the whole string to
1147 format_color().
1148
1149 If you give this function the string 'Email: {user}@{org}', then
1150 it will return 'Email: {{user}}@@{{org}}'. Then if you pass that
1151 to format_color() it will return 'Email: {user}@{org}'. In this
1152 way format_color() is the reverse of this function and so it is
1153 safe to call this function on any incoming data if it will even‐
1154 tually be passed to format_color().
1155
1156 In addition to expanding { => {{, } => }}, and @ => @@, this
1157 function will also replace any instances of @!, @/, @_, and @|
1158 with @{atexclamation}, @{atfwdslash}, @{atunderscore}, and @{at‐
1159 bar} respectively. And then there are corresponding keys in the
1160 ansi dict to convert them back.
1161
1162 For example, if you pass the string '|@ Notice @|' to this func‐
1163 tion it will return '|@@ Notice @{atbar}'. And since ansi('at‐
1164 bar') always returns @|, even when
1165 disable_ansi_color_substitution_globally() has been called, the
1166 result of passing that string to format_color() will be '|@ No‐
1167 tice @|' again.
1168
1169 There are two main strategies for constructing strings which use
1170 both the Python str.format() function and the colorization anno‐
1171 tations.
1172
1173 One way is to just build each piece and concatenate the result:
1174
1175 print_color("@{r}", "{error}".format(error=error_str))
1176 # Or using print (remember to include an ansi reset)
1177 print(format_color("@{r}" + "{error}".format(error=error_str) + "@|"))
1178
1179 Another way is to use this function on the format string, con‐
1180 catenate to the annotations, pass the whole string to
1181 format_color(), and then format the whole thing:
1182
1183 print(format_color("@{r}" + sanitize("{error}") + "@|")
1184 .format(error=error_str))
1185
1186 However, the most common use for this function is to sanitize
1187 incoming strings which may have unknown content:
1188
1189 def my_func(user_content):
1190 print_color("@{y}" + sanitize(user_content))
1191
1192 This function is not intended to be used on strings with color
1193 annotations.
1194
1195 Parameters
1196 msg (str) – string message to be sanitized
1197
1198 Returns
1199 sanitized string
1200
1201 Return type
1202 str
1203
1204 osrf_pycommon.terminal_color.split_by_ansi_escape_sequence(string, in‐
1205 clude_delimiters=False)
1206 Splits a string into a list using any ansi escape sequence as a
1207 delimiter.
1208
1209 Parameters
1210
1211 • string (str) – string to be split
1212
1213 • include_delimiters (bool) – If True include matched es‐
1214 cape sequences in the list (default: False)
1215
1216 Returns
1217 list of strings, split from original string by escape se‐
1218 quences
1219
1220 Return type
1221 list
1222
1223 osrf_pycommon.terminal_color.test_colors(file=None)
1224 Prints a color testing block using print_color()
1225
1227 This module has a miscellaneous set of functions for working with ter‐
1228 minals.
1229
1230 You can use the get_terminal_dimensions() to get the width and height
1231 of the terminal as a tuple.
1232
1233 You can also use the is_tty() function to determine if a given object
1234 is a tty.
1235
1236 exception osrf_pycommon.terminal_utils.GetTerminalDimensionsError
1237 Raised when the terminal dimensions cannot be determined.
1238
1239 osrf_pycommon.terminal_utils.get_terminal_dimensions()
1240 Returns the width and height of the terminal.
1241
1242 Returns
1243 width and height in that order as a tuple
1244
1245 Return type
1246 tuple
1247
1248 Raises GetTerminalDimensionsError when the terminal dimensions
1249 cannot be determined
1250
1251 osrf_pycommon.terminal_utils.is_tty(stream)
1252 Returns True if the given stream is a tty, else False
1253
1254 Parameters
1255 stream – object to be checked for being a tty
1256
1257 Returns
1258 True if the given object is a tty, otherwise False
1259
1260 Return type
1261 bool
1262
1264 Given that you have a copy of the source code, you can install osrf_py‐
1265 common like this:
1266
1267 $ python setup.py install
1268
1269 NOTE:
1270 If you are installing to a system Python you may need to use sudo.
1271
1272 If you do not want to install osrf_pycommon into your system Python, or
1273 you don’t have access to sudo, then you can use a virtualenv.
1274
1276 Because osrf_pycommon uses setuptools you can (and should) use the
1277 develop feature:
1278
1279 $ python setup.py develop
1280
1281 NOTE:
1282 If you are developing against the system Python, you may need sudo.
1283
1284 This will “install” osrf_pycommon to your Python path, but rather than
1285 copying the source files, it will instead place a marker file in the
1286 PYTHONPATH redirecting Python to your source directory. This allows
1287 you to use it as if it were installed but where changes to the source
1288 code take immediate affect.
1289
1290 When you are done with develop mode you can (and should) undo it like
1291 this:
1292
1293 $ python setup.py develop -u
1294
1295 NOTE:
1296 If you are developing against the system Python, you may need sudo.
1297
1298 That will “uninstall” the hooks into the PYTHONPATH which point to your
1299 source directory, but you should be wary that sometimes console scripts
1300 do not get removed from the bin folder.
1301
1303 In order to run the tests you will need to install flake8.
1304
1305 Once you have installed those, then run unittest:
1306
1307 $ python3 -m unittest discover -v tests
1308
1310 In order to build the docs you will need to first install Sphinx.
1311
1312 You can build the documentation by invoking the Sphinx provided make
1313 target in the docs folder:
1314
1315 $ # In the docs folder
1316 $ make html
1317 $ open _build/html/index.html
1318
1319 Sometimes Sphinx does not pickup on changes to modules in packages
1320 which utilize the __all__ mechanism, so on repeat builds you may need
1321 to clean the docs first:
1322
1323 $ # In the docs folder
1324 $ make clean
1325 $ make html
1326 $ open _build/html/index.html
1327
1329 William Woodall
1330
1332 2022, Open Source Robotics Foundation
1333
1334
1335
1336
13370.0 Oct 16, 2022 OSRF_PYCOMMON(1)