1OSRF_PYCOMMON(1)                 osrf_pycommon                OSRF_PYCOMMON(1)
2
3
4

NAME

6       osrf_pycommon - osrf_pycommon Documentation
7
8       osrf_pycommon  is  a python package which contains commonly used Python
9       boilerplate code and patterns.  Things  like  ansi  terminal  coloring,
10       capturing colored output from programs using subprocess, or even a sim‐
11       ple logging system which provides  some  nice  functionality  over  the
12       built-in Python logging system.
13
14       The  functionality  provided here should be generic enough to be reused
15       in arbitrary scenarios and should avoid bringing in dependencies  which
16       are  not  part of the standard Python library.  Where possible Windows,
17       Linux, and macOS should be supported, and where it cannot it should  be
18       gracefully degrading.  Code should be pure Python 3.
19
20       Contents:
21

THE CLI_UTILS MODULE

23       This  module  provides functions and patterns for creating Command Line
24       Interface (CLI) tools.
25
26   Common CLI Functions
27       Commonly used, CLI related functions.
28
29       osrf_pycommon.cli_utils.common.extract_argument_group(args,    delimit‐
30       ing_option)
31              Extract  a  group  of arguments from a list of arguments using a
32              delimiter.
33
34              Here is an example:
35
36                 >>> extract_argument_group(['foo', '--args', 'bar', '--baz'], '--args')
37                 (['foo'], ['bar', '--baz'])
38
39              The group can always be ended using the double  hyphen  --.   In
40              order  to  pass  a double hyphen as arguments, use three hyphens
41              ---.  Any set of hyphens encountered after the delimiter, and up
42              to  --,  which have three or more hyphens and are isolated, will
43              be captured and reduced by one hyphen.
44
45              For example:
46
47                 >> extract_argument_group(['foo',
48                                            '--args', 'bar', '--baz', '---', '--',
49                                            '--foo-option'], '--args')
50                 (['foo', '--foo-option'], ['bar', '--baz', '--'])
51
52              In the result the -- comes from  the  ---  in  the  input.   The
53              --args and the corresponding -- are removed entirely.
54
55              The delimiter and -- terminator combination can also happen mul‐
56              tiple times, in which case the bodies of arguments are  combined
57              and returned in the order they appeared.
58
59              For example:
60
61                 >> extract_argument_group(['foo',
62                                            '--args', 'ping', '--',
63                                            'bar',
64                                            '--args', 'pong', '--',
65                                            'baz',
66                                            '--args', '--'], '--args')
67                 (['foo', 'bar', 'baz'], ['ping', 'pong'])
68
69              Note: -- cannot be used as the delimiting_option.
70
71              Parameters
72
73args  (list)  – list of strings which are ordered argu‐
74                       ments.
75
76delimiting_option (str) – option which denotes where to
77                       split the args.
78
79              Returns
80                     tuple of arguments before and after the delimiter.
81
82              Return type
83                     tuple
84
85              Raises ValueError if the delimiting_option is --.
86
87       osrf_pycommon.cli_utils.common.extract_jobs_flags(arguments)
88              Extracts  make  job  flags from a list of other make flags, i.e.
89              -j8 -l8
90
91              The input arguments are given as a string  separated  by  white‐
92              space.   Make  job  flags are matched and removed from the argu‐
93              ments, and the Make job flags and what is left over from the in‐
94              put arguments are returned.
95
96              If  no  job  flags  are encountered, then an empty string is re‐
97              turned as the first element of the returned tuple.
98
99              Examples:
100
101                 >> extract_jobs_flags('-j8 -l8')
102                 ('-j8 -l8', '')
103                 >> extract_jobs_flags('-j8 ')
104                 ('-j8', ' ')
105                 >> extract_jobs_flags('target -j8 -l8 --some-option')
106                 ('-j8 -l8', 'target --some-option')
107                 >> extract_jobs_flags('target --some-option')
108                 ('', 'target --some-option')
109
110              Parameters
111                     arguments (str) – string  of  space  separated  arguments
112                     which may or may not contain make job flags
113
114              Returns
115                     tuple  of make jobs flags as a space separated string and
116                     leftover arguments as a space separated string
117
118              Return type
119                     tuple
120
121   The Verb Pattern
122       The verb pattern is a pattern where a single command aggregates  multi‐
123       ple  related commands by taking a required positional argument which is
124       the “verb” for the action you want to  perform.   For  example,  catkin
125       build  is  an  example  of a command and verb pair, where catkin is the
126       command and build is the verb.  In this  example,  the  catkin  command
127       groups  “actions” which are related to catkin together using verbs like
128       build which will build a workspace of catkin packages.
129
130   Command Boilerplate
131       This is an example boilerplate of a command which will use verbs:
132
133          from __future__ import print_function
134
135          import argparse
136          import sys
137
138          from osrf_pycommon.cli_utils.verb_pattern import create_subparsers
139          from osrf_pycommon.cli_utils.verb_pattern import list_verbs
140          from osrf_pycommon.cli_utils.verb_pattern import split_arguments_by_verb
141
142          COMMAND_NAME = '<INSERT COMMAND NAME HERE>'
143
144          VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
145
146
147          def main(sysargs=None):
148              # Assign sysargs if not set
149              sysargs = sys.argv[1:] if sysargs is None else sysargs
150
151              # Create a top level parser
152              parser = argparse.ArgumentParser(
153                  description="{0} command".format(COMMAND_NAME)
154              )
155
156              # Generate a list of verbs available
157              verbs = list_verbs(VERBS_ENTRY_POINT)
158
159              # Create the subparsers for each verb and collect the arg preprocessors
160              argument_preprocessors, verb_subparsers = create_subparsers(
161                  parser,
162                  COMMAND_NAME,
163                  verbs,
164                  VERBS_ENTRY_POINT,
165                  sysargs,
166              )
167
168              # Determine the verb, splitting arguments into pre and post verb
169              verb, pre_verb_args, post_verb_args = split_arguments_by_verb(sysargs)
170
171              # Short circuit -h and --help
172              if '-h' in pre_verb_args or '--help' in pre_verb_args:
173                  parser.print_help()
174                  sys.exit(0)
175
176              # Error on no verb provided
177              if verb is None:
178                  print(parser.format_usage())
179                  sys.exit("Error: No verb provided.")
180              # Error on unknown verb provided
181              if verb not in verbs:
182                  print(parser.format_usage())
183                  sys.exit("Error: Unknown verb '{0}' provided.".format(verb))
184
185              # Short circuit -h and --help for verbs
186              if '-h' in post_verb_args or '--help' in post_verb_args:
187                  verb_subparsers[verb].print_help()
188                  sys.exit(0)
189
190              # First allow the verb's argument preprocessor to strip any args
191              # and return any "extra" information it wants as a dict
192              processed_post_verb_args, extras = \
193                  argument_preprocessors[verb](post_verb_args)
194              # Then allow argparse to process the left over post-verb arguments along
195              # with the pre-verb arguments and the verb itself
196              args = parser.parse_args(pre_verb_args + [verb] + processed_post_verb_args)
197              # Extend the argparse result with the extras from the preprocessor
198              for key, value in extras.items():
199                  setattr(args, key, value)
200
201              # Finally call the subparser's main function with the processed args
202              # and the extras which the preprocessor may have returned
203              sys.exit(args.main(args) or 0)
204
205       This function is mostly boilerplate in that it will likely  not  change
206       much  between  commands  of  different types, but it would also be less
207       transparent to have this function created for you.  If  you  are  using
208       this  boilerplate to implement your command, then you should be careful
209       to update COMMAND_NAME to reflect your command’s name.
210
211       This line defines the entry_point group for your command’s verbs:
212
213          VERBS_ENTRY_POINT = '{0}.verbs'.format(COMMAND_NAME)
214
215       In the case that your command is called  foo  then  this  would  become
216       foo.verbs.   This  name  is  important because it is how verbs for this
217       command can be provided by your Python package or others.  For example,
218       each  verb  for your command foo will need entry in the setup.py of its
219       containing package, like this:
220
221          setup(
222              ...
223              entry_points={
224                  ...
225                  'foo.verbs': [
226                      'bar = foo.verbs.bar:entry_point_data',
227                  ],
228              }
229          )
230
231       You can see here that you are defining bar to be a  an  entry_point  of
232       type  foo.verbs  which  in  turn  points  to  a  module  and  reference
233       foo.verbs.bar and entry_point_data.  At run time this verb pattern will
234       let your command lookup all things defined as foo.verbs and load up the
235       reference to which they point.
236
237   Adding Verbs
238       In order to add a verb to your command, a few things must happen.
239
240       First you must have an entry in the setup.py as described above.   This
241       allows  the  command to find the entry_point for your verb at run time.
242       The entry_point for these verbs should point to a dictionary which  de‐
243       scribes the verb being added.
244
245       This is an example of an entry_point_data dictionary for a verb:
246
247          entry_point_data = dict(
248              verb='build',
249              description='Builds a workspace of packages',
250              # Called for execution, given parsed arguments object
251              main=main,
252              # Called first to setup argparse, given argparse parser
253              prepare_arguments=prepare_arguments,
254              # Called after prepare_arguments, but before argparse.parse_args
255              argument_preprocessor=argument_preprocessor,
256          )
257
258       As  you can see this dictionary describes the verb and gives references
259       to functions which allow the command to describe the  verb,  hook  into
260       argparse parameter creation for the verb, and to execute the verb.  The
261       verb, description, main, and prepare_arguments keys of  the  dictionary
262       are required, but the argument_preprocessor key is optional.
263
264verb:  This  is  the  name  of the verb, and is how the command knows
265         which verb implementation to match to a verb on the command line.
266
267description: This is used by the argument  parsing  to  describe  the
268         verb in --help.
269
270prepare_arguments:  This  function  gets  called to allow the verb to
271         setup it’s own argparse options. This function should always take one
272         parameter  which  is  the  argparse.ArgumentParser  for this verb, to
273         which arguments can be added. It can optionally take a second parame‐
274         ter  which are the current command line arguments. This is not always
275         needed, but can be useful in some cases. This function should  always
276         return the parser.
277
278argument_preprocessor:  This  function  is  optional, but allows your
279         verb an opportunity to process the  raw  arguments  before  they  are
280         passed  to  argparse’s  parse_args  function. This can be useful when
281         argparse is not capable of processing the options correctly.
282
283main: This is the implementation of the verb, it gets called last and
284         is  passed  the parsed arguments. The return type of this function is
285         used for sys.exit, a return type of None is interpreted as 0.
286
287       Here is an invented  example  of  main,  prepare_arguments,  and  argu‐
288       ment_preprocessor:
289
290          def prepare_arguments(parser):
291              parser.add_argument('--some-argument', action='store_true', default=False)
292              return parser
293
294          def argument_preprocessor(args):
295              extras = {}
296
297              if '-strange-argument' in args:
298                  args.remove('-strange-argument')
299                  extras['strange_argument'] = True
300
301              return args, extras
302
303          def main(options):
304              print('--some-argument:', options.some_argument)
305              print('-strange-argument:', options.strange_argument)
306              if options.strange_argument:
307                  return 1
308              return 0
309
310       The  above example is simply to illustrate the signature of these func‐
311       tions and how they might be used.
312
313   Verb Pattern API
314       API for implementing commands and verbs which used the verb pattern.
315
316       osrf_pycommon.cli_utils.verb_pattern.call_prepare_arguments(func,
317       parser, sysargs=None)
318              Call a prepare_arguments function with the correct number of pa‐
319              rameters.
320
321              The prepare_arguments function of a verb can either take one pa‐
322              rameter,  parser,  or two parameters parser and args, where args
323              are the current arguments being processed.
324
325              Parameters
326
327func (Callable) – Callable prepare_arguments function.
328
329parser (argparse.ArgumentParser) – parser which is  al‐
330                       ways passed to the function
331
332sysargs  (list)  –  arguments to optionally pass to the
333                       function, if needed
334
335              Returns
336                     return value of function or the parser  if  the  function
337                     returns None.
338
339              Return type
340                     argparse.ArgumentParser
341
342              Raises ValueError if a function with the wrong number of parame‐
343                     ters is given
344
345       osrf_pycommon.cli_utils.verb_pattern.create_subparsers(parser,
346       cmd_name, verbs, group, sysargs, title=None)
347              Creates  argparse  subparsers for each verb which can be discov‐
348              ered.
349
350              Using the verbs parameter,  the  available  verbs  are  iterated
351              through.   For each verb a subparser is created for it using the
352              parser parameter.  The cmd_name is used to fill  the  title  and
353              description  of the add_subparsers function call.  The group pa‐
354              rameter is used with each verb to load the  verb’s  description,
355              prepare_arguments  function,  and the verb’s argument_preproces‐
356              sors if available.  Each verb’s  prepare_arguments  function  is
357              called, allowing them to add arguments.  Finally a list of argu‐
358              ment_preprocessors functions and verb subparsers  are  returned,
359              one for each verb.
360
361              Parameters
362
363parser (argparse.ArgumentParser) – parser for this com‐
364                       mand
365
366cmd_name (str) – name of the command to which the verbs
367                       are being added
368
369verbs (list) – list of verbs (by name as a string)
370
371group  (str)  –  name  of the entry_point group for the
372                       verbs
373
374sysargs (list) – list of system arguments
375
376title (str) – optional custom title for the command
377
378              Returns
379                     tuple of argument_preprocessors and verb subparsers
380
381              Return type
382                     tuple
383
384       osrf_pycommon.cli_utils.verb_pattern.default_argument_preproces‐
385       sor(args)
386              Return unmodified args and an empty dict for extras
387
388       osrf_pycommon.cli_utils.verb_pattern.list_verbs(group)
389              List verbs available for a given entry_point group.
390
391              Parameters
392                     group  (str)  –  entry_point  group name for the verbs to
393                     list
394
395              Returns
396                     list of verb names for the given entry_point group
397
398              Return type
399                     list of str
400
401       osrf_pycommon.cli_utils.verb_pattern.load_verb_description(verb_name,
402       group)
403              Load description of a verb in a given group by name.
404
405              Parameters
406
407verb_name (str) – name of the verb to load, as a string
408
409group  (str) – entry_point group name which the verb is
410                       in
411
412              Returns
413                     verb description
414
415              Return type
416                     dict
417
418       osrf_pycommon.cli_utils.verb_pattern.split_arguments_by_verb(arguments)
419              Split arguments by verb.
420
421              Given a list of arguments (list of strings), the verb,  the  pre
422              verb arguments, and the post verb arguments are returned.
423
424              For example:
425
426                 >>> args = ['--command-arg1', 'verb', '--verb-arg1', '--verb-arg2']
427                 >>> split_arguments_by_verb(args)
428                 ('verb', ['--command-arg1'], ['--verb-arg1', '--verb-arg2'])
429
430              Parameters
431                     arguments (list) – list of system arguments
432
433              Returns
434                     the  verb (str), pre verb args (list), and post verb args
435                     (list)
436
437              Return type
438                     tuple
439

THE PROCESS_UTILS MODULE

441       This module provides functions for doing process management.
442
443       These are the main sections of this module:
444
445Asynchronous Process Utilities
446
447Synchronous Process Utilities
448
449Utility Functions
450
451   Asynchronous Process Utilities
452       There is a function and class which can be used together with your cus‐
453       tom asyncio run loop.
454
455       The  osrf_pycommon.process_utils.async_execute_process()  function is a
456       coroutine which allows you to run a process and get the output back bit
457       by  bit  in  real-time, either with stdout and stderr separated or com‐
458       bined.  This function also allows you to emulate the terminal  using  a
459       pty simply by toggling a flag in the parameters.
460
461       Along     side     this     coroutine     is    a    Protocol    class,
462       osrf_pycommon.process_utils.AsyncSubprocessProtocol, from which you can
463       inherit in order to customize how the yielded output is handled.
464
465       Because  this  coroutine is built on the asyncio framework’s subprocess
466       functions, it is portable and should behave the same on all major OS’s.
467       (including on Windows where an IOCP implementation is used)
468
469       async osrf_pycommon.process_utils.async_execute_process(protocol_class,
470       cmd=None,   cwd=None,   env=None,    shell=False,    emulate_tty=False,
471       stderr_to_stdout=True)
472              Coroutine  to  execute  a  subprocess  and yield the output back
473              asynchronously.
474
475              This function is meant to be used with the Python  asyncio  mod‐
476              ule, which is available in Python 3.5 or greater.
477
478              Here is an example of how to use this function:
479
480                 import asyncio
481                 from osrf_pycommon.process_utils import async_execute_process
482                 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
483                 from osrf_pycommon.process_utils import get_loop
484
485
486                 async def setup():
487                     transport, protocol = await async_execute_process(
488                         AsyncSubprocessProtocol, ['ls', '/usr'])
489                     returncode = await protocol.complete
490                     return returncode
491
492                 retcode = get_loop().run_until_complete(setup())
493                 get_loop().close()
494
495              Tthe  first argument is the default AsyncSubprocessProtocol pro‐
496              tocol class, which simply prints output from  stdout  to  stdout
497              and output from stderr to stderr.
498
499              If you want to capture and do something with the output or write
500              to  the  stdin,   then   you   need   to   subclass   from   the
501              AsyncSubprocessProtocol  class,  and  override the on_stdout_re‐
502              ceived, on_stderr_received, and on_process_exited functions.
503
504              See the documentation for the AsyncSubprocessProtocol class  for
505              more  details,  but  here  is an example which uses asyncio from
506              Python 3.5:
507
508                 import asyncio
509                 from osrf_pycommon.process_utils import async_execute_process
510                 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
511                 from osrf_pycommon.process_utils import get_loop
512
513
514                 class MyProtocol(AsyncSubprocessProtocol):
515                     def __init__(self, file_name, **kwargs):
516                         self.fh = open(file_name, 'w')
517                         AsyncSubprocessProtocol.__init__(self, **kwargs)
518
519                     def on_stdout_received(self, data):
520                         # Data has line endings intact, but is bytes in Python 3
521                         self.fh.write(data.decode('utf-8'))
522
523                     def on_stderr_received(self, data):
524                         self.fh.write(data.decode('utf-8'))
525
526                     def on_process_exited(self, returncode):
527                         self.fh.write("Exited with return code: {0}".format(returncode))
528                         self.fh.close()
529
530
531                 async def log_command_to_file(cmd, file_name):
532
533                     def create_protocol(**kwargs):
534                         return MyProtocol(file_name, **kwargs)
535
536                     transport, protocol = await async_execute_process(
537                         create_protocol, cmd)
538                     returncode = await protocol.complete
539                     return returncode
540
541                 get_loop().run_until_complete(
542                     log_command_to_file(['ls', '/'], '/tmp/out.txt'))
543                 get_loop().close()
544
545              See the subprocess.Popen class for more details on some  of  the
546              parameters to this function like cwd, env, and shell.
547
548              See  the  osrf_pycommon.process_utils.execute_process() function
549              for more details on the emulate_tty parameter.
550
551              Parameters
552
553protocol_class (AsyncSubprocessProtocol or a  subclass)
554                       – Protocol class which handles subprocess callbacks
555
556cmd  (list) – list of arguments where the executable is
557                       the first item
558
559cwd (str) – directory in which to run the command
560
561env (dict) – a dictionary of environment variable names
562                       to values
563
564shell (bool) – if True, the cmd variable is interpreted
565                       by a the shell
566
567emulate_tty (bool) – if True, pty’s are passed  to  the
568                       subprocess     for     stdout     and    stderr,    see
569                       osrf_pycommon.process_utils.execute_process().
570
571stderr_to_stdout (bool) – if True, stderr  is  directed
572                       to stdout, so they are not captured separately.
573
574       class   osrf_pycommon.process_utils.AsyncSubprocessProtocol(stdin=None,
575       stdout=None, stderr=None)
576              Protocol to subclass to get events from async_execute_process().
577
578              When subclassing this Protocol class, you should override  these
579              functions:
580
581                 def on_stdout_received(self, data):
582                     # ...
583
584                 def on_stderr_received(self, data):
585                     # ...
586
587                 def on_process_exited(self, returncode):
588                     # ...
589
590              By  default  these  functions  just print the data received from
591              stdout and stderr and does nothing when the process exits.
592
593              Data received by the on_stdout_received  and  on_stderr_received
594              functions is always in bytes.  Therefore, it may be necessary to
595              call .decode() on the data before printing to the screen.
596
597              Additionally, the data received will  not  be  stripped  of  new
598              lines, so take that into consideration when printing the result.
599
600              You can also override these less commonly used functions:
601
602                 def on_stdout_open(self):
603                     # ...
604
605                 def on_stdout_close(self, exc):
606                     # ...
607
608                 def on_stderr_open(self):
609                     # ...
610
611                 def on_stderr_close(self, exc):
612                     # ...
613
614              These  functions  are  called  when stdout/stderr are opened and
615              closed, and can be useful when using pty’s for example. The  exc
616              parameter  of  the *_close functions is None unless there was an
617              exception.
618
619              In addition to the overridable functions this class  has  a  few
620              useful public attributes.  The stdin attribute is a reference to
621              the PipeProto which follows  the  asyncio.WriteTransport  inter‐
622              face.   The  stdout  and  stderr attributes also reference their
623              PipeProto.  The complete attribute is a asyncio.Future which  is
624              set to complete when the process exits and its result is the re‐
625              turn code.
626
627              The complete attribute can be used like this:
628
629                 import asyncio
630                 from osrf_pycommon.process_utils import async_execute_process
631                 from osrf_pycommon.process_utils import AsyncSubprocessProtocol
632                 from osrf_pycommon.process_utils import get_loop
633
634
635                 async def setup():
636                     transport, protocol = await async_execute_process(
637                         AsyncSubprocessProtocol, ['ls', '-G', '/usr'])
638                     retcode = await protocol.complete
639                     print("Exited with", retcode)
640
641                 # This will block until the protocol.complete Future is done.
642                 get_loop().run_until_complete(setup())
643                 get_loop().close()
644
645              connection_made(transport)
646                     Called when a connection is made.
647
648                     The argument is the transport representing the pipe  con‐
649                     nection.   To  receive  data,  wait  for  data_received()
650                     calls.  When the connection is closed,  connection_lost()
651                     is called.
652
653              pipe_data_received(fd, data)
654                     Called when the subprocess writes data into stdout/stderr
655                     pipe.
656
657                     fd is int file descriptor.  data is bytes object.
658
659              process_exited()
660                     Called when subprocess has exited.
661
662       In addtion to these functions, there is a utility function for  getting
663       the correct asyncio event loop:
664
665       osrf_pycommon.process_utils.get_loop()
666              This  function will return the proper event loop for the subpro‐
667              cess async calls.
668
669              On Unix this just returns asyncio.get_event_loop(), but on  Win‐
670              dows it will set and return a asyncio.ProactorEventLoop instead.
671
672   Treatment of File Descriptors
673       Like  Python  3.4’s  subprocess.Popen  (and newer versions), all of the
674       process_utils     functions     do      not      close      inheritable
675       <https://docs.python.org/3.4/library/os.html#fd-inheritance>  file  de‐
676       scriptors before starting subprocesses.  This is equivalent to  passing
677       close_fds=False to subprocess.Popen on all Python versions.
678
679       For historical context, in Python 3.2, the subprocess.Popen default for
680       the close_fds option changed from False to True so that  file  descrip‐
681       tors opened by the parent process were closed before spawning the child
682       process.  In Python 3.4, PEP 0446 additionally made  it  so  even  when
683       close_fds=False  file  descriptors  which are non-inheritable are still
684       closed before spawning the subprocess.
685
686       If you want to be able to pass  file  descriptors  to  subprocesses  in
687       Python  3.4  or higher, you will need to make sure they are inheritable
688       <https://docs.python.org/3.4/library/os.html#fd-inheritance>.
689
690   Synchronous Process Utilities
691       For synchronous execution and output capture of subprocess,  there  are
692       two functions:
693
694osrf_pycommon.process_utils.execute_process()
695
696osrf_pycommon.process_utils.execute_process_split()
697
698       These  functions  are not yet using the asyncio framework as a back-end
699       and therefore on Windows will not stream the data from  the  subprocess
700       as  it  does  on Unix machines.  Instead data will not be yielded until
701       the subprocess is finished and all output is buffered (the normal warn‐
702       ings about long running programs with lots of output apply).
703
704       The streaming of output does not work on Windows because on Windows the
705       select.select() method only works on sockets and not file-like  objects
706       which  are used with subprocess pipes.  asyncio implements Windows sub‐
707       process support by implementing a Proactor event loop based on Window’s
708       IOCP  API.   One  future  option  will be to implement this synchronous
709       style method using IOCP in this module, but another option is  to  just
710       make synchronous the asynchronous calls, but there are issues with that
711       as well.  In the mean time, if you need streaming  of  output  in  both
712       Windows and Unix, use the asynchronous calls.
713
714       osrf_pycommon.process_utils.execute_process(cmd,   cwd=None,  env=None,
715       shell=False, emulate_tty=False)
716              Executes a command with arguments and  returns  output  line  by
717              line.
718
719              All  arguments,  except emulate_tty, are passed directly to sub‐
720              process.Popen.
721
722              execute_process returns a generator  which  yields  the  output,
723              line  by  line, until the subprocess finishes at which point the
724              return code is yielded.
725
726              This is an example of how this function should be used:
727
728                 from __future__ import print_function
729                 from osrf_pycommon.process_utils import execute_process
730
731                 cmd = ['ls', '-G']
732                 for line in execute_process(cmd, cwd='/usr'):
733                     if isinstance(line, int):
734                         # This is a return code, the command has exited
735                         print("'{0}' exited with: {1}".format(' '.join(cmd), line))
736                         continue  # break would also be appropriate here
737                     # In Python 3, it will be a bytes array which needs to be decoded
738                     if not isinstance(line, str):
739                         line = line.decode('utf-8')
740                     # Then print it to the screen
741                     print(line, end='')
742
743              stdout and stderr are always captured together and returned line
744              by line through the returned generator.  New line characters are
745              preserved in the output, so if re-printing the data take care to
746              use end='' or first rstrip the output lines.
747
748              When emulate_tty is used on Unix systems, commands will identify
749              that they are on a tty and should output color to the screen  as
750              if  you  were  running  it  on the terminal, and therefore there
751              should not be any need to pass arguments like -c color.ui=always
752              to  commands like git.  Additionally, programs might also behave
753              differently in when emulate_tty  is  being  used,  for  example,
754              Python will default to unbuffered output when it detects a tty.
755
756              emulate_tty  works  by  using psuedo-terminals on Unix machines,
757              and so if you are running this command many  times  in  parallel
758              (like hundreds of times) then you may get one of a few different
759              OSError’s.  For example, “OSError:  [Errno  24]  Too  many  open
760              files:  ‘/dev/ttyp0’”  or  “OSError:  out  of pty devices”.  You
761              should also be aware that you share pty devices with the rest of
762              the  system,  so even if you are not using a lot, it is possible
763              to get this error.  You can catch this error before getting data
764              from  the generator, so when using emulate_tty you might want to
765              do something like this:
766
767                 from __future__ import print_function
768                 from osrf_pycommon.process_utils import execute_process
769
770                 cmd = ['ls', '-G', '/usr']
771                 try:
772                     output = execute_process(cmd, emulate_tty=True)
773                 except OSError:
774                     output = execute_process(cmd, emulate_tty=False)
775                 for line in output:
776                     if isinstance(line, int):
777                         print("'{0}' exited with: {1}".format(' '.join(cmd), line))
778                         continue
779                     # In Python 3, it will be a bytes array which needs to be decoded
780                     if not isinstance(line, str):
781                         line = line.decode('utf-8')
782                     print(line, end='')
783
784              This way if a pty cannot be opened in order to emulate  the  tty
785              then  you can try again without emulation, and any other OSError
786              should raise again with emulate_tty set  to  False.   Obviously,
787              you only want to do this if emulating the tty is non-critical to
788              your processing, like when you are using it to capture color.
789
790              Any color information that the command outputs  as  ANSI  escape
791              sequences  is  captured by this command.  That way you can print
792              the output to the screen and preserve the color formatting.
793
794              If you do not want color to be in the output, then  try  setting
795              emulate_tty  to False, but that does not guarantee that there is
796              no color in the output, instead it only will cause  called  pro‐
797              cesses  to  identify  that they are not being run in a terminal.
798              Most well behaved programs will not output color if they  detect
799              that  they  are  not  being  executed  in  a  terminal,  but you
800              shouldn’t rely on that.
801
802              If you want to ensure there is no color in the  output  from  an
803              executed process, then use this function:
804
805              osrf_pycommon.terminal_color.remove_ansi_escape_sequences()
806
807              Exceptions  can be raised by functions called by the implementa‐
808              tion, for example, subprocess.Popen can raise  an  OSError  when
809              the  given  command  is not found.  If you want to check for the
810              existence of an executable on the path, see: which().   However,
811              this function itself does not raise any special exceptions.
812
813              Parameters
814
815cmd  (list) – list of strings with the first item being
816                       a command and subsequent items being any  arguments  to
817                       that command; passed directly to subprocess.Popen.
818
819cwd  (str) – path in which to run the command, defaults
820                       to None which means os.getcwd()  is  used;  passed  di‐
821                       rectly to subprocess.Popen.
822
823env  (dict) – environment dictionary to use for execut‐
824                       ing the command, default is None which uses the os.env‐
825                       iron environment; passed directly to subprocess.Popen.
826
827shell  (bool)  –  If  True  the system shell is used to
828                       evaluate the command, default is False; passed directly
829                       to subprocess.Popen.
830
831emulate_tty  (bool)  – If True attempts to use a pty to
832                       convince subprocess’s that they are being run in a ter‐
833                       minal. Typically this is useful for capturing colorized
834                       output from commands. This does not work on Windows (no
835                       pty’s),  so it is considered False even when True.  De‐
836                       faults to False.
837
838              Returns
839                     a generator which yields output from the command line  by
840                     line
841
842              Return type
843                     generator which yields strings
844
845       Availability: Unix (streaming), Windows (blocking)
846
847       osrf_pycommon.process_utils.execute_process_split(cmd,        cwd=None,
848       env=None, shell=False, emulate_tty=False)
849              execute_process(), except stderr is returned separately.
850
851              Instead of yielding output line by line until yielding a  return
852              code,  this function always a triplet of stdout, stderr, and re‐
853              turn code.  Each time only one of the three will  not  be  None.
854              Once you receive a non-None return code (type will be int) there
855              will be no more stdout or stderr.  Therefore  you  can  use  the
856              command like this:
857
858                 from __future__ import print_function
859                 import sys
860                 from osrf_pycommon.process_utils import execute_process_split
861
862                 cmd = ['time', 'ls', '-G']
863                 for out, err, ret in execute_process_split(cmd, cwd='/usr'):
864                     # In Python 3, it will be a bytes array which needs to be decoded
865                     out = out.decode('utf-8') if out is not None else None
866                     err = err.decode('utf-8') if err is not None else None
867                     if ret is not None:
868                         # This is a return code, the command has exited
869                         print("'{0}' exited with: {1}".format(' '.join(cmd), ret))
870                         break
871                     if out is not None:
872                         print(out, end='')
873                     if err is not None:
874                         print(err, end='', file=sys.stderr)
875
876              When  using this, it is possible that the stdout and stderr data
877              can be returned in a different order than what would  happen  on
878              the  terminal.   This  is due to the fact that the subprocess is
879              given different buffers for stdout and stderr and so there is  a
880              race  condition  on the subprocess writing to the different buf‐
881              fers and this command reading the buffers.  This can be  avoided
882              in  most  scenarios  by using emulate_tty, because of the use of
883              pty’s, though the ordering can still not be guaranteed  and  the
884              number  of pty’s is finite as explained in the documentation for
885              execute_process().  For situations where output ordering between
886              stdout and stderr are critical, they should not be returned sep‐
887              arately  and  instead  should   share   one   buffer,   and   so
888              execute_process() should be used.
889
890              For    all    other    parameters    and    documentation   see:
891              execute_process()
892
893       Availability: Unix (streaming), Windows (blocking)
894
895   Utility Functions
896       Currently there is only one utility function, a  Python  implementation
897       of the which shell command.
898
899       osrf_pycommon.process_utils.which(cmd, mode=1, path=None, **kwargs)
900              Given  a command, mode, and a PATH string, return the path which
901              conforms to the given mode on the PATH, or None if there  is  no
902              such file.
903
904              mode  defaults to os.F_OK | os.X_OK. path defaults to the result
905              of os.environ.get("PATH"), or can be overridden  with  a  custom
906              search path.
907
908              Backported            from           shutil.which()           (‐
909              https://docs.python.org/3.3/library/shutil.html#shutil.which),
910              available in Python 3.3.
911

THE TERMINAL_COLOR MODULE

913       This module provides tools for colorizing terminal output.
914
915       This  module  defines the ansi escape sequences used for colorizing the
916       output from terminal programs in Linux.  You can access the ansi escape
917       sequences using the ansi() function:
918
919          >>> from osrf_pycommon.terminal_color import ansi
920          >>> print(["This is ", ansi('red'), "red", ansi('reset'), "."])
921          ['This is ', '\x1b[31m', 'red', '\x1b[0m', '.']
922
923       You  can  also  use  format_color()  to do in-line substitution of keys
924       wrapped in @{} markers for their ansi escape sequences:
925
926          >>> from osrf_pycommon.terminal_color import format_color
927          >>> print(format_color("This is @{bf}blue@{reset}.").split())
928          ['This', 'is', '\x1b[34mblue\x1b[0m.']
929
930       This is a list of all of the available substitutions:
931
932                        ┌──────────────┬─────────┬──────────┐
933                        │Long Form     │ Shorter │ Value    │
934                        ├──────────────┼─────────┼──────────┤
935@{blackf}     @{kf}   \033[30m 
936                        ├──────────────┼─────────┼──────────┤
937@{redf}       @{rf}   \033[31m 
938                        ├──────────────┼─────────┼──────────┤
939@{greenf}     @{gf}   \033[32m 
940                        ├──────────────┼─────────┼──────────┤
941@{yellowf}    @{yf}   \033[33m 
942                        ├──────────────┼─────────┼──────────┤
943@{bluef}      @{bf}   \033[34m 
944                        ├──────────────┼─────────┼──────────┤
945@{purplef}    @{pf}   \033[35m 
946                        ├──────────────┼─────────┼──────────┤
947@{cyanf}      @{cf}   \033[36m 
948                        ├──────────────┼─────────┼──────────┤
949@{whitef}     @{wf}   \033[37m 
950                        ├──────────────┼─────────┼──────────┤
951@{blackb}     @{kb}   \033[40m 
952                        ├──────────────┼─────────┼──────────┤
953@{redb}       @{rb}   \033[41m 
954                        ├──────────────┼─────────┼──────────┤
955@{greenb}     @{gb}   \033[42m 
956                        ├──────────────┼─────────┼──────────┤
957@{yellowb}    @{yb}   \033[43m 
958                        ├──────────────┼─────────┼──────────┤
959@{blueb}      @{bb}   \033[44m 
960                        ├──────────────┼─────────┼──────────┤
961@{purpleb}    @{pb}   \033[45m 
962                        ├──────────────┼─────────┼──────────┤
963@{cyanb}      @{cb}   \033[46m 
964                        ├──────────────┼─────────┼──────────┤
965@{whiteb}     @{wb}   \033[47m 
966                        ├──────────────┼─────────┼──────────┤
967@{escape}     │         │ \033     
968                        └──────────────┴─────────┴──────────┘
969
970
971@{reset}      @|      \033[0m  
972                        ├──────────────┼─────────┼──────────┤
973@{boldon}     @!      \033[1m  
974                        ├──────────────┼─────────┼──────────┤
975@{italicson}  @/      \033[3m  
976                        ├──────────────┼─────────┼──────────┤
977@{ulon}       @_      \033[4m  
978                        ├──────────────┼─────────┼──────────┤
979@{invon}      │         │ \033[7m  
980                        ├──────────────┼─────────┼──────────┤
981@{boldoff}    │         │ \033[22m 
982                        ├──────────────┼─────────┼──────────┤
983@{italicsoff} │         │ \033[23m 
984                        ├──────────────┼─────────┼──────────┤
985@{uloff}      │         │ \033[24m 
986                        ├──────────────┼─────────┼──────────┤
987@{invoff}     │         │ \033[27m 
988                        └──────────────┴─────────┴──────────┘
989
990       These substitution’s values come from the ANSI color escape  sequences,
991       see: http://en.wikipedia.org/wiki/ANSI_escape_code
992
993       Also  for  any of the keys which have a trailing f, you can safely drop
994       the trailing f and get the same thing.
995
996       For example,  format_color("@{redf}")  and  format_color("@{red}")  are
997       functionally equivalent.
998
999       Also,  many  of  the  substitutions have shorten forms for convenience,
1000       such that @{redf}, @{rf}, @{red}, and @{r} are all the same.
1001
1002       Note that a trailing b is always required when specifying a background.
1003
1004       Some of the most common non-color sequences have {}’less versions.
1005
1006       For example, @{boldon}’s shorter form is @!.
1007
1008       By default, the substitutions (and calls to ansi()) resolve  to  escape
1009       sequences,  but  if you call disable_ansi_color_substitution_globally()
1010       then they will resolve to empty strings.
1011
1012       This allows you to always use the substitution strings and disable them
1013       globally when desired.
1014
1015       On  Windows  the  substitutions are always resolved to empty strings as
1016       the ansi escape sequences do not work on Windows.  Instead strings  an‐
1017       notated  with @{} style substitutions or raw \x1b[xxm style ansi escape
1018       sequences must be passed to print_color() in order  for  colors  to  be
1019       displayed  on  windows.  Also the print_ansi_color_win32() function can
1020       be used on strings which only contain ansi escape sequences.
1021
1022       NOTE:
1023          There are existing Python modules like colorama which  provide  ansi
1024          colorization  on  multiple  platforms,  so a valid question is: “why
1025          write this module?”.  The reason for writing this module is to  pro‐
1026          vide  the  color annotation of strings and functions for removing or
1027          replacing ansi escape sequences which are not  provided  by  modules
1028          like colorama.  This module could have depended on colorama for col‐
1029          orization on Windows, but colorama works by replacing  the  built-in
1030          sys.stdout  and  sys.stderr,  which we did not want and it has extra
1031          functionality that we do not need.  So, instead of depending on col‐
1032          orama,  the  Windows color printing code was used as the inspiration
1033          for the Windows color printing in the windows.py module in this ter‐
1034          minal_color  package.  The colorama license was placed in the header
1035          of that file and the colorama license is compatible with this  pack‐
1036          age’s license.
1037
1038       osrf_pycommon.terminal_color.ansi(key)
1039              Returns the escape sequence for a given ansi color key.
1040
1041       osrf_pycommon.terminal_color.disable_ansi_color_substitution_globally()
1042              Causes  format_color()  to  replace color annotations with empty
1043              strings.
1044
1045              It also affects ansi().
1046
1047              This is not the case by default, so if you want to make all sub‐
1048              stitutions given to either function mentioned above return empty
1049              strings then call this function.
1050
1051              The   default   behavior   can   be    restored    by    calling
1052              enable_ansi_color_substitution_globally().
1053
1054       osrf_pycommon.terminal_color.enable_ansi_color_substitution_globally()
1055              Causes  format_color()  to  replace  color annotations with ansi
1056              esacpe sequences.
1057
1058              It also affects ansi().
1059
1060              This is the case by default, so there is no need  to  call  this
1061              everytime.
1062
1063              If  you  have previously caused all substitutions to evaluate to
1064              an          empty          string           by           calling
1065              disable_ansi_color_substitution_globally(), then you can restore
1066              the escape sequences for substitutions by calling this function.
1067
1068       osrf_pycommon.terminal_color.format_color(msg)
1069              Replaces color annotations with ansi escape sequences.
1070
1071              See this module’s documentation for the list of  available  sub‐
1072              stitutions.
1073
1074              If  disable_ansi_color_substitution_globally()  has  been called
1075              then all color annotations will be replaced by empty strings.
1076
1077              Also, on Windows all color annotations  will  be  replaced  with
1078              empty  strings.   If  you want colorization on Windows, you must
1079              pass annotated strings to print_color().
1080
1081              Parameters
1082                     msg (str) – string message to be colorized
1083
1084              Returns
1085                     colorized string
1086
1087              Return type
1088                     str
1089
1090       osrf_pycommon.terminal_color.get_ansi_dict()
1091              Returns a copy of the dictionary of keys  and  ansi  escape  se‐
1092              quences.
1093
1094       osrf_pycommon.terminal_color.print_ansi_color_win32(*args, **kwargs)
1095              Prints  color string containing ansi escape sequences to console
1096              in Windows.
1097
1098              If called on a non-Windows system, a NotImplementedError occurs.
1099
1100              Does not respect disable_ansi_color_substitution_globally().
1101
1102              Does not substitute color  annotations  like  @{r}  or  @!,  the
1103              string  must  already  contain the \033[1m style ansi escape se‐
1104              quences.
1105
1106              Works by splitting each argument up  by  ansi  escape  sequence,
1107              printing  the  text  between the sequences, and doing the corre‐
1108              sponding win32 action for each ansi sequence encountered.
1109
1110       osrf_pycommon.terminal_color.print_color(*args, **kwargs)
1111              Colorizes and prints with an implicit ansi reset at the end
1112
1113              Calls format_color() on each positional argument and then  sends
1114              all positional and keyword arguments to print.
1115
1116              If  the end keyword argument is not present then the default end
1117              value ansi('reset') + '\n' is used and passed to print.
1118
1119              os.linesep is used to determine the actual value for \n.
1120
1121              Therefore, if you use the end keyword argument be  sure  to  in‐
1122              clude an ansi reset escape sequence if necessary.
1123
1124              On  Windows  the substituted arguments and keyword arguments are
1125              passed to print_ansi_color_win32() instead of just print.
1126
1127       osrf_pycommon.terminal_color.remove_ansi_escape_senquences(string)
1128              Removes any ansi escape sequences found in the given string  and
1129              returns it.
1130
1131       osrf_pycommon.terminal_color.remove_ansi_escape_sequences(string)
1132              Removes  any ansi escape sequences found in the given string and
1133              returns it.
1134
1135       osrf_pycommon.terminal_color.sanitize(msg)
1136              Sanitizes the given string to prevent format_color()  from  sub‐
1137              stituting content.
1138
1139              For  example, when the string 'Email: {user}@{org}' is passed to
1140              format_color() the @{org} will be incorrectly  recognized  as  a
1141              colorization  annotation  and  it will fail to substitute with a
1142              KeyError: org.
1143
1144              In order to prevent this, you can first “sanitize”  the  string,
1145              add  color  annotations,  and  then  pass  the  whole  string to
1146              format_color().
1147
1148              If you give this function the string 'Email: {user}@{org}', then
1149              it will return 'Email: {{user}}@@{{org}}'. Then if you pass that
1150              to format_color() it will return 'Email: {user}@{org}'.  In this
1151              way  format_color() is the reverse of this function and so it is
1152              safe to call this function on any incoming data if it will even‐
1153              tually be passed to format_color().
1154
1155              In  addition  to  expanding  { => {{, } => }}, and @ => @@, this
1156              function will also replace any instances of @!, @/, @_,  and  @|
1157              with @{atexclamation}, @{atfwdslash}, @{atunderscore}, and @{at‐
1158              bar} respectively.  And then there are corresponding keys in the
1159              ansi dict to convert them back.
1160
1161              For example, if you pass the string '|@ Notice @|' to this func‐
1162              tion it will return '|@@ Notice @{atbar}'.  And since  ansi('at‐
1163              bar')       always       returns       @|,       even       when
1164              disable_ansi_color_substitution_globally() has been called,  the
1165              result  of passing that string to format_color() will be '|@ No‐
1166              tice @|' again.
1167
1168              There are two main strategies for constructing strings which use
1169              both the Python str.format() function and the colorization anno‐
1170              tations.
1171
1172              One way is to just build each piece and concatenate the result:
1173
1174                 print_color("@{r}", "{error}".format(error=error_str))
1175                 # Or using print (remember to include an ansi reset)
1176                 print(format_color("@{r}" + "{error}".format(error=error_str) + "@|"))
1177
1178              Another way is to use this function on the format  string,  con‐
1179              catenate   to   the   annotations,  pass  the  whole  string  to
1180              format_color(), and then format the whole thing:
1181
1182                 print(format_color("@{r}" + sanitize("{error}") + "@|")
1183                       .format(error=error_str))
1184
1185              However, the most common use for this function  is  to  sanitize
1186              incoming strings which may have unknown content:
1187
1188                 def my_func(user_content):
1189                     print_color("@{y}" + sanitize(user_content))
1190
1191              This  function  is not intended to be used on strings with color
1192              annotations.
1193
1194              Parameters
1195                     msg (str) – string message to be sanitized
1196
1197              Returns
1198                     sanitized string
1199
1200              Return type
1201                     str
1202
1203       osrf_pycommon.terminal_color.split_by_ansi_escape_sequence(string,  in‐
1204       clude_delimiters=False)
1205              Splits  a string into a list using any ansi escape sequence as a
1206              delimiter.
1207
1208              Parameters
1209
1210string (str) – string to be split
1211
1212include_delimiters (bool) – If True include matched es‐
1213                       cape sequences in the list (default: False)
1214
1215              Returns
1216                     list of strings, split from original string by escape se‐
1217                     quences
1218
1219              Return type
1220                     list
1221
1222       osrf_pycommon.terminal_color.test_colors(file=None)
1223              Prints a color testing block using print_color()
1224

THE TERMINAL_UTILS MODULE

1226       This module has a miscellaneous set of functions for working with  ter‐
1227       minals.
1228
1229       You  can  use the get_terminal_dimensions() to get the width and height
1230       of the terminal as a tuple.
1231
1232       You can also use the is_tty() function to determine if a  given  object
1233       is a tty.
1234
1235       exception osrf_pycommon.terminal_utils.GetTerminalDimensionsError
1236              Raised when the terminal dimensions cannot be determined.
1237
1238       osrf_pycommon.terminal_utils.get_terminal_dimensions()
1239              Returns the width and height of the terminal.
1240
1241              Returns
1242                     width and height in that order as a tuple
1243
1244              Return type
1245                     tuple
1246
1247              Raises GetTerminalDimensionsError  when  the terminal dimensions
1248                     cannot be determined
1249
1250       osrf_pycommon.terminal_utils.is_tty(stream)
1251              Returns True if the given stream is a tty, else False
1252
1253              Parameters
1254                     stream – object to be checked for being a tty
1255
1256              Returns
1257                     True if the given object is a tty, otherwise False
1258
1259              Return type
1260                     bool
1261

INSTALLING FROM SOURCE

1263       Given that you have a copy of the source code, you can install osrf_py‐
1264       common like this:
1265
1266          $ python setup.py install
1267
1268       NOTE:
1269          If you are installing to a system Python you may need to use sudo.
1270
1271       If you do not want to install osrf_pycommon into your system Python, or
1272       you don’t have access to sudo, then you can use a virtualenv.
1273

HACKING

1275       Because osrf_pycommon uses setuptools you  can  (and  should)  use  the
1276       develop feature:
1277
1278          $ python setup.py develop
1279
1280       NOTE:
1281          If you are developing against the system Python, you may need sudo.
1282
1283       This  will “install” osrf_pycommon to your Python path, but rather than
1284       copying the source files, it will instead place a marker  file  in  the
1285       PYTHONPATH  redirecting  Python  to your source directory.  This allows
1286       you to use it as if it were installed but where changes to  the  source
1287       code take immediate affect.
1288
1289       When  you  are done with develop mode you can (and should) undo it like
1290       this:
1291
1292          $ python setup.py develop -u
1293
1294       NOTE:
1295          If you are developing against the system Python, you may need sudo.
1296
1297       That will “uninstall” the hooks into the PYTHONPATH which point to your
1298       source directory, but you should be wary that sometimes console scripts
1299       do not get removed from the bin folder.
1300

TESTING

1302       In order to run the tests you will need to install flake8.
1303
1304       Once you have installed those, then run unittest:
1305
1306          $ python3 -m unittest discover -v tests
1307

BUILDING THE DOCUMENTATION

1309       In order to build the docs you will need to first install Sphinx.
1310
1311       You can build the documentation by invoking the  Sphinx  provided  make
1312       target in the docs folder:
1313
1314          $ # In the docs folder
1315          $ make html
1316          $ open _build/html/index.html
1317
1318       Sometimes  Sphinx  does  not  pickup  on changes to modules in packages
1319       which utilize the __all__ mechanism, so on repeat builds you  may  need
1320       to clean the docs first:
1321
1322          $ # In the docs folder
1323          $ make clean
1324          $ make html
1325          $ open _build/html/index.html
1326

AUTHOR

1328       William Woodall
1329
1331       2023, Open Source Robotics Foundation
1332
1333
1334
1335
13360.0                              Aug 23, 2023                 OSRF_PYCOMMON(1)
Impressum