1docstrip_util(n)           Literate programming tool          docstrip_util(n)
2
3
4
5______________________________________________________________________________
6

NAME

8       docstrip_util - Docstrip-related utilities
9

SYNOPSIS

11       package require Tcl  8.4
12
13       package require docstrip  ?1.2?
14
15       package require docstrip::util  ?1.3.1?
16
17       pkgProvide name version terminals
18
19       pkgIndex ?terminal ...?
20
21       fileoptions ?option value ...?
22
23       docstrip::util::index_from_catalogue dir pattern ?option value ...?
24
25       docstrip::util::modules_from_catalogue target source ?option value ...?
26
27       docstrip::util::classical_preamble  metaprefix  message  target ?source
28       terminals ...?
29
30       docstrip::util::classical_postamble metaprefix message  target  ?source
31       terminals ...?
32
33       docstrip::util::packages_provided text ?setup-script?
34
35       docstrip::util::ddt2man text
36
37       docstrip::util::guards subcmd text
38
39       docstrip::util::patch  source-var terminals fromtext diff ?option value
40       ...?
41
42       docstrip::util::thefile filename ?option value ...?
43
44       docstrip::util::import_unidiff diff-text ?warning-var?
45
46______________________________________________________________________________
47

DESCRIPTION

49       The docstrip::util package is meant for collecting various utility pro‐
50       cedures  that are mainly useful at installation or development time. It
51       is separate from the base package to avoid overhead when the latter  is
52       used to source code.
53

PACKAGE INDEXING COMMANDS

55       Like  raw  ".tcl"  files,  code  lines  in docstrip source files can be
56       searched  for  package  declarations  and  corresponding  indices  con‐
57       structed.  A complication is however that one cannot tell from the code
58       blocks themselves which will fit together to make  a  working  package;
59       normally  that  information  would  be  found in an accompanying ".ins"
60       file, but parsing one of those is not an  easy  task.   Therefore  doc‐
61       strip::util  introduces an alternative encoding of such information, in
62       the form of a declarative Tcl script: the catalogue (of the contents in
63       a source file).
64
65       The special commands which are available inside a catalogue are:
66
67       pkgProvide name version terminals
68              Declares  that the code for a package with name name and version
69              version is made up from those modules in the source  file  which
70              are  selected  by  the terminals list of guard expression termi‐
71              nals. This code should preferably not contain a package  provide
72              command  for the package, as one will be provided by the package
73              loading mechanisms.
74
75       pkgIndex ?terminal ...?
76              Declares that the code for a package is made up from those  mod‐
77              ules  in  the source file which are selected by the listed guard
78              expression terminals. The name and version of  this  package  is
79              determined  from  package  provide command(s) found in that code
80              (hence there must be such a command in there).
81
82       fileoptions ?option value ...?
83              Declares the fconfigure options that should  be  in  force  when
84              reading  the  source; this can usually be ignored for pure ASCII
85              files, but if the file needs to be interpreted according to some
86              other  -encoding  then  this  is  how to specify it. The command
87              should normally appear first  in  the  catalogue,  as  it  takes
88              effect only for commands following it.
89
90       Other  Tcl  commands are supported too — a catalogue is parsed by being
91       evaluated in a safe interpreter — but they are rarely needed. To  allow
92       for  future  extensions, unknown commands in the catalogue are silently
93       ignored.
94
95       To simplify distribution  of  catalogues  together  with  their  source
96       files,  the  catalogue  is stored in the source file itself as a module
97       selected by the terminal 'docstrip.tcl::catalogue'.  This supports both
98       the  style of collecting all catalogue lines in one place and the style
99       of putting each catalogue line in close proximity of the code  that  it
100       declares.
101
102       Putting  catalogue  entries  next  to the code they declare may look as
103       follows
104
105
106              %    First there's the catalogue entry
107              %    \begin{tcl}
108              %<docstrip.tcl::catalogue>pkgProvide foo::bar 1.0 {foobar load}
109              %    \end{tcl}
110              %    second a metacomment used to include a copyright message
111              %    \begin{macrocode}
112              %<*foobar>
113              %% This file is placed in the public domain.
114              %    \end{macrocode}
115              %    third the package implementation
116              %    \begin{tcl}
117              namespace eval foo::bar {
118                 # ... some clever piece of Tcl code elided ...
119              %    \end{tcl}
120              %    which at some point may have variant code to make use of a
121              %    |load|able extension
122              %    \begin{tcl}
123              %<*load>
124                 load [file rootname [info script]][info sharedlibextension]
125              %</load>
126              %<*!load>
127                 # ... even more clever scripted counterpart of the extension
128                 # also elided ...
129              %</!load>
130              }
131              %</foobar>
132              %    \end{tcl}
133              %    and that's it!
134
135       The corresponding set-up with pkgIndex would be
136
137
138              %    First there's the catalogue entry
139              %    \begin{tcl}
140              %<docstrip.tcl::catalogue>pkgIndex foobar load
141              %    \end{tcl}
142              %    second a metacomment used to include a copyright message
143              %    \begin{tcl}
144              %<*foobar>
145              %% This file is placed in the public domain.
146              %    \end{tcl}
147              %    third the package implementation
148              %    \begin{tcl}
149              package provide foo::bar 1.0
150              namespace eval foo::bar {
151                 # ... some clever piece of Tcl code elided ...
152              %    \end{tcl}
153              %    which at some point may have variant code to make use of a
154              %    |load|able extension
155              %    \begin{tcl}
156              %<*load>
157                 load [file rootname [info script]][info sharedlibextension]
158              %</load>
159              %<*!load>
160                 # ... even more clever scripted counterpart of the extension
161                 # also elided ...
162              %</!load>
163              }
164              %</foobar>
165              %    \end{tcl}
166              %    and that's it!
167
168
169       docstrip::util::index_from_catalogue dir pattern ?option value ...?
170              This command is a sibling of the standard  pkg_mkIndex  command,
171              in  that  it  adds  package entries to "pkgIndex.tcl" files. The
172              difference is that it indexes docstrip-style source files rather
173              than raw ".tcl" or loadable library files.  Only packages listed
174              in the catalogue of a file are considered.
175
176              The dir argument is the directory in which  to  look  for  files
177              (and  whose "pkgIndex.tcl" file should be amended).  The pattern
178              argument is a glob pattern of files  to  look  into;  a  typical
179              value  would  be  *.dtx  or *.{dtx,ddt}. Remaining arguments are
180              option-value pairs, where the supported options are:
181
182              -recursein dirpattern
183                     If this option is given,  then  the  index_from_catalogue
184                     operation  will  be  repeated  in each subdirectory whose
185                     name matches the dirpattern. -recursein * will cause  the
186                     entire subtree rooted at dir to be indexed.
187
188              -sourceconf dictionary
189                     Specify fileoptions to use when reading the catalogues of
190                     files (and also for reading the packages if the catalogue
191                     does  not  contain  a  fileoptions  command). Defaults to
192                     being empty. Primarily useful if your system encoding  is
193                     very different from that of the source file (e.g., one is
194                     a two-byte encoding and the other is  a  one-byte  encod‐
195                     ing).  ascii  and  utf-8  are  not very different in that
196                     sense.
197
198              -options terminals
199                     The terminals is a list of terminals in addition to  doc‐
200                     strip.tcl::catalogue  that  should  be  held as true when
201                     extracting the catalogue. Defaults to being  empty.  This
202                     makes  it  possible  to make use of "variant sections" in
203                     the catalogue itself, e.g. gaurd  some  entries  with  an
204                     extra "experimental" and thus prevent them from appearing
205                     in the index unless that is generated with "experimental"
206                     among the -options.
207
208              -report boolean
209                     If  the  boolean  is true then the return value will be a
210                     textual, probably multiline, report  on  what  was  done.
211                     Defaults  to  false, in which case there is no particular
212                     return value.
213
214              -reportcmd commandPrefix
215                     Every item in the report is handed as an  extra  argument
216                     to  the  command prefix. Since index_from_catalogue would
217                     typically be used at a rather high level in  installation
218                     scripts and the like, the commandPrefix defaults to "puts
219                     stdout".  Use list to effectively disable  this  feature.
220                     The return values from the prefix are ignored.
221
222              The  package  ifneeded  scripts  that  are generated contain one
223              package require docstrip command  and  one  docstrip::sourcefrom
224              command.  If the catalogue entry was of the pkgProvide kind then
225              the package ifneeded script also contains  the  package  provide
226              command.
227
228              Note  that  index_from_catalogue  never removes anything from an
229              existing "pkgIndex.tcl" file. Hence you may need  to  delete  it
230              (or  have  pkg_mkIndex  recreate it from scratch) before running
231              index_from_catalogue to update some piece of  information,  such
232              as a package version number.
233
234
235       docstrip::util::modules_from_catalogue target source ?option value ...?
236              This  command  is  an  alternative to index_from_catalogue which
237              creates Tcl Module  (".tm")  files  rather  than  "pkgIndex.tcl"
238              entries.  Since  this  action  is  more similar to what docstrip
239              classically does, it has features for putting pre-  and  postam‐
240              bles on the generated files.
241
242              The  source  argument is the name of the source file to generate
243              ".tm" files from. The target argument  is  the  directory  which
244              should  count  as a module path, i.e., this is what the relative
245              paths derived from package names are joined  to.  The  supported
246              options are:
247
248              -preamble message
249                     A  message  to put in the preamble (initial block of com‐
250                     ments) of generated files. Defaults to a  space.  May  be
251                     several lines, which are then separated by newlines. Tra‐
252                     ditionally used for copyright notices or  the  like,  but
253                     metacomment lines provide an alternative to that.
254
255              -postamble message
256                     Like  -preamble, but the message is put at the end of the
257                     file instead of the beginning. Defaults to being empty.
258
259              -sourceconf dictionary
260                     Specify fileoptions to use when reading the catalogue  of
261                     the source (and also for reading the packages if the cat‐
262                     alogue does not contain a fileoptions command).  Defaults
263                     to  being empty. Primarily useful if your system encoding
264                     is very different from that of the source file (e.g., one
265                     is a two-byte encoding and the other is a one-byte encod‐
266                     ing). ascii and utf-8 are  not  very  different  in  that
267                     sense.
268
269              -options terminals
270                     The  terminals is a list of terminals in addition to doc‐
271                     strip.tcl::catalogue that should be  held  as  true  when
272                     extracting  the  catalogue. Defaults to being empty. This
273                     makes it possible to make use of  "variant  sections"  in
274                     the  catalogue  itself,  e.g.  gaurd some entries with an
275                     extra "experimental" guard and  thus  prevent  them  from
276                     contributing  packages  unless  those  are generated with
277                     "experimental" among the -options.
278
279              -formatpreamble commandPrefix
280                     Command prefix used  to  actually  format  the  preamble.
281                     Takes  four additional arguments message, targetFilename,
282                     sourceFilename, and terminalList and returns a fully for‐
283                     matted  preamble.  Defaults  to  using classical_preamble
284                     with a metaprefix of '##'.
285
286              -formatpostamble commandPrefix
287                     Command prefix used to  actually  format  the  postamble.
288                     Takes  four additional arguments message, targetFilename,
289                     sourceFilename, and terminalList and returns a fully for‐
290                     matted  postamble.  Defaults to using classical_postamble
291                     with a metaprefix of '##'.
292
293              -report boolean
294                     If the boolean is true (which is the  default)  then  the
295                     return  value  will  be  a  textual,  probably multiline,
296                     report on what was done. If it is false then there is  no
297                     particular return value.
298
299              -reportcmd commandPrefix
300                     Every  item  in the report is handed as an extra argument
301                     to this command prefix. Defaults to  list,  which  effec‐
302                     tively  disables this feature. The return values from the
303                     prefix are ignored. Use for example "puts stdout" to  get
304                     report items written immediately to the terminal.
305
306              An  existing  file of the same name as one to be created will be
307              overwritten.
308
309       docstrip::util::classical_preamble metaprefix  message  target  ?source
310       terminals ...?
311              This command returns a preamble in the classical docstrip style
312
313
314              ##
315              ## This is `TARGET',
316              ## generated by the docstrip::util package.
317              ##
318              ## The original source files were:
319              ##
320              ## SOURCE (with options: `foo,bar')
321              ##
322              ## Some message line 1
323              ## line2
324              ## line3
325
326
327              if called as
328
329
330              docstrip::util::classical_preamble {##}\
331                "\nSome message line 1\nline2\nline3" TARGET SOURCE {foo bar}
332
333
334              The command supports preambles for files generated from multiple
335              sources, even though modules_from_catalogue at present does  not
336              need that.
337
338       docstrip::util::classical_postamble  metaprefix  message target ?source
339       terminals ...?
340              This command returns a postamble in the classical docstrip style
341
342
343              ## Some message line 1
344              ## line2
345              ## line3
346              ##
347              ## End of file `TARGET'.
348
349
350              if called as
351
352
353              docstrip::util::classical_postamble {##}\
354                "Some message line 1\nline2\nline3" TARGET SOURCE {foo bar}
355
356
357              In other words, the source and terminals arguments are  ignored,
358              but supported for symmetry with classical_preamble.
359
360       docstrip::util::packages_provided text ?setup-script?
361              This  command  returns  a list where every even index element is
362              the name of a package provided by text when that is evaluated as
363              a  Tcl script, and the following odd index element is the corre‐
364              sponding version. It is used to do package indexing of extracted
365              pieces of code, in the manner of pkg_mkIndex.
366
367              One difference to pkg_mkIndex is that the text gets evaluated in
368              a  safe  interpreter.  package  require  commands  are  silently
369              ignored,  as  are  unknown  commands  (which includes source and
370              load). Other errors cause processing of the  text  to  stop,  in
371              which case only those package declarations that had been encoun‐
372              tered before the error will be included in the return value.
373
374              The setup-script argument can be used to customise  the  evalua‐
375              tion  environment,  if  the  code  in text has some very special
376              needs. The setup-script is evaluated in the local context of the
377              packages_provided  procedure  just before the text is processed.
378              At that time, the name of the slave command for the safe  inter‐
379              preter  that  will do this processing is kept in the local vari‐
380              able c. To for example copy the contents of the ::env  array  to
381              the safe interpreter, one might use a setup-script of
382
383                $c eval [list array set env [array get ::env]]
384

SOURCE PROCESSING COMMANDS

386       Unlike   the   previous   group  of  commands,  which  would  use  doc‐
387       strip::extract to extract some code lines and then process  those  fur‐
388       ther, the following commands operate on text consisting of all types of
389       lines.
390
391       docstrip::util::ddt2man text
392              The ddt2man command reformats text  from  the  general  docstrip
393              format  to  doctools ".man" format (Tcl Markup Language for Man‐
394              pages). The different line types are treated as follows:
395
396              comment and metacomment lines
397                     The '%' and '%%' prefixes are removed, the  rest  of  the
398                     text is kept as it is.
399
400              empty lines
401                     These  are kept as they are. (Effectively this means that
402                     they will count as comment lines after a comment line and
403                     as code lines after a code line.)
404
405              code lines
406                     example_begin  and example_end commands are placed at the
407                     beginning and end of  every  block  of  consecutive  code
408                     lines. Brackets in a code line are converted to lb and rb
409                     commands.
410
411              verbatim guards
412                     These are processed as usual, so they do not show  up  in
413                     the  result but every line in a verbatim block is treated
414                     as a code line.
415
416              other guards
417                     These are treated as code lines, except that  the  actual
418                     guard is emphasised.
419
420              At  the time of writing, no project has employed doctools markup
421              in master source files, so experience of what works well is  not
422              available. A source file could however look as follows
423
424
425              % [manpage_begin gcd n 1.0]
426              % [keywords divisor]
427              % [keywords math]
428              % [moddesc {Greatest Common Divisor}]
429              % [require gcd [opt 1.0]]
430              % [description]
431              %
432              % [list_begin definitions]
433              % [call [cmd gcd] [arg a] [arg b]]
434              %   The [cmd gcd] procedure takes two arguments [arg a] and [arg b] which
435              %   must be integers and returns their greatest common divisor.
436              proc gcd {a b} {
437              %   The first step is to take the absolute values of the arguments.
438              %   This relieves us of having to worry about how signs will be treated
439              %   by the remainder operation.
440                 set a [expr {abs($a)}]
441                 set b [expr {abs($b)}]
442              %   The next line does all of Euclid's algorithm! We can make do
443              %   without a temporary variable, since $a is substituted before the
444              %   [lb]set a $b[rb] and thus continues to hold a reference to the
445              %   "old" value of [var a].
446                 while {$b>0} { set b [expr { $a % [set a $b] }] }
447              %   In Tcl 8.3 we might want to use [cmd set] instead of [cmd return]
448              %   to get the slight advantage of byte-compilation.
449              %<tcl83>  set a
450              %<!tcl83>   return $a
451              }
452              % [list_end]
453              %
454              % [manpage_end]
455
456
457              If  the  above  text is fed through docstrip::util::ddt2man then
458              the result will be a  syntactically  correct  doctools  manpage,
459              even though its purpose is a bit different.
460
461              It  is  suggested  that  master  source code files with doctools
462              markup are given the suffix ".ddt", hence the "ddt" in ddt2man.
463
464       docstrip::util::guards subcmd text
465              The guards command returns information (mostly of a  statistical
466              nature)  about  the  ordinary  docstrip guards that occur in the
467              text. The subcmd selects what is returned.
468
469              counts List the guard expression terminals with counts. The for‐
470                     mat  of  the  return value is a dictionary which maps the
471                     terminal name to the number of occurencies of it  in  the
472                     file.
473
474              exprcount
475                     List the guard expressions with counts. The format of the
476                     return value is a dictionary which maps the expression to
477                     the number of occurencies of it in the file.
478
479              exprerr
480                     List  the syntactically incorrect guard expressions (e.g.
481                     parentheses do not match, or a terminal is missing).  The
482                     return  value is a list, with the elements in no particu‐
483                     lar order.
484
485              expressions
486                     List the guard expressions. The return value is  a  list,
487                     with the elements in no particular order.
488
489              exprmods
490                     List  the guard expressions with modifiers. The format of
491                     the return value is a dictionary where each  index  is  a
492                     guard  expression  and  each  entry  is a string with one
493                     character for every guard line that has this  expression.
494                     The  characters  in  the  entry specify what modifier was
495                     used in that line: +, -, *, /, or (for guard without mod‐
496                     ifier:)  space.  This  is  the most primitive form of the
497                     information gathered by guards.
498
499              names  List the guard expression terminals. The return value  is
500                     a list, with the elements in no particular order.
501
502              rotten List  the  malformed  guard  lines (this does not include
503                     lines where only the expression  is  malformed,  though).
504                     The format of the return value is a dictionary which maps
505                     line numbers to their contents.
506
507       docstrip::util::patch source-var terminals fromtext diff ?option  value
508       ...?
509              This  command tries to apply a diff file (for example a contrib‐
510              uted patch) that was computed for a generated file to  the  doc‐
511              strip  source. This can be useful if someone has edited a gener‐
512              ated file, thus mistaking it for being the source.  This command
513              makes  no  presumptions which are specific for the case that the
514              generated file is a Tcl script.
515
516              patch requires that the source file to patch is kept as  a  list
517              of  lines  in  a  variable, and the name of that variable in the
518              calling context is what goes into the source-var argument.   The
519              terminals is the list of terminals used to extract the file that
520              has been patched. The diff is the actual diff  to  apply  (in  a
521              format  as  explained below) and the fromtext is the contents of
522              the file which served as "from"  when  the  diff  was  computed.
523              Options can be used to further control the process.
524
525              The process works by "lifting" the hunks in the diff from gener‐
526              ated to source file, and then applying them to the  elements  of
527              the  source-var. In order to do this lifting, it is necessary to
528              determine how lines in the fromtext correspond  to  elements  of
529              the  source-var,  and  that  is where the terminals come in; the
530              source is first extracted under the  given  terminals,  and  the
531              result  of  that is then matched against the fromtext. This pro‐
532              duces a map which translates line numbers stated in the diff  to
533              element  numbers  in source-var, which is what is needed to lift
534              the hunks.
535
536              The reason that both the terminals  and  the  fromtext  must  be
537              given  is  twofold. First, it is very difficult to keep track of
538              how many lines of preamble are supplied some other way  than  by
539              copying  lines from source files. Second, a generated file might
540              contain material from several source files. Both make it  impos‐
541              sible  to  predict what line number an extracted file would have
542              in the generated file, so instead the  algorithm  for  computing
543              the  line  number map looks for a block of lines in the fromtext
544              which matches what can be extracted from the source. This match‐
545              ing is affected by the following options:
546
547              -matching mode
548                     How  equal  must two lines be in order to match? The sup‐
549                     ported modes are:
550
551                     exact  Lines must  be  equal  as  strings.  This  is  the
552                            default.
553
554                     anyspace
555                            All  sequences  of  whitespace characters are con‐
556                            verted to single spaces before comparing.
557
558                     nonspace
559                            Only non-whitespace characters are considered when
560                            comparing.
561
562                     none   Any two lines are considered to be equal.
563
564              -metaprefix string
565                     The -metaprefix value to use when extracting. Defaults to
566                     "%%", but for Tcl code it is more likely that "#" or "##"
567                     had been used for the generated file.
568
569              -trimlines boolean
570                     The  -trimlines value to use when extracting. Defaults to
571                     true.
572
573              The return value is in the form of a  unified  diff,  containing
574              only  those  hunks which were not applied or were only partially
575              applied; a comment in the header of each  hunk  specifies  which
576              case  is  at  hand.  It is normally necessary to manually review
577              both the return value from patch and the patched text itself, as
578              this command cannot adjust comment lines to match new content.
579
580              An example use would look like
581
582
583              set sourceL [split [docstrip::util::thefile from.dtx] \n]
584              set terminals {foo bar baz}
585              set fromtext [docstrip::util::thefile from.tcl]
586              set difftext [exec diff --unified from.tcl to.tcl]
587              set leftover [docstrip::util::patch sourceL $terminals $fromtext\
588                [docstrip::util::import_unidiff $difftext] -metaprefix {#}]
589              set F [open to.dtx w]; puts $F [join $sourceL \n]; close $F
590              return $leftover
591
592
593              Here,  "from.dtx" was used as source for "from.tcl", which some‐
594              one modified into "to.tcl". We're trying to construct a "to.dtx"
595              which can be used as source for "to.tcl".
596
597       docstrip::util::thefile filename ?option value ...?
598              The  thefile  command  opens the file filename, reads it to end,
599              closes it, and returns the contents (dropping a final newline if
600              there  is one). The option-value pairs are passed on to fconfig‐
601              ure to configure the open file channel before anything  is  read
602              from it.
603
604       docstrip::util::import_unidiff diff-text ?warning-var?
605              This command parses a unified (diff flags -U and --unified) for‐
606              mat  diff  into  the  list-of-hunks  format  expected  by   doc‐
607              strip::util::patch.  The diff-text argument is the text to parse
608              and the warning-var is, if specified, the name  in  the  calling
609              context  of a variable to which any warnings about parsing prob‐
610              lems will be appended.
611
612              The return value is a list of hunks. Each hunk is a list of five
613              elements  "start1  end1  start2 end2 lines". start1 and end1 are
614              line numbers in the "from" file of the first  and  last  respec‐
615              tively lines of the hunk.  start2 and end2 are the corresponding
616              line numbers in the "to" file. Line  numbers  start  at  1.  The
617              lines is a list with two elements for each line in the hunk; the
618              first specifies the type of a line and the second is the  actual
619              line  contents. The type is - for lines only in the "from" file,
620              + for lines that are only in the "to" file, and 0 for lines that
621              are in both.
622

SEE ALSO

624       docstrip, doctools, doctools_fmt
625

KEYWORDS

627       \.ddt,  .dtx,  LaTeX,  Tcl module, catalogue, diff, docstrip, doctools,
628       documentation, literate programming, module, package  indexing,  patch,
629       source
630

CATEGORY

632       Documentation tools
633
635       Copyright (c) 2003–2010 Lars Hellström <Lars dot Hellstrom at residenset dot net>
636
637
638
639
640tcllib                               1.3.1                    docstrip_util(n)
Impressum