1docstrip_util(n) Literate programming tool docstrip_util(n)
2______________________________________________________________________________
6
8 docstrip_util - Docstrip-related utilities
9
11 package require Tcl 8.4
12 package require docstrip ?1.2?
14 package require docstrip::util ?1.3.1?
16 pkgProvide name version terminals
18 pkgIndex ?terminal ...?
20 fileoptions ?option value ...?
22 docstrip::util::index_from_catalogue dir pattern ?option value ...?
24 docstrip::util::modules_from_catalogue target source ?option value ...?
26 docstrip::util::classical_preamble metaprefix message target ?source
28 terminals ...?
29 docstrip::util::classical_postamble metaprefix message target ?source
31 terminals ...?
32 docstrip::util::packages_provided text ?setup-script?
34 docstrip::util::ddt2man text
36 docstrip::util::guards subcmd text
38 docstrip::util::patch source-var terminals fromtext diff ?option value
40 ...?
41 docstrip::util::thefile filename ?option value ...?
43 docstrip::util::import_unidiff diff-text ?warning-var?
45______________________________________________________________________________
47
49 The docstrip::util package is meant for collecting various utility pro‐
50 cedures that are mainly useful at installation or development time. It
51 is separate from the base package to avoid overhead when the latter is
52 used to source code.
53
55 Like raw ".tcl" files, code lines in docstrip source files can be
56 searched for package declarations and corresponding indices construc‐
57 ted. A complication is however that one cannot tell from the code
58 blocks themselves which will fit together to make a working package;
59 normally that information would be found in an accompanying ".ins"
60 file, but parsing one of those is not an easy task. Therefore doc‐
61 strip::util introduces an alternative encoding of such information, in
62 the form of a declarative Tcl script: the catalogue (of the contents in
63 a source file).
64 The special commands which are available inside a catalogue are:
66 pkgProvide name version terminals
68 Declares that the code for a package with name name and version
69 version is made up from those modules in the source file which
70 are selected by the terminals list of guard expression termi‐
71 nals. This code should preferably not contain a package provide
72 command for the package, as one will be provided by the package
73 loading mechanisms.
74 pkgIndex ?terminal ...?
76 Declares that the code for a package is made up from those mod‐
77 ules in the source file which are selected by the listed guard
78 expression terminals. The name and version of this package is
79 determined from package provide command(s) found in that code
80 (hence there must be such a command in there).
81 fileoptions ?option value ...?
83 Declares the fconfigure options that should be in force when
84 reading the source; this can usually be ignored for pure ASCII
85 files, but if the file needs to be interpreted according to some
86 other -encoding then this is how to specify it. The command
87 should normally appear first in the catalogue, as it takes ef‐
88 fect only for commands following it.
89 Other Tcl commands are supported too — a catalogue is parsed by being
91 evaluated in a safe interpreter — but they are rarely needed. To allow
92 for future extensions, unknown commands in the catalogue are silently
93 ignored.
94 To simplify distribution of catalogues together with their source
96 files, the catalogue is stored in the source file itself as a module
97 selected by the terminal 'docstrip.tcl::catalogue'. This supports both
98 the style of collecting all catalogue lines in one place and the style
99 of putting each catalogue line in close proximity of the code that it
100 declares.
101 Putting catalogue entries next to the code they declare may look as
103 follows
104 % First there's the catalogue entry
107 % \begin{tcl}
108 %<docstrip.tcl::catalogue>pkgProvide foo::bar 1.0 {foobar load}
109 % \end{tcl}
110 % second a metacomment used to include a copyright message
111 % \begin{macrocode}
112 %<*foobar>
113 %% This file is placed in the public domain.
114 % \end{macrocode}
115 % third the package implementation
116 % \begin{tcl}
117 namespace eval foo::bar {
118 # ... some clever piece of Tcl code elided ...
119 % \end{tcl}
120 % which at some point may have variant code to make use of a
121 % |load|able extension
122 % \begin{tcl}
123 %<*load>
124 load [file rootname [info script]][info sharedlibextension]
125 %</load>
126 %<*!load>
127 # ... even more clever scripted counterpart of the extension
128 # also elided ...
129 %</!load>
130 }
131 %</foobar>
132 % \end{tcl}
133 % and that's it!
134 The corresponding set-up with pkgIndex would be
136 % First there's the catalogue entry
139 % \begin{tcl}
140 %<docstrip.tcl::catalogue>pkgIndex foobar load
141 % \end{tcl}
142 % second a metacomment used to include a copyright message
143 % \begin{tcl}
144 %<*foobar>
145 %% This file is placed in the public domain.
146 % \end{tcl}
147 % third the package implementation
148 % \begin{tcl}
149 package provide foo::bar 1.0
150 namespace eval foo::bar {
151 # ... some clever piece of Tcl code elided ...
152 % \end{tcl}
153 % which at some point may have variant code to make use of a
154 % |load|able extension
155 % \begin{tcl}
156 %<*load>
157 load [file rootname [info script]][info sharedlibextension]
158 %</load>
159 %<*!load>
160 # ... even more clever scripted counterpart of the extension
161 # also elided ...
162 %</!load>
163 }
164 %</foobar>
165 % \end{tcl}
166 % and that's it!
167 docstrip::util::index_from_catalogue dir pattern ?option value ...?
170 This command is a sibling of the standard pkg_mkIndex command,
171 in that it adds package entries to "pkgIndex.tcl" files. The
172 difference is that it indexes docstrip-style source files rather
173 than raw ".tcl" or loadable library files. Only packages listed
174 in the catalogue of a file are considered.
175 The dir argument is the directory in which to look for files
177 (and whose "pkgIndex.tcl" file should be amended). The pattern
178 argument is a glob pattern of files to look into; a typical
179 value would be *.dtx or *.{dtx,ddt}. Remaining arguments are op‐
180 tion-value pairs, where the supported options are:
181 -recursein dirpattern
183 If this option is given, then the index_from_catalogue
184 operation will be repeated in each subdirectory whose
185 name matches the dirpattern. -recursein * will cause the
186 entire subtree rooted at dir to be indexed.
187 -sourceconf dictionary
189 Specify fileoptions to use when reading the catalogues of
190 files (and also for reading the packages if the catalogue
191 does not contain a fileoptions command). Defaults to be‐
192 ing empty. Primarily useful if your system encoding is
193 very different from that of the source file (e.g., one is
194 a two-byte encoding and the other is a one-byte encod‐
195 ing). ascii and utf-8 are not very different in that
196 sense.
197 -options terminals
199 The terminals is a list of terminals in addition to doc‐
200 strip.tcl::catalogue that should be held as true when ex‐
201 tracting the catalogue. Defaults to being empty. This
202 makes it possible to make use of "variant sections" in
203 the catalogue itself, e.g. gaurd some entries with an ex‐
204 tra "experimental" and thus prevent them from appearing
205 in the index unless that is generated with "experimental"
206 among the -options.
207 -report boolean
209 If the boolean is true then the return value will be a
210 textual, probably multiline, report on what was done. De‐
211 faults to false, in which case there is no particular re‐
212 turn value.
213 -reportcmd commandPrefix
215 Every item in the report is handed as an extra argument
216 to the command prefix. Since index_from_catalogue would
217 typically be used at a rather high level in installation
218 scripts and the like, the commandPrefix defaults to "puts
219 stdout". Use list to effectively disable this feature.
220 The return values from the prefix are ignored.
221 The package ifneeded scripts that are generated contain one
223 package require docstrip command and one docstrip::sourcefrom
224 command. If the catalogue entry was of the pkgProvide kind then
225 the package ifneeded script also contains the package provide
226 command.
227 Note that index_from_catalogue never removes anything from an
229 existing "pkgIndex.tcl" file. Hence you may need to delete it
230 (or have pkg_mkIndex recreate it from scratch) before running
231 index_from_catalogue to update some piece of information, such
232 as a package version number.
233 docstrip::util::modules_from_catalogue target source ?option value ...?
236 This command is an alternative to index_from_catalogue which
237 creates Tcl Module (".tm") files rather than "pkgIndex.tcl" en‐
238 tries. Since this action is more similar to what docstrip clas‐
239 sically does, it has features for putting pre- and postambles on
240 the generated files.
241 The source argument is the name of the source file to generate
243 ".tm" files from. The target argument is the directory which
244 should count as a module path, i.e., this is what the relative
245 paths derived from package names are joined to. The supported
246 options are:
247 -preamble message
249 A message to put in the preamble (initial block of com‐
250 ments) of generated files. Defaults to a space. May be
251 several lines, which are then separated by newlines. Tra‐
252 ditionally used for copyright notices or the like, but
253 metacomment lines provide an alternative to that.
254 -postamble message
256 Like -preamble, but the message is put at the end of the
257 file instead of the beginning. Defaults to being empty.
258 -sourceconf dictionary
260 Specify fileoptions to use when reading the catalogue of
261 the source (and also for reading the packages if the cat‐
262 alogue does not contain a fileoptions command). Defaults
263 to being empty. Primarily useful if your system encoding
264 is very different from that of the source file (e.g., one
265 is a two-byte encoding and the other is a one-byte encod‐
266 ing). ascii and utf-8 are not very different in that
267 sense.
268 -options terminals
270 The terminals is a list of terminals in addition to doc‐
271 strip.tcl::catalogue that should be held as true when ex‐
272 tracting the catalogue. Defaults to being empty. This
273 makes it possible to make use of "variant sections" in
274 the catalogue itself, e.g. gaurd some entries with an ex‐
275 tra "experimental" guard and thus prevent them from con‐
276 tributing packages unless those are generated with "ex‐
277 perimental" among the -options.
278 -formatpreamble commandPrefix
280 Command prefix used to actually format the preamble.
281 Takes four additional arguments message, targetFilename,
282 sourceFilename, and terminalList and returns a fully for‐
283 matted preamble. Defaults to using classical_preamble
284 with a metaprefix of '##'.
285 -formatpostamble commandPrefix
287 Command prefix used to actually format the postamble.
288 Takes four additional arguments message, targetFilename,
289 sourceFilename, and terminalList and returns a fully for‐
290 matted postamble. Defaults to using classical_postamble
291 with a metaprefix of '##'.
292 -report boolean
294 If the boolean is true (which is the default) then the
295 return value will be a textual, probably multiline, re‐
296 port on what was done. If it is false then there is no
297 particular return value.
298 -reportcmd commandPrefix
300 Every item in the report is handed as an extra argument
301 to this command prefix. Defaults to list, which effec‐
302 tively disables this feature. The return values from the
303 prefix are ignored. Use for example "puts stdout" to get
304 report items written immediately to the terminal.
305 An existing file of the same name as one to be created will be
307 overwritten.
308 docstrip::util::classical_preamble metaprefix message target ?source
310 terminals ...?
311 This command returns a preamble in the classical docstrip style
312 ##
315 ## This is `TARGET',
316 ## generated by the docstrip::util package.
317 ##
318 ## The original source files were:
319 ##
320 ## SOURCE (with options: `foo,bar')
321 ##
322 ## Some message line 1
323 ## line2
324 ## line3
325 if called as
328 docstrip::util::classical_preamble {##}\
331 "\nSome message line 1\nline2\nline3" TARGET SOURCE {foo bar}
332 The command supports preambles for files generated from multiple
335 sources, even though modules_from_catalogue at present does not
336 need that.
337 docstrip::util::classical_postamble metaprefix message target ?source
339 terminals ...?
340 This command returns a postamble in the classical docstrip style
341 ## Some message line 1
344 ## line2
345 ## line3
346 ##
347 ## End of file `TARGET'.
348 if called as
351 docstrip::util::classical_postamble {##}\
354 "Some message line 1\nline2\nline3" TARGET SOURCE {foo bar}
355 In other words, the source and terminals arguments are ignored,
358 but supported for symmetry with classical_preamble.
359 docstrip::util::packages_provided text ?setup-script?
361 This command returns a list where every even index element is
362 the name of a package provided by text when that is evaluated as
363 a Tcl script, and the following odd index element is the corre‐
364 sponding version. It is used to do package indexing of extracted
365 pieces of code, in the manner of pkg_mkIndex.
366 One difference to pkg_mkIndex is that the text gets evaluated in
368 a safe interpreter. package require commands are silently ig‐
369 nored, as are unknown commands (which includes source and load).
370 Other errors cause processing of the text to stop, in which case
371 only those package declarations that had been encountered before
372 the error will be included in the return value.
373 The setup-script argument can be used to customise the evalua‐
375 tion environment, if the code in text has some very special
376 needs. The setup-script is evaluated in the local context of the
377 packages_provided procedure just before the text is processed.
378 At that time, the name of the slave command for the safe inter‐
379 preter that will do this processing is kept in the local vari‐
380 able c. To for example copy the contents of the ::env array to
381 the safe interpreter, one might use a setup-script of
382 $c eval [list array set env [array get ::env]]
384
386 Unlike the previous group of commands, which would use docstrip::ex‐
387 tract to extract some code lines and then process those further, the
388 following commands operate on text consisting of all types of lines.
389 docstrip::util::ddt2man text
391 The ddt2man command reformats text from the general docstrip
392 format to doctools ".man" format (Tcl Markup Language for Man‐
393 pages). The different line types are treated as follows:
394 comment and metacomment lines
396 The '%' and '%%' prefixes are removed, the rest of the
397 text is kept as it is.
398 empty lines
400 These are kept as they are. (Effectively this means that
401 they will count as comment lines after a comment line and
402 as code lines after a code line.)
403 code lines
405 example_begin and example_end commands are placed at the
406 beginning and end of every block of consecutive code
407 lines. Brackets in a code line are converted to lb and rb
408 commands.
409 verbatim guards
411 These are processed as usual, so they do not show up in
412 the result but every line in a verbatim block is treated
413 as a code line.
414 other guards
416 These are treated as code lines, except that the actual
417 guard is emphasised.
418 At the time of writing, no project has employed doctools markup
420 in master source files, so experience of what works well is not
421 available. A source file could however look as follows
422 % [manpage_begin gcd n 1.0]
425 % [keywords divisor]
426 % [keywords math]
427 % [moddesc {Greatest Common Divisor}]
428 % [require gcd [opt 1.0]]
429 % [description]
430 %
431 % [list_begin definitions]
432 % [call [cmd gcd] [arg a] [arg b]]
433 % The [cmd gcd] procedure takes two arguments [arg a] and [arg b] which
434 % must be integers and returns their greatest common divisor.
435 proc gcd {a b} {
436 % The first step is to take the absolute values of the arguments.
437 % This relieves us of having to worry about how signs will be treated
438 % by the remainder operation.
439 set a [expr {abs($a)}]
440 set b [expr {abs($b)}]
441 % The next line does all of Euclid's algorithm! We can make do
442 % without a temporary variable, since $a is substituted before the
443 % [lb]set a $b[rb] and thus continues to hold a reference to the
444 % "old" value of [var a].
445 while {$b>0} { set b [expr { $a % [set a $b] }] }
446 % In Tcl 8.3 we might want to use [cmd set] instead of [cmd return]
447 % to get the slight advantage of byte-compilation.
448 %<tcl83> set a
449 %<!tcl83> return $a
450 }
451 % [list_end]
452 %
453 % [manpage_end]
454 If the above text is fed through docstrip::util::ddt2man then
457 the result will be a syntactically correct doctools manpage,
458 even though its purpose is a bit different.
459 It is suggested that master source code files with doctools
461 markup are given the suffix ".ddt", hence the "ddt" in ddt2man.
462 docstrip::util::guards subcmd text
464 The guards command returns information (mostly of a statistical
465 nature) about the ordinary docstrip guards that occur in the
466 text. The subcmd selects what is returned.
467 counts List the guard expression terminals with counts. The for‐
469 mat of the return value is a dictionary which maps the
470 terminal name to the number of occurencies of it in the
471 file.
472 exprcount
474 List the guard expressions with counts. The format of the
475 return value is a dictionary which maps the expression to
476 the number of occurencies of it in the file.
477 exprerr
479 List the syntactically incorrect guard expressions (e.g.
480 parentheses do not match, or a terminal is missing). The
481 return value is a list, with the elements in no particu‐
482 lar order.
483 expressions
485 List the guard expressions. The return value is a list,
486 with the elements in no particular order.
487 exprmods
489 List the guard expressions with modifiers. The format of
490 the return value is a dictionary where each index is a
491 guard expression and each entry is a string with one
492 character for every guard line that has this expression.
493 The characters in the entry specify what modifier was
494 used in that line: +, -, *, /, or (for guard without mod‐
495 ifier:) space. This is the most primitive form of the in‐
496 formation gathered by guards.
497 names List the guard expression terminals. The return value is
499 a list, with the elements in no particular order.
500 rotten List the malformed guard lines (this does not include
502 lines where only the expression is malformed, though).
503 The format of the return value is a dictionary which maps
504 line numbers to their contents.
505 docstrip::util::patch source-var terminals fromtext diff ?option value
507 ...?
508 This command tries to apply a diff file (for example a contrib‐
509 uted patch) that was computed for a generated file to the doc‐
510 strip source. This can be useful if someone has edited a gener‐
511 ated file, thus mistaking it for being the source. This command
512 makes no presumptions which are specific for the case that the
513 generated file is a Tcl script.
514 patch requires that the source file to patch is kept as a list
516 of lines in a variable, and the name of that variable in the
517 calling context is what goes into the source-var argument. The
518 terminals is the list of terminals used to extract the file that
519 has been patched. The diff is the actual diff to apply (in a
520 format as explained below) and the fromtext is the contents of
521 the file which served as "from" when the diff was computed. Op‐
522 tions can be used to further control the process.
523 The process works by "lifting" the hunks in the diff from gener‐
525 ated to source file, and then applying them to the elements of
526 the source-var. In order to do this lifting, it is necessary to
527 determine how lines in the fromtext correspond to elements of
528 the source-var, and that is where the terminals come in; the
529 source is first extracted under the given terminals, and the re‐
530 sult of that is then matched against the fromtext. This produces
531 a map which translates line numbers stated in the diff to ele‐
532 ment numbers in source-var, which is what is needed to lift the
533 hunks.
534 The reason that both the terminals and the fromtext must be
536 given is twofold. First, it is very difficult to keep track of
537 how many lines of preamble are supplied some other way than by
538 copying lines from source files. Second, a generated file might
539 contain material from several source files. Both make it impos‐
540 sible to predict what line number an extracted file would have
541 in the generated file, so instead the algorithm for computing
542 the line number map looks for a block of lines in the fromtext
543 which matches what can be extracted from the source. This match‐
544 ing is affected by the following options:
545 -matching mode
547 How equal must two lines be in order to match? The sup‐
548 ported modes are:
549 exact Lines must be equal as strings. This is the de‐
551 fault.
552 anyspace
554 All sequences of whitespace characters are con‐
555 verted to single spaces before comparing.
556 nonspace
558 Only non-whitespace characters are considered when
559 comparing.
560 none Any two lines are considered to be equal.
562 -metaprefix string
564 The -metaprefix value to use when extracting. Defaults to
565 "%%", but for Tcl code it is more likely that "#" or "##"
566 had been used for the generated file.
567 -trimlines boolean
569 The -trimlines value to use when extracting. Defaults to
570 true.
571 The return value is in the form of a unified diff, containing
573 only those hunks which were not applied or were only partially
574 applied; a comment in the header of each hunk specifies which
575 case is at hand. It is normally necessary to manually review
576 both the return value from patch and the patched text itself, as
577 this command cannot adjust comment lines to match new content.
578 An example use would look like
580 set sourceL [split [docstrip::util::thefile from.dtx] \n]
583 set terminals {foo bar baz}
584 set fromtext [docstrip::util::thefile from.tcl]
585 set difftext [exec diff --unified from.tcl to.tcl]
586 set leftover [docstrip::util::patch sourceL $terminals $fromtext\
587 [docstrip::util::import_unidiff $difftext] -metaprefix {#}]
588 set F [open to.dtx w]; puts $F [join $sourceL \n]; close $F
589 return $leftover
590 Here, "from.dtx" was used as source for "from.tcl", which some‐
593 one modified into "to.tcl". We're trying to construct a "to.dtx"
594 which can be used as source for "to.tcl".
595 docstrip::util::thefile filename ?option value ...?
597 The thefile command opens the file filename, reads it to end,
598 closes it, and returns the contents (dropping a final newline if
599 there is one). The option-value pairs are passed on to fconfig‐
600 ure to configure the open file channel before anything is read
601 from it.
602 docstrip::util::import_unidiff diff-text ?warning-var?
604 This command parses a unified (diff flags -U and --unified) for‐
605 mat diff into the list-of-hunks format expected by doc‐
606 strip::util::patch. The diff-text argument is the text to parse
607 and the warning-var is, if specified, the name in the calling
608 context of a variable to which any warnings about parsing prob‐
609 lems will be appended.
610 The return value is a list of hunks. Each hunk is a list of five
612 elements "start1 end1 start2 end2 lines". start1 and end1 are
613 line numbers in the "from" file of the first and last respec‐
614 tively lines of the hunk. start2 and end2 are the corresponding
615 line numbers in the "to" file. Line numbers start at 1. The
616 lines is a list with two elements for each line in the hunk; the
617 first specifies the type of a line and the second is the actual
618 line contents. The type is - for lines only in the "from" file,
619 + for lines that are only in the "to" file, and 0 for lines that
620 are in both.
621
623 docstrip, doctools, doctools_fmt
624
626 \.ddt, .dtx, LaTeX, Tcl module, catalogue, diff, docstrip, doctools,
627 documentation, literate programming, module, package indexing, patch,
628 source
629
631 Documentation tools
632
634 Copyright (c) 2003–2010 Lars Hellström <Lars dot Hellstrom at residenset dot net>
635tcllib 1.3.1 docstrip_util(n)