1pt_import_api(i) Parser Tools pt_import_api(i)
2______________________________________________________________________________
6
8 pt_import_api - Parser Tools Import API
9
11 package require Tcl 8.5
12 CONVERTER convert text
14 IncludeFile currentfile path
16 ::import text
18______________________________________________________________________________
20
22 Are you lost ? Do you have trouble understanding this document ? In
23 that case please read the overview provided by the Introduction to
24 Parser Tools. This document is the entrypoint to the whole system the
25 current package is a part of.
26 This document describes two APIs. First the API shared by all packages
28 for the conversion of some other format into Parsing Expression Gram‐
29 mars , and then the API shared by the packages which implement the im‐
30 port plugins sitting on top of the conversion packages.
31 Its intended audience are people who wish to create their own converter
33 for some type of input, and/or an import plugin for their or some other
34 converter.
35 It resides in the Import section of the Core Layer of Parser Tools.
37 IMAGE: arch_core_import
39
41 Any (grammar) import converter has to follow the rules set out below:
42 [1] A converter is a package. Its name is arbitrary, however it is
44 recommended to put it under the ::pt::peg::from namespace.
45 [2] The package provides either a single Tcl command following the
47 API outlined below, or a class command whose instances follow
48 the same API. The commands which follow the API are called con‐
49 verter commands.
50 [3] A converter command has to provide the following single method
52 with the given signature and semantic. Converter commands are
53 allowed to provide more methods of their own, but not less, and
54 they may not provide different semantics for the standardized
55 method.
56 CONVERTER convert text
58 This method has to accept some text, a parsing expression
59 grammar in some format. The result of the method has to
60 be the canonical serialization of a parsing expression
61 grammar, as specified in section PEG serialization for‐
62 mat, the result of reading and converting the input text.
63
65 Any (grammar) import plugin has to follow the rules set out below:
66 [1] A plugin is a package.
68 [2] The name of a plugin package has the form pt::peg::import::FOO,
70 where FOO is the name of the format the plugin will accept input
71 for.
72 [3] The plugin can expect that the package pt::peg::import::plugin
74 is present, as indicator that it was invoked from a genuine
75 plugin manager.
76 It is recommended that a plugin does check for the presence of
78 this package.
79 [4] The plugin can expect that a command named IncludeFile is
81 present, with the signature
82 IncludeFile currentfile path
84 This command has to be invoked by the plugin when it has
85 to process an included file, if the format has the con‐
86 cept of such.
87 The plugin has to supply the following arguments
89 string currentfile
91 The path of the file it is currently processing.
92 This may be the empty string if no such is known.
93 string path
95 The path of the include file as specified in the
96 include directive being processed.
97 The result of the command will be a 5-element list con‐
99 taining
100 [1] A boolean flag indicating the success (True) or
102 failure (False) of the operation.
103 [2] In case of success the contents of the included
105 file, and the empty string otherwise.
106 [3] The resolved, i.e. absolute path of the included
108 file, if possible, or the unchanged path argument.
109 This is for display in an error message, or as the
110 currentfile argument of another call to Include‐
111 File should this file contain more files.
112 [4] In case of success an empty string, and for fail‐
114 ure a code indicating the reason for it, one of
115 notfound
117 The specified file could not be found.
118 notread
120 The specified file was found, but not be
121 read into memory.
122 [5] An empty string in case of success of a notfound
124 failure, and an additional error message describ‐
125 ing the reason for a notread error in more detail.
126 [5] A plugin has to provide a single command, in the global name‐
128 space, with the signature shown below. Plugins are allowed to
129 provide more commands of their own, but not less, and they may
130 not provide different semantics for the standardized command.
131 ::import text
133 This command has to accept the a text containing a pars‐
134 ing expression grammar in some format. The result of the
135 command has to be the result of the converter invoked by
136 the plugin for the input grammar, the canonical serial‐
137 ization of the parsing expression grammar contained in
138 the input.
139 string text
141 This argument will contain the parsing expression
142 grammar for which to generate the serialization.
143 The specification of what a canonical serializa‐
144 tion is can be found in the section PEG serializa‐
145 tion format.
146 [6] A single usage cycle of a plugin consists of an invokation of
148 the command import. This call has to leave the plugin in a state
149 where another usage cycle can be run without problems.
150
152 To use a converter do
153 # Get the converter (single command here, not class)
156 package require the-converter-package
157 # Perform the conversion
159 set serial [theconverter convert $thegrammartext]
160 ... process the result ...
162 To use a plugin FOO do
164 # Get an import plugin manager
167 package require pt::peg::import
168 pt::peg::import I
169 # Run the plugin, and the converter inside.
171 set serial [I import serial $thegrammartext FOO]
172 ... process the result ...
174
177 Here we specify the format used by the Parser Tools to serialize Pars‐
178 ing Expression Grammars as immutable values for transport, comparison,
179 etc.
180 We distinguish between regular and canonical serializations. While a
182 PEG may have more than one regular serialization only exactly one of
183 them will be canonical.
184 regular serialization
186 [1] The serialization of any PEG is a nested Tcl dictionary.
188 [2] This dictionary holds a single key, pt::grammar::peg, and
190 its value. This value holds the contents of the grammar.
191 [3] The contents of the grammar are a Tcl dictionary holding
193 the set of nonterminal symbols and the starting expres‐
194 sion. The relevant keys and their values are
195 rules The value is a Tcl dictionary whose keys are the
197 names of the nonterminal symbols known to the
198 grammar.
199 [1] Each nonterminal symbol may occur only
201 once.
202 [2] The empty string is not a legal nonterminal
204 symbol.
205 [3] The value for each symbol is a Tcl dictio‐
207 nary itself. The relevant keys and their
208 values in this dictionary are
209 is The value is the serialization of
211 the parsing expression describing
212 the symbols sentennial structure, as
213 specified in the section PE serial‐
214 ization format.
215 mode The value can be one of three values
217 specifying how a parser should han‐
218 dle the semantic value produced by
219 the symbol.
220 value The semantic value of the
222 nonterminal symbol is an ab‐
223 stract syntax tree consisting
224 of a single node node for the
225 nonterminal itself, which has
226 the ASTs of the symbol's
227 right hand side as its chil‐
228 dren.
229 leaf The semantic value of the
231 nonterminal symbol is an ab‐
232 stract syntax tree consisting
233 of a single node node for the
234 nonterminal, without any
235 children. Any ASTs generated
236 by the symbol's right hand
237 side are discarded.
238 void The nonterminal has no seman‐
240 tic value. Any ASTs generated
241 by the symbol's right hand
242 side are discarded (as well).
243 start The value is the serialization of the start pars‐
245 ing expression of the grammar, as specified in the
246 section PE serialization format.
247 [4] The terminal symbols of the grammar are specified implic‐
249 itly as the set of all terminal symbols used in the start
250 expression and on the RHS of the grammar rules.
251 canonical serialization
253 The canonical serialization of a grammar has the format as spec‐
254 ified in the previous item, and then additionally satisfies the
255 constraints below, which make it unique among all the possible
256 serializations of this grammar.
257 [1] The keys found in all the nested Tcl dictionaries are
259 sorted in ascending dictionary order, as generated by
260 Tcl's builtin command lsort -increasing -dict.
261 [2] The string representation of the value is the canonical
263 representation of a Tcl dictionary. I.e. it does not con‐
264 tain superfluous whitespace.
265 EXAMPLE
267 Assuming the following PEG for simple mathematical expressions
268 PEG calculator (Expression)
270 Digit <- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9' ;
271 Sign <- '-' / '+' ;
272 Number <- Sign? Digit+ ;
273 Expression <- Term (AddOp Term)* ;
274 MulOp <- '*' / '/' ;
275 Term <- Factor (MulOp Factor)* ;
276 AddOp <- '+'/'-' ;
277 Factor <- '(' Expression ')' / Number ;
278 END;
279 then its canonical serialization (except for whitespace) is
282 pt::grammar::peg {
284 rules {
285 AddOp {is {/ {t -} {t +}} mode value}
286 Digit {is {/ {t 0} {t 1} {t 2} {t 3} {t 4} {t 5} {t 6} {t 7} {t 8} {t 9}} mode value}
287 Expression {is {x {n Term} {* {x {n AddOp} {n Term}}}} mode value}
288 Factor {is {/ {x {t (} {n Expression} {t )}} {n Number}} mode value}
289 MulOp {is {/ {t *} {t /}} mode value}
290 Number {is {x {? {n Sign}} {+ {n Digit}}} mode value}
291 Sign {is {/ {t -} {t +}} mode value}
292 Term {is {x {n Factor} {* {x {n MulOp} {n Factor}}}} mode value}
293 }
294 start {n Expression}
295 }
296
299 Here we specify the format used by the Parser Tools to serialize Pars‐
300 ing Expressions as immutable values for transport, comparison, etc.
301 We distinguish between regular and canonical serializations. While a
303 parsing expression may have more than one regular serialization only
304 exactly one of them will be canonical.
305 Regular serialization
307 Atomic Parsing Expressions
309 [1] The string epsilon is an atomic parsing expres‐
311 sion. It matches the empty string.
312 [2] The string dot is an atomic parsing expression. It
314 matches any character.
315 [3] The string alnum is an atomic parsing expression.
317 It matches any Unicode alphabet or digit charac‐
318 ter. This is a custom extension of PEs based on
319 Tcl's builtin command string is.
320 [4] The string alpha is an atomic parsing expression.
322 It matches any Unicode alphabet character. This is
323 a custom extension of PEs based on Tcl's builtin
324 command string is.
325 [5] The string ascii is an atomic parsing expression.
327 It matches any Unicode character below U0080. This
328 is a custom extension of PEs based on Tcl's
329 builtin command string is.
330 [6] The string control is an atomic parsing expres‐
332 sion. It matches any Unicode control character.
333 This is a custom extension of PEs based on Tcl's
334 builtin command string is.
335 [7] The string digit is an atomic parsing expression.
337 It matches any Unicode digit character. Note that
338 this includes characters outside of the [0..9]
339 range. This is a custom extension of PEs based on
340 Tcl's builtin command string is.
341 [8] The string graph is an atomic parsing expression.
343 It matches any Unicode printing character, except
344 for space. This is a custom extension of PEs based
345 on Tcl's builtin command string is.
346 [9] The string lower is an atomic parsing expression.
348 It matches any Unicode lower-case alphabet charac‐
349 ter. This is a custom extension of PEs based on
350 Tcl's builtin command string is.
351 [10] The string print is an atomic parsing expression.
353 It matches any Unicode printing character, includ‐
354 ing space. This is a custom extension of PEs based
355 on Tcl's builtin command string is.
356 [11] The string punct is an atomic parsing expression.
358 It matches any Unicode punctuation character. This
359 is a custom extension of PEs based on Tcl's
360 builtin command string is.
361 [12] The string space is an atomic parsing expression.
363 It matches any Unicode space character. This is a
364 custom extension of PEs based on Tcl's builtin
365 command string is.
366 [13] The string upper is an atomic parsing expression.
368 It matches any Unicode upper-case alphabet charac‐
369 ter. This is a custom extension of PEs based on
370 Tcl's builtin command string is.
371 [14] The string wordchar is an atomic parsing expres‐
373 sion. It matches any Unicode word character. This
374 is any alphanumeric character (see alnum), and any
375 connector punctuation characters (e.g. under‐
376 score). This is a custom extension of PEs based on
377 Tcl's builtin command string is.
378 [15] The string xdigit is an atomic parsing expression.
380 It matches any hexadecimal digit character. This
381 is a custom extension of PEs based on Tcl's
382 builtin command string is.
383 [16] The string ddigit is an atomic parsing expression.
385 It matches any decimal digit character. This is a
386 custom extension of PEs based on Tcl's builtin
387 command regexp.
388 [17] The expression [list t x] is an atomic parsing ex‐
390 pression. It matches the terminal string x.
391 [18] The expression [list n A] is an atomic parsing ex‐
393 pression. It matches the nonterminal A.
394 Combined Parsing Expressions
396 [1] For parsing expressions e1, e2, ... the result of
398 [list / e1 e2 ... ] is a parsing expression as
399 well. This is the ordered choice, aka prioritized
400 choice.
401 [2] For parsing expressions e1, e2, ... the result of
403 [list x e1 e2 ... ] is a parsing expression as
404 well. This is the sequence.
405 [3] For a parsing expression e the result of [list *
407 e] is a parsing expression as well. This is the
408 kleene closure, describing zero or more repeti‐
409 tions.
410 [4] For a parsing expression e the result of [list +
412 e] is a parsing expression as well. This is the
413 positive kleene closure, describing one or more
414 repetitions.
415 [5] For a parsing expression e the result of [list &
417 e] is a parsing expression as well. This is the
418 and lookahead predicate.
419 [6] For a parsing expression e the result of [list !
421 e] is a parsing expression as well. This is the
422 not lookahead predicate.
423 [7] For a parsing expression e the result of [list ?
425 e] is a parsing expression as well. This is the
426 optional input.
427 Canonical serialization
429 The canonical serialization of a parsing expression has the for‐
430 mat as specified in the previous item, and then additionally
431 satisfies the constraints below, which make it unique among all
432 the possible serializations of this parsing expression.
433 [1] The string representation of the value is the canonical
435 representation of a pure Tcl list. I.e. it does not con‐
436 tain superfluous whitespace.
437 [2] Terminals are not encoded as ranges (where start and end
439 of the range are identical).
440 EXAMPLE
442 Assuming the parsing expression shown on the right-hand side of the
443 rule
444 Expression <- Term (AddOp Term)*
446 then its canonical serialization (except for whitespace) is
449 {x {n Term} {* {x {n AddOp} {n Term}}}}
451
454 This document, and the package it describes, will undoubtedly contain
455 bugs and other problems. Please report such in the category pt of the
456 Tcllib Trackers [http://core.tcl.tk/tcllib/reportlist]. Please also
457 report any ideas for enhancements you may have for either package
458 and/or documentation.
459 When proposing code changes, please provide unified diffs, i.e the out‐
461 put of diff -u.
462 Note further that attachments are strongly preferred over inlined
464 patches. Attachments can be made by going to the Edit form of the
465 ticket immediately after its creation, and then using the left-most
466 button in the secondary navigation bar.
467
469 EBNF, LL(k), PEG, TDPL, context-free languages, expression, grammar,
470 matching, parser, parsing expression, parsing expression grammar, push
471 down automaton, recursive descent, state, top-down parsing languages,
472 transducer
473
475 Parsing and Grammars
476
478 Copyright (c) 2009 Andreas Kupries <andreas_kupries@users.sourceforge.net>
479tcllib 1 pt_import_api(i)