1Pegex::Tutorial::JSON(3U)ser Contributed Perl DocumentatiPoengex::Tutorial::JSON(3)
2
6 This document details the creation of the CPAN module: Pegex::JSON
7 which is a JSON parser/decoder written in Perl using the Pegex parsing
8 framework. The code lives on github here:
9 <https://github.com/ingydotnet/pegex-json-pm>.
10
12 Have a look at
13 <https://github.com/ingydotnet/pegex-json-pm/blob/master/test/test.t>.
14 This simple test has a bunch of small pieces of JSON and their YAML
15 equivalents. It asserts that when the JSON is decoded, it will match
16 the YAML.
17 The test is written in a testing language known as TestML. TestML just
19 happens to also use Pegex in its compiler. Both TestML and Pegex are
20 Acmeist frameworks, meaning that they are intended to work in multiple
21 programming languages.
22 You can run the test like normal:
24 > prove -lv t/test.t
26
28 The next thing to do is write the JSON grammar in the Pegex grammar
29 language. Writing grammars is the heart and soul of using Pegex. A
30 grammar is simply a definition of a language that specifies what is
31 what, and how it must be structured.
32 Since Pegex is Acmeist, I put the JSON grammar in its own repo so that
34 it could be shared by many different projects in different programming
35 languages. The grammar file is here:
36 <https://github.com/ingydotnet/json-pgx/blob/master/json.pgx>.
37 Let's look at this small but complete language definition in detail.
39 The file starts with some comments. Comments can be used liberally in
41 Pegex and go from a '#' to the end of a line. Just as you would expect.
42 # A simple grammar for the simple JSON data language.
44 # For parser implementations that use this grammar, see:
45 # * https://github.com/ingydotnet/pegex-json-pm
46 Next we have what is called the Meta section of the grammar.
48 %grammar json
50 %version 0.0.1
51 Meta section lines are of the form:
53 %key value
55 Your grammar should have at least a name and a version.
57 Everything else in the grammar is a set of things called "rules". A
59 rule has a name and a definition. The first rule in a grammar is
60 special. When a parser starts parsing it defaults to using the first
61 rule as the starting rule (although this can be overridden).
62 We start the JSON grammar with this rule:
64 json: map | seq
66 The name of this rule is 'json'. When we start parsing JSON we say that
68 the entire text must match the rule 'json'. This makes a lot of sense.
69 This style of parsing is known as Top Down and Recursive Descent. Pegex
71 is both of these. It should be noted that Pegex does not tokenize a
72 text before parsing it. The rules themselves form a kind of tokenizer,
73 pulling out the desired data segments as needed.
74 In this rule, we are saying that a 'json' document is either a 'map'
76 (aka mapping or hash) OR it is a 'seq' (aka sequence or array), which
77 assuming you know JSON (almost everybody does), is the only thing
78 allowed at the top level.
79 In this rule, 'map' and 'seq' are called 'rule references'. They point
81 to other named rules that are expected to be in the grammar. References
82 are usually just the name of the subrule itself, but can also be
83 enclosed in angle brackets (which are sometimes required). ie the rule
84 above could also be written like this:
85 json: map | seq
87 We are also introduced to the OR operator which is a single PIPE
89 character. It should also be noted that a COLON simply separates a rule
90 name and its definition.
91 The next line defines a new rule called 'node':
93 node: map | seq | scalar
95 We are calling 'node' the list of general structures that any given
97 point in the JSON data graph can be. This is simply a mapping, sequence
98 or scalar.
99 Moving on, we need rules describing 'map', 'seq' 'scalar'. A grammar is
101 complete when all of its rule references are defined.
102 Let's start with map:
104 map:
106 /- LCURLY -/
107 pair* % /- COMMA -/
108 /- RCURLY -/
109 This seems a lot more complicated, but let's break things down, one at
111 a time. What we will find this rule to mean, is that a map is a '{'
112 followed by zero or more (key/value) pairs separated by a comma, then
113 ending with a '}'. Along the way there may also be intermittent
114 whitespace. The '-' character indicates whitespace, but we'll cover
115 that more later.
116 The first thing we notice is that a rule definition can span multiple
118 lines. A rule definition ends when the next rule begins. Pegex actually
119 allows for multiple rules on one line, but they must be separated by
120 semicolons, like so:
121 rule1: a|b; rule2: c|d; rule3: e|f
123 The next thing we see are forward slash characters. Like in Perl and
125 JavaScript, a pair of slashes indicate a regular expression. In this
126 rule we have 3 regular expressions.
127 It is a good time to note that Pegex grammars get compiled into a
129 simple data structure that you can express as JSON or YAML. In fact the
130 repository containing the Pegex grammar that we are discussing also
131 contains the matching compiled forms. See:
132 <https://github.com/ingydotnet/json-pgx/blob/master/json.pgx.yaml>. A
133 quick glance at this file shows all the same rule definitions, but the
134 regexes look much different.
135 That's because Pegex tries to make regexes readable and composable.
137 That means that complex regexes can be defined as small parts that get
138 composed into bigger parts. By the time they get compiled, they can be
139 quite hard to understand.
140 For example if we had this Pegex grammar:
142 greeting: / hello COMMA - world /
144 hello: / (:'O' SPACE 'HAI' | 'Hey' SPACE 'there') /
145 world: / (:Earth|Mars|Venus) /
146 It would compile to:
148 greeting:
150 .rgx: (?:O\ HAI|Hey\ there),\s*(?:Earth|Mars|Venus)
151 Note the the 'hello' and 'world' rules are gone, but their definitions
153 have been baked into the one big regex for 'greeting'.
154 Additionally there are references to things like COMMA and SPACE. These
156 are called Pegex Atoms, and there are atoms for all the punctuation
157 characters, whitespace chars, and others. The full list is here:
158 Pegex::Grammar::Atoms.
159 Having to write out 'SEMI' instead of ';' seems strange at first, but
161 it is how Pegex easily separates metasyntax from text to be matched.
162 Once you get used to it, it is very readable.
163 The actual whitespace (and comments) inside a regex are completely
165 ignored by Pegex. This is the same as Perl's 'x' regex flag.
166 Finally the '-' is Pegex's 'possible whitespace' indicator, and usually
168 expands to "\s*". It actually expands to "ws1", which expands to "ws*",
169 which expands to "WS*", which expands to "\s*" (unless you override any
170 of those rules).
171 Getting back to JSON...
173 The rule we defined for 'map' should now be more readable. Let's look
175 at it again, but this time in a more compact form:
176 map: /- LCURLY -/ (pair* % /- COMMA -/) /- RCURLY -/
178 I've compacted the regexes (since they just mean curlies and commas
180 with possible whitespace), and I've added parentheses around the middle
181 stuff to indicate the the '%' operator has a tighter binding.
182 So what is the '%' operator? It was borrowed from Perl 6 Rules.
184 Consider:
185 a+ % b
187 This means one or more 'a', separated by 'b'. Simple. The %% operator
189 means the same thing, except it indicates that a trailing 'b' is OK.
190 This notation is handy for things like comma separated lists. (Which is
192 exactly what we are using it for here.)
193 The rule above means zero or more 'pair's separated by commas.
195 (trailing comma not allowed, which is strictly correct for JSON).
196 Now is a good time to bring up 'rule quantifiers'. A rule quantifier is
198 a suffix to a rule reference, and can be one of ? * or +. These
199 suffixes mean the same thing that they would in regexes.
200 There are two other quantifier suffixes. '2+' is equivalent to the
202 regex syntax {2,} and 2-5 is the same as {2,5}. When you use one of
203 these two forms, you need to put the rule reference in angle brackets,
204 or else the number looks like part of the rule name. For example:
205 rule1: <rule2>5-9 <rule3>29+
207 not:
209 rule1: rule25-9 rule329+
211 Let's take a look at that rule after Pegex compilation:
213 map:
215 .all:
216 - .rgx: \s*\{\s*
217 - +min: 0
218 .ref: pair
219 .sep:
220 .rgx: \s*,\s*
221 - .rgx: \s*\}\s*
222 The rule for 'map' says that the text must match 3 thing: a regex
224 (opening curly brace), zero or more occurrences of a rule called 'pair'
225 separated by a regex (comma), and finally another regex (closing
226 curly).
227 One thing that we have silently covered is the AND operator. That's
229 because there is no operator symbol for it. Consider the rules:
230 a: b c+ /d/
232 b: c | d
233 c: d e | f % g
234 The PIPE character between 2 things means OR. No symbol between 2
236 things means AND. A PERCENT means ALTERNATION. ALTERNATION binds
237 tightest and OR binds loosest, with AND in the middle. Precedence can
238 be disambiguated with parentheses. Thus the rule for 'c' can be
239 restated:
240 c: ((d e) | (f % g))
242 OK. I think we've covered just about everything needed so far. That was
244 a lot of learning for one rule, but now you know most of Pegex!
245 The next three rules need no new knowledge. Take a look at these and
247 see if you can figure them out.
248 pair:
250 string
251 /- COLON -/
252 node
253 seq:
255 /- LSQUARE -/
256 node* % /- COMMA -/
257 /- RSQUARE -/
258 scalar:
260 string |
261 number |
262 boolean |
263 null
264 A pair (you know... a hash key/value), is a string, a colon, and some
266 node. A seq is zero or more comma-separated nodes between square
267 brackets. A scalar can be one of 4 forms. Simple.
268 One interesting point is that has just arisen here, is the use of
270 recursion. The rules for pair and seq both reference the rule for
271 node. Thus, the grammar is recursive descent. It starts with the rule
272 for the thing as a whole (ie 'json') and descends (recursively) until
273 it matches all the specific characters.
274 Pegex Regexes in More Depth
276 Next we have the definition for a JSON string:
277 # string and number are interpretations of http://www.json.org/
279 string: /
280 DOUBLE
281 (
282 (:
283 BACK (: # Backslash escapes
284 [
285 DOUBLE # Double Quote
286 BACK # Back Slash
287 SLASH # Forward Slash
288 b # Back Space
289 f # Form Feed
290 n # New Line
291 r # Carriage Return
292 t # Horizontal Tab
293 ]
294 |
295 u HEX{4} # Unicode octet pair
296 )
297 |
298 [^ DOUBLE CONTROLS ] # Anything else
299 )*
300 )
301 DOUBLE
302 /
303 which Pegex compiles to this (simple and obvious;) regex:
305 /"((?:\\(?:["\\/bfnrt]|u[0-9a-fA-F]{4})|[^"\x00-\x1f])*)"/
307 Let's see what's new here...
309 First off, we have lots of whitespace and comments. This should make it
311 pretty easy to at least get the overall picture of what is being
312 accomplished.
313 Understanding how the text between a pair of '/' characters gets
315 transformed into a real regular expression, is the key to really
316 understanding Pegex.
317 Notice the '*', the '{4}', the '|', the '(...)' and the '^...'. All of
319 this punctuation gets passed on verbatim into the compiled regex. There
320 are just a few exceptions. Let's cover them in detail.
321 Everything inside "<some_rule_ref>" gets replaced by the regex that the
323 reference points to. Rule references inside a regex must point directly
324 to another reference, although those rules can point to even more regex
325 parts.
326 The "-" characters get replaced by 'ws1', which is subsequently
328 replaced by its rule definition. "+" gets replaced by "ws2" (which
329 resolves to "\s+" by default).
330 Finally '(:' gets replaced by '(?:)'. This is simply to make your non-
332 capturing paren syntax be a little prettier. In general, you can leave
333 out '?' after a '(' and Pegex will put them in for you.
334 That's it. Everything else that you put between slash characters, will
336 go verbatim into the regex.
337 In some sense, Pegex is just a very highly organized way to write a
339 parser from regular expressions. To be really good at Pegex does
340 require fairly solid understanding of how regexes work, but given that
341 regexes are so very common, Pegex makes the task of turning them into a
342 Parser, quite simple.
343 Capturing Data
345 The next thing to cover is regex capturing. When you are parsing data,
346 it is generally important to pull out certain chunks of text so that
347 you can do things with them.
348 Pegex has one very simple, straightforward and obvious way to do this:
350 Any capturing parens in any regexes will capture data and pass it on to
351 the receiver object. Since Pegex is built over regexes, this make
352 perfect sense.
353 We will talk about receiver objects in the next section. For now, just
355 know that a receiver object is the thing that makes something out of
356 the data and events matched during a parse. In our JSON case here, our
357 receiver will make a Perl data structure that matches the JSON we are
358 parsing. Obvious!
359 In the rule for 'string' above, we are capturing the characters between
361 the double quotes. This is the raw data that will be turned into a Perl
362 scalar.
363 Finishing up the JSON Grammar
365 There are just 5 more simple rules needed to complete the JSON Pegex
366 grammar:
367 number: /(
369 DASH?
370 (: 0 | [1-9] DIGIT* )
371 (: DOT DIGIT* )?
372 (: [eE] [ DASH PLUS ]? DIGIT+ )?
373 )/
374 boolean: true | false
376 true: /true/
378 false: /false/
380 null: /null/
382 Note that the rule for 'number' captures data, but the other rules
384 don't. That's because as long as the receiver is told that a 'null'
385 rule was matched, it can turn it into a Perl "undef". It's not
386 important what the actual matching text was (even though in this case
387 it has to be exactly the string 'null').
388
390 The Pegex JSON grammar that we just looked at in excruciating detail
391 gets compiled into a Perl data structure and then embedded into a
392 grammar class. A Pegex::Parser object (the thing that does the parsing)
393 requires a grammar object (what to look for) and a receiver object
394 (what to do when you find it).
395 It should be noted that Pegex uses Pegex to parse Pegex grammars. That
397 is, this grammar:
398 <https://github.com/ingydotnet/pegex-pgx/blob/master/pegex.pgx> is used
399 by Pegex to parse our json.pgx grammar (and yes, it can parse pegex.pgx
400 too).
401 It is conceivable that every time we wanted to parse JSON, we could
403 parse the json.pgx grammar first, but that would be a waste of time. So
404 we cache the big grammar tree as a pure Perl data structure inside
405 Pegex::JSON::Grammar.
406 If for some reason we did need to change the json.pgx, we would have to
408 recompile it to Perl and copy/paste it into our module. This would be a
409 pain, so there is a special command to do it for us. Just run this:
410 perl -Ilib -MPegex::JSON:Grammar=compile
412 If you are in heavy development mode and changing the grammar a lot,
414 you can simply set an environment variable like this:
415 export PERL_PEGEX_AUTO_COMPILE=Pegex::JSON::Grammar
417 and recompilation will happen automatically. This is possible because
419 of this line in the grammar module:
420 use constant file => '../json-pgx/json.pgx';
422 That line is only used during development and requires the grammar file
424 to be in that location.
425
427 Every Pegex parse requires a grammar object (the JSON grammar), and
428 input object (the JSON being parsed) and a receiver object (the Perl
429 maker).
430 One of the hallmarks of Pegex, is that it keeps the grammar separate
432 from the (receiver) code, thus
433perl v5.30.0 2019-07-26 Pegex::Tutorial::JSON(3)