1HTML::TreeBuilder(3) User Contributed Perl Documentation HTML::TreeBuilder(3)
2
6 HTML::TreeBuilder - Parser that builds a HTML syntax tree
7
9 foreach my $file_name (@ARGV) {
10 my $tree = HTML::TreeBuilder->new; # empty tree
11 $tree->parse_file($file_name);
12 print "Hey, here's a dump of the parse tree of $file_name:\n";
13 $tree->dump; # a method we inherit from HTML::Element
14 print "And here it is, bizarrely rerendered as HTML:\n",
15 $tree->as_HTML, "\n";
16 # Now that we're done with it, we must destroy it.
18 $tree = $tree->delete;
19 }
20
22 (This class is part of the HTML::Tree dist.)
23 This class is for HTML syntax trees that get built out of HTML source.
25 The way to use it is to:
26 1. start a new (empty) HTML::TreeBuilder object,
28 2. then use one of the methods from HTML::Parser (presumably with
30 $tree->parse_file($filename) for files, or with $tree->parse($docu‐
31 ment_content) and $tree->eof if you've got the content in a string) to
32 parse the HTML document into the tree $tree.
33 (You can combine steps 1 and 2 with the "new_from_file" or
35 "new_from_content" methods.)
36 2b. call $root->elementify() if you want.
38 3. do whatever you need to do with the syntax tree, presumably involv‐
40 ing traversing it looking for some bit of information in it,
41 4. and finally, when you're done with the tree, call $tree->delete() to
43 erase the contents of the tree from memory. This kind of thing usually
44 isn't necessary with most Perl objects, but it's necessary for Tree‐
45 Builder objects. See HTML::Element for a more verbose explanation of
46 why this is the case.
47
49 Objects of this class inherit the methods of both HTML::Parser and
50 HTML::Element. The methods inherited from HTML::Parser are used for
51 building the HTML tree, and the methods inherited from HTML::Element
52 are what you use to scrutinize the tree. Besides this (HTML::Tree‐
53 Builder) documentation, you must also carefully read the HTML::Element
54 documentation, and also skim the HTML::Parser documentation -- probably
55 only its parse and parse_file methods are of interest.
56 Most of the following methods native to HTML::TreeBuilder control how
58 parsing takes place; they should be set before you try parsing into the
59 given object. You can set the attributes by passing a TRUE or FALSE
60 value as argument. E.g., $root->implicit_tags returns the current set‐
61 ting for the implicit_tags option, $root->implicit_tags(1) turns that
62 option on, and $root->implicit_tags(0) turns it off.
63 $root = HTML::TreeBuilder->new_from_file(...)
65 This "shortcut" constructor merely combines constructing a new
66 object (with the "new" method, below), and calling
67 $new->parse_file(...) on it. Returns the new object. Note that
68 this provides no way of setting any parse options like store_com‐
69 ments (for that, call new, and then set options, before calling
70 parse_file). See the notes (below) on parameters to parse_file.
71 $root = HTML::TreeBuilder->new_from_content(...)
73 This "shortcut" constructor merely combines constructing a new
74 object (with the "new" method, below), and calling
75 for(...){$new->parse($_)} and $new->eof on it. Returns the new
76 object. Note that this provides no way of setting any parse
77 options like store_comments (for that, call new, and then set
78 options, before calling parse_file). Example usages: HTML::Tree‐
79 Builder->new_from_content(@lines), or HTML::Tree‐
80 Builder->new_from_content($content)
81 $root = HTML::TreeBuilder->new()
83 This creates a new HTML::TreeBuilder object. This method takes no
84 attributes.
85 $root->parse_file(...)
87 [An important method inherited from HTML::Parser, which see. Cur‐
88 rent versions of HTML::Parser can take a filespec, or a filehandle
89 object, like *FOO, or some object from class IO::Handle, IO::File,
90 IO::Socket) or the like. I think you should check that a given
91 file exists before calling $root->parse_file($filespec).]
92 $root->parse(...)
94 [A important method inherited from HTML::Parser, which see. See
95 the note below for $root->eof().]
96 $root->eof()
98 This signals that you're finished parsing content into this tree;
99 this runs various kinds of crucial cleanup on the tree. This is
100 called for you when you call $root->parse_file(...), but not when
101 you call $root->parse(...). So if you call $root->parse(...), then
102 you must call $root->eof() once you've finished feeding all the
103 chunks to parse(...), and before you actually start doing anything
104 else with the tree in $root.
105 "$root->parse_content(...)"
107 Basically a happly alias for "$root->parse(...); $root->eof".
108 Takes the exact same arguments as "$root->parse()".
109 $root->delete()
111 [An important method inherited from HTML::Element, which see.]
112 $root->elementify()
114 This changes the class of the object in $root from HTML::Tree‐
115 Builder to the class used for all the rest of the elements in that
116 tree (generally HTML::Element). Returns $root.
117 For most purposes, this is unnecessary, but if you call this after
119 (after!!) you've finished building a tree, then it keeps you from
120 accidentally trying to call anything but HTML::Element methods on
121 it. (I.e., if you accidentally call "$root->parse_file(...)" on
122 the already-complete and elementified tree, then instead of charg‐
123 ing ahead and wreaking havoc, it'll throw a fatal error -- since
124 $root is now an object just of class HTML::Element which has no
125 "parse_file" method.
126 Note that elementify currently deletes all the private attributes
128 of $root except for "_tag", "_parent", "_content", "_pos", and
129 "_implicit". If anyone requests that I change this to leave in yet
130 more private attributes, I might do so, in future versions.
131 @nodes = $root->guts()
133 $parent_for_nodes = $root->guts()
134 In list context (as in the first case), this method returns the
135 topmost non-implicit nodes in a tree. This is useful when you're
136 parsing HTML code that you know doesn't expect an HTML document,
137 but instead just a fragment of an HTML document. For example, if
138 you wanted the parse tree for a file consisting of just this:
139 <li>I like pie!
141 Then you would get that with "@nodes = $root->guts();". It so hap‐
143 pens that in this case, @nodes will contain just one element
144 object, representing the "li" node (with "I like pie!" being its
145 text child node). However, consider if you were parsing this:
146 <hr>Hooboy!<hr>
148 In that case, "$root->guts()" would return three items: an element
150 object for the first "hr", a text string "Hooboy!", and another
151 "hr" element object.
152 For cases where you want definitely one element (so you can treat
154 it as a "document fragment", roughly speaking), call "guts()" in
155 scalar context, as in "$parent_for_nodes = $root->guts()". That
156 works like "guts()" in list context; in fact, "guts()" in list con‐
157 text would have returned exactly one value, and if it would have
158 been an object (as opposed to a text string), then that's what
159 "guts" in scalar context will return. Otherwise, if "guts()" in
160 list context would have returned no values at all, then "guts()" in
161 scalar context returns undef. In all other cases, "guts()" in
162 scalar context returns an implicit 'div' element node, with chil‐
163 dren consisting of whatever nodes "guts()" in list context would
164 have returned. Note that that may detach those nodes from $root's
165 tree.
166 @nodes = $root->disembowel()
168 $parent_for_nodes = $root->disembowel()
169 The "disembowel()" method works just like the "guts()" method,
170 except that disembowel definitively destroys the tree above the
171 nodes that are returned. Usually when you want the guts from a
172 tree, you're just going to toss out the rest of the tree anyway, so
173 this saves you the bother. (Remember, "disembowel" means "remove
174 the guts from".)
175 $root->implicit_tags(value)
177 Setting this attribute to true will instruct the parser to try to
178 deduce implicit elements and implicit end tags. If it is false you
179 get a parse tree that just reflects the text as it stands, which is
180 unlikely to be useful for anything but quick and dirty parsing.
181 (In fact, I'd be curious to hear from anyone who finds it useful to
182 have implicit_tags set to false.) Default is true.
183 Implicit elements have the implicit() attribute set.
185 $root->implicit_body_p_tag(value)
187 This controls an aspect of implicit element behavior, if
188 implicit_tags is on: If a text element (PCDATA) or a phrasal ele‐
189 ment (such as "<em>") is to be inserted under "<body>", two things
190 can happen: if implicit_body_p_tag is true, it's placed under a
191 new, implicit "<p>" tag. (Past DTDs suggested this was the only
192 correct behavior, and this is how past versions of this module
193 behaved.) But if implicit_body_p_tag is false, nothing is impli‐
194 cated -- the PCDATA or phrasal element is simply placed under
195 "<body>". Default is false.
196 $root->ignore_unknown(value)
198 This attribute controls whether unknown tags should be represented
199 as elements in the parse tree, or whether they should be ignored.
200 Default is true (to ignore unknown tags.)
201 $root->ignore_text(value)
203 Do not represent the text content of elements. This saves space if
204 all you want is to examine the structure of the document. Default
205 is false.
206 $root->ignore_ignorable_whitespace(value)
208 If set to true, TreeBuilder will try to avoid creating ignorable
209 whitespace text nodes in the tree. Default is true. (In fact, I'd
210 be interested in hearing if there's ever a case where you need this
211 off, or where leaving it on leads to incorrect behavior.)
212 $root->no_space_compacting(value)
214 This determines whether TreeBuilder compacts all whitespace strings
215 in the document (well, outside of PRE or TEXTAREA elements), or
216 leaves them alone. Normally (default, value of 0), each string of
217 contiguous whitespace in the document is turned into a single
218 space. But that's not done if no_space_compacting is set to 1.
219 Setting no_space_compacting to 1 might be useful if you want to
221 read in a tree just to make some minor changes to it before writing
222 it back out.
223 This method is experimental. If you use it, be sure to report any
225 problems you might have with it.
226 $root->p_strict(value)
228 If set to true (and it defaults to false), TreeBuilder will take a
229 narrower than normal view of what can be under a "p" element; if it
230 sees a non-phrasal element about to be inserted under a "p", it
231 will close that "p". Otherwise it will close p elements only for
232 other "p"'s, headings, and "form" (altho the latter may be removed
233 in future versions).
234 For example, when going thru this snippet of code,
236 <p>stuff
238 <ul>
239 TreeBuilder will normally (with "p_strict" false) put the "ul" ele‐
241 ment under the "p" element. However, with "p_strict" set to true,
242 it will close the "p" first.
243 In theory, there should be strictness options like this for
245 other/all elements besides just "p"; but I treat this as a specal
246 case simply because of the fact that "p" occurs so frequently and
247 its end-tag is omitted so often; and also because application of
248 strictness rules at parse-time across all elements often makes tiny
249 errors in HTML coding produce drastically bad parse-trees, in my
250 experience.
251 If you find that you wish you had an option like this to enforce
253 content-models on all elements, then I suggest that what you want
254 is content-model checking as a stage after TreeBuilder has finished
255 parsing.
256 $root->store_comments(value)
258 This determines whether TreeBuilder will normally store comments
259 found while parsing content into $root. Currently, this is off by
260 default.
261 $root->store_declarations(value)
263 This determines whether TreeBuilder will normally store markup dec‐
264 larations found while parsing content into $root. This is on by
265 default.
266 $root->store_pis(value)
268 This determines whether TreeBuilder will normally store processing
269 instructions found while parsing content into $root -- assuming a
270 recent version of HTML::Parser (old versions won't parse PIs cor‐
271 rectly). Currently, this is off (false) by default.
272 It is somewhat of a known bug (to be fixed one of these days, if
274 anyone needs it?) that PIs in the preamble (before the "html"
275 start-tag) end up actually under the "html" element.
276 $root->warn(value)
278 This determines whether syntax errors during parsing should gener‐
279 ate warnings, emitted via Perl's "warn" function.
280 This is off (false) by default.
282
284 HTML is rather harder to parse than people who write it generally sus‐
285 pect.
286 Here's the problem: HTML is a kind of SGML that permits "minimization"
288 and "implication". In short, this means that you don't have to close
289 every tag you open (because the opening of a subsequent tag may implic‐
290 itly close it), and if you use a tag that can't occur in the context
291 you seem to using it in, under certain conditions the parser will be
292 able to realize you mean to leave the current context and enter the new
293 one, that being the only one that your code could correctly be inter‐
294 preted in.
295 Now, this would all work flawlessly and unproblematically if: 1) all
297 the rules that both prescribe and describe HTML were (and had been)
298 clearly set out, and 2) everyone was aware of these rules and wrote
299 their code in compliance to them.
300 However, it didn't happen that way, and so most HTML pages are diffi‐
302 cult if not impossible to correctly parse with nearly any set of
303 straightforward SGML rules. That's why the internals of HTML::Tree‐
304 Builder consist of lots and lots of special cases -- instead of being
305 just a generic SGML parser with HTML DTD rules plugged in.
306
308 The techniques that HTML::TreeBuilder uses to perform what I consider
309 very robust parses on everyday code are not things that can work only
310 in Perl. To date, the algorithms at the center of HTML::TreeBuilder
311 have been implemented only in Perl, as far as I know; and I don't fore‐
312 see getting around to implementing them in any other language any time
313 soon.
314 If, however, anyone is looking for a semester project for an applied
316 programming class (or if they merely enjoy extra-curricular masochism),
317 they might do well to see about choosing as a topic the implementa‐
318 tion/adaptation of these routines to any other interesting programming
319 language that you feel currently suffers from a lack of robust
320 HTML-parsing. I welcome correspondence on this subject, and point out
321 that one can learn a great deal about languages by trying to translate
322 between them, and then comparing the result.
323 The HTML::TreeBuilder source may seem long and complex, but it is
325 rather well commented, and symbol names are generally self-explanatory.
326 (You are encouraged to read the Mozilla HTML parser source for compari‐
327 son.) Some of the complexity comes from little-used features, and some
328 of it comes from having the HTML tokenizer (HTML::Parser) being a sepa‐
329 rate module, requiring somewhat of a different interface than you'd
330 find in a combined tokenizer and tree-builder. But most of the length
331 of the source comes from the fact that it's essentially a long list of
332 special cases, with lots and lots of sanity-checking, and sanity-recov‐
333 ery -- because, as Roseanne Rosannadanna once said, "it's always some‐
334 thing".
335 Users looking to compare several HTML parsers should look at the source
337 for Raggett's Tidy ("<http://www.w3.org/People/Raggett/tidy/>"),
338 Mozilla ("<http://www.mozilla.org/>"), and possibly root around the
339 browsers section of Yahoo to find the various open-source ones
340 ("<http://dir.yahoo.com/Computers_and_Internet/Software/Inter‐
341 net/World_Wide_Web/Browsers/>").
342
344 * Framesets seem to work correctly now. Email me if you get a strange
345 parse from a document with framesets.
346 * Really bad HTML code will, often as not, make for a somewhat objec‐
348 tionable parse tree. Regrettable, but unavoidably true.
349 * If you're running with implicit_tags off (God help you!), consider
351 that $tree->content_list probably contains the tree or grove from the
352 parse, and not $tree itself (which will, oddly enough, be an implicit
353 'html' element). This seems counter-intuitive and problematic; but
354 seeing as how almost no HTML ever parses correctly with implicit_tags
355 off, this interface oddity seems the least of your problems.
356
358 When a document parses in a way different from how you think it should,
359 I ask that you report this to me as a bug. The first thing you should
360 do is copy the document, trim out as much of it as you can while still
361 producing the bug in question, and then email me that mini-document and
362 the code you're using to parse it, to the HTML::Tree bug queue at
363 "bug-html-tree at rt.cpan.org".
364 Include a note as to how it parses (presumably including its
366 $tree->dump output), and then a careful and clear explanation of where
367 you think the parser is going astray, and how you would prefer that it
368 work instead.
369
371 HTML::Tree; HTML::Parser, HTML::Element, HTML::Tagset
372 HTML::DOMbo
374
376 Copyright 1995-1998 Gisle Aas, 1999-2004 Sean M. Burke, 2005 Andy
377 Lester, 2006 Pete Krawczyk.
378 This library is free software; you can redistribute it and/or modify it
380 under the same terms as Perl itself.
381 This program is distributed in the hope that it will be useful, but
383 without any warranty; without even the implied warranty of mer‐
384 chantability or fitness for a particular purpose.
385
387 Currently maintained by Pete Krawczyk "<petek@cpan.org>"
388 Original authors: Gisle Aas, Sean Burke and Andy Lester.
390perl v5.8.8 2006-08-04 HTML::TreeBuilder(3)