1HTML::TreeBuilder(3) User Contributed Perl Documentation HTML::TreeBuilder(3)
2
6 HTML::TreeBuilder - Parser that builds a HTML syntax tree
7
9 This document describes version 5.03 of HTML::TreeBuilder, released
10 September 22, 2012 as part of HTML-Tree.
11
13 use HTML::TreeBuilder 5 -weak; # Ensure weak references in use
14 foreach my $file_name (@ARGV) {
16 my $tree = HTML::TreeBuilder->new; # empty tree
17 $tree->parse_file($file_name);
18 print "Hey, here's a dump of the parse tree of $file_name:\n";
19 $tree->dump; # a method we inherit from HTML::Element
20 print "And here it is, bizarrely rerendered as HTML:\n",
21 $tree->as_HTML, "\n";
22 # Now that we're done with it, we must destroy it.
24 # $tree = $tree->delete; # Not required with weak references
25 }
26
28 (This class is part of the HTML::Tree dist.)
29 This class is for HTML syntax trees that get built out of HTML source.
31 The way to use it is to:
32 1. start a new (empty) HTML::TreeBuilder object,
34 2. then use one of the methods from HTML::Parser (presumably with
36 "$tree->parse_file($filename)" for files, or with
37 "$tree->parse($document_content)" and "$tree->eof" if you've got the
38 content in a string) to parse the HTML document into the tree $tree.
39 (You can combine steps 1 and 2 with the "new_from_file" or
41 "new_from_content" methods.)
42 2b. call "$root->elementify()" if you want.
44 3. do whatever you need to do with the syntax tree, presumably
46 involving traversing it looking for some bit of information in it,
47 4. previous versions of HTML::TreeBuilder required you to call
49 "$tree->delete()" to erase the contents of the tree from memory when
50 you're done with the tree. This is not normally required anymore. See
51 "Weak References" in HTML::Element for details.
52
54 Most of the following attributes native to HTML::TreeBuilder control
55 how parsing takes place; they should be set before you try parsing into
56 the given object. You can set the attributes by passing a TRUE or
57 FALSE value as argument. E.g., "$root->implicit_tags" returns the
58 current setting for the "implicit_tags" option,
59 "$root->implicit_tags(1)" turns that option on, and
60 "$root->implicit_tags(0)" turns it off.
61 implicit_tags
63 Setting this attribute to true will instruct the parser to try to
64 deduce implicit elements and implicit end tags. If it is false you get
65 a parse tree that just reflects the text as it stands, which is
66 unlikely to be useful for anything but quick and dirty parsing. (In
67 fact, I'd be curious to hear from anyone who finds it useful to have
68 "implicit_tags" set to false.) Default is true.
69 Implicit elements have the "implicit" in HTML::Element attribute set.
71 implicit_body_p_tag
73 This controls an aspect of implicit element behavior, if
74 "implicit_tags" is on: If a text element (PCDATA) or a phrasal element
75 (such as "<em>") is to be inserted under "<body>", two things can
76 happen: if "implicit_body_p_tag" is true, it's placed under a new,
77 implicit "<p>" tag. (Past DTDs suggested this was the only correct
78 behavior, and this is how past versions of this module behaved.) But
79 if "implicit_body_p_tag" is false, nothing is implicated -- the PCDATA
80 or phrasal element is simply placed under "<body>". Default is false.
81 no_expand_entities
83 This attribute controls whether entities are decoded during the initial
84 parse of the source. Enable this if you don't want entities decoded to
85 their character value. e.g. '&' is decoded to '&' by default, but
86 will be unchanged if this is enabled. Default is false (entities will
87 be decoded.)
88 ignore_unknown
90 This attribute controls whether unknown tags should be represented as
91 elements in the parse tree, or whether they should be ignored. Default
92 is true (to ignore unknown tags.)
93 ignore_text
95 Do not represent the text content of elements. This saves space if all
96 you want is to examine the structure of the document. Default is
97 false.
98 ignore_ignorable_whitespace
100 If set to true, TreeBuilder will try to avoid creating ignorable
101 whitespace text nodes in the tree. Default is true. (In fact, I'd be
102 interested in hearing if there's ever a case where you need this off,
103 or where leaving it on leads to incorrect behavior.)
104 no_space_compacting
106 This determines whether TreeBuilder compacts all whitespace strings in
107 the document (well, outside of PRE or TEXTAREA elements), or leaves
108 them alone. Normally (default, value of 0), each string of contiguous
109 whitespace in the document is turned into a single space. But that's
110 not done if "no_space_compacting" is set to 1.
111 Setting "no_space_compacting" to 1 might be useful if you want to read
113 in a tree just to make some minor changes to it before writing it back
114 out.
115 This method is experimental. If you use it, be sure to report any
117 problems you might have with it.
118 p_strict
120 If set to true (and it defaults to false), TreeBuilder will take a
121 narrower than normal view of what can be under a "<p>" element; if it
122 sees a non-phrasal element about to be inserted under a "<p>", it will
123 close that "<p>". Otherwise it will close "<p>" elements only for
124 other "<p>"'s, headings, and "<form>" (although the latter may be
125 removed in future versions).
126 For example, when going thru this snippet of code,
128 <p>stuff
130 <ul>
131 TreeBuilder will normally (with "p_strict" false) put the "<ul>"
133 element under the "<p>" element. However, with "p_strict" set to true,
134 it will close the "<p>" first.
135 In theory, there should be strictness options like this for other/all
137 elements besides just "<p>"; but I treat this as a special case simply
138 because of the fact that "<p>" occurs so frequently and its end-tag is
139 omitted so often; and also because application of strictness rules at
140 parse-time across all elements often makes tiny errors in HTML coding
141 produce drastically bad parse-trees, in my experience.
142 If you find that you wish you had an option like this to enforce
144 content-models on all elements, then I suggest that what you want is
145 content-model checking as a stage after TreeBuilder has finished
146 parsing.
147 store_comments
149 This determines whether TreeBuilder will normally store comments found
150 while parsing content into $root. Currently, this is off by default.
151 store_declarations
153 This determines whether TreeBuilder will normally store markup
154 declarations found while parsing content into $root. This is on by
155 default.
156 store_pis
158 This determines whether TreeBuilder will normally store processing
159 instructions found while parsing content into $root -- assuming a
160 recent version of HTML::Parser (old versions won't parse PIs
161 correctly). Currently, this is off (false) by default.
162 It is somewhat of a known bug (to be fixed one of these days, if anyone
164 needs it?) that PIs in the preamble (before the "<html>" start-tag) end
165 up actually under the "<html>" element.
166 warn
168 This determines whether syntax errors during parsing should generate
169 warnings, emitted via Perl's "warn" function.
170 This is off (false) by default.
172
174 Objects of this class inherit the methods of both HTML::Parser and
175 HTML::Element. The methods inherited from HTML::Parser are used for
176 building the HTML tree, and the methods inherited from HTML::Element
177 are what you use to scrutinize the tree. Besides this
178 (HTML::TreeBuilder) documentation, you must also carefully read the
179 HTML::Element documentation, and also skim the HTML::Parser
180 documentation -- probably only its parse and parse_file methods are of
181 interest.
182 new_from_file
184 $root = HTML::TreeBuilder->new_from_file($filename_or_filehandle);
185 This "shortcut" constructor merely combines constructing a new object
187 (with the "new" method, below), and calling "$new->parse_file(...)" on
188 it. Returns the new object. Note that this provides no way of setting
189 any parse options like "store_comments" (for that, call "new", and then
190 set options, before calling "parse_file"). See the notes (below) on
191 parameters to "parse_file".
192 If HTML::TreeBuilder is unable to read the file, then "new_from_file"
194 dies. The error can also be found in $!. (This behavior is new in
195 HTML-Tree 5. Previous versions returned a tree with only implicit
196 elements.)
197 new_from_content
199 $root = HTML::TreeBuilder->new_from_content(...);
200 This "shortcut" constructor merely combines constructing a new object
202 (with the "new" method, below), and calling "for(...){$new->parse($_)}"
203 and "$new->eof" on it. Returns the new object. Note that this
204 provides no way of setting any parse options like "store_comments" (for
205 that, call "new", and then set options, before calling "parse").
206 Example usages: "HTML::TreeBuilder->new_from_content(@lines)", or
207 "HTML::TreeBuilder->new_from_content($content)".
208 new_from_url
210 $root = HTML::TreeBuilder->new_from_url($url)
211 This "shortcut" constructor combines constructing a new object (with
213 the "new" method, below), loading LWP::UserAgent, fetching the
214 specified URL, and calling "$new->parse( $response->decoded_content)"
215 and "$new->eof" on it. Returns the new object. Note that this
216 provides no way of setting any parse options like "store_comments".
217 If LWP is unable to fetch the URL, or the response is not HTML (as
219 determined by "content_is_html" in HTTP::Headers), then "new_from_url"
220 dies, and the HTTP::Response object is found in
221 $HTML::TreeBuilder::lwp_response.
222 You must have installed LWP::UserAgent for this method to work. LWP is
224 not installed automatically, because it's a large set of modules and
225 you might not need it.
226 new
228 $root = HTML::TreeBuilder->new();
229 This creates a new HTML::TreeBuilder object. This method takes no
231 attributes.
232 parse_file
234 $root->parse_file(...)
235 [An important method inherited from HTML::Parser, which see. Current
237 versions of HTML::Parser can take a filespec, or a filehandle object,
238 like *FOO, or some object from class IO::Handle, IO::File, IO::Socket)
239 or the like. I think you should check that a given file exists before
240 calling "$root->parse_file($filespec)".]
241 When you pass a filename to "parse_file", HTML::Parser opens it in
243 binary mode, which means it's interpreted as Latin-1 (ISO-8859-1). If
244 the file is in another encoding, like UTF-8 or UTF-16, this will not do
245 the right thing.
246 One solution is to open the file yourself using the proper ":encoding"
248 layer, and pass the filehandle to "parse_file". You can automate this
249 process by using "html_file" in IO::HTML, which will use the HTML5
250 encoding sniffing algorithm to automatically determine the proper
251 ":encoding" layer and apply it.
252 In the next major release of HTML-Tree, I plan to have it use IO::HTML
254 automatically. If you really want your file opened in binary mode, you
255 should open it yourself and pass the filehandle to "parse_file".
256 The return value is "undef" if there's an error opening the file. In
258 that case, the error will be in $!.
259 parse
261 $root->parse(...)
262 [A important method inherited from HTML::Parser, which see. See the
264 note below for "$root->eof()".]
265 eof
267 $root->eof();
268 This signals that you're finished parsing content into this tree; this
270 runs various kinds of crucial cleanup on the tree. This is called for
271 you when you call "$root->parse_file(...)", but not when you call
272 "$root->parse(...)". So if you call "$root->parse(...)", then you must
273 call "$root->eof()" once you've finished feeding all the chunks to
274 "parse(...)", and before you actually start doing anything else with
275 the tree in $root.
276 parse_content
278 $root->parse_content(...);
279 Basically a handy alias for "$root->parse(...); $root->eof". Takes the
281 exact same arguments as "$root->parse()".
282 delete
284 $root->delete();
285 [A previously important method inherited from HTML::Element, which
287 see.]
288 elementify
290 $root->elementify();
291 This changes the class of the object in $root from HTML::TreeBuilder to
293 the class used for all the rest of the elements in that tree (generally
294 HTML::Element). Returns $root.
295 For most purposes, this is unnecessary, but if you call this after
297 (after!!) you've finished building a tree, then it keeps you from
298 accidentally trying to call anything but HTML::Element methods on it.
299 (I.e., if you accidentally call "$root->parse_file(...)" on the
300 already-complete and elementified tree, then instead of charging ahead
301 and wreaking havoc, it'll throw a fatal error -- since $root is now an
302 object just of class HTML::Element which has no "parse_file" method.
303 Note that "elementify" currently deletes all the private attributes of
305 $root except for "_tag", "_parent", "_content", "_pos", and
306 "_implicit". If anyone requests that I change this to leave in yet
307 more private attributes, I might do so, in future versions.
308 guts
310 @nodes = $root->guts();
311 $parent_for_nodes = $root->guts();
312 In list context (as in the first case), this method returns the topmost
314 non-implicit nodes in a tree. This is useful when you're parsing HTML
315 code that you know doesn't expect an HTML document, but instead just a
316 fragment of an HTML document. For example, if you wanted the parse
317 tree for a file consisting of just this:
318 <li>I like pie!
320 Then you would get that with "@nodes = $root->guts();". It so happens
322 that in this case, @nodes will contain just one element object,
323 representing the "<li>" node (with "I like pie!" being its text child
324 node). However, consider if you were parsing this:
325 <hr>Hooboy!<hr>
327 In that case, "$root->guts()" would return three items: an element
329 object for the first "<hr>", a text string "Hooboy!", and another
330 "<hr>" element object.
331 For cases where you want definitely one element (so you can treat it as
333 a "document fragment", roughly speaking), call "guts()" in scalar
334 context, as in "$parent_for_nodes = $root->guts()". That works like
335 "guts()" in list context; in fact, "guts()" in list context would have
336 returned exactly one value, and if it would have been an object (as
337 opposed to a text string), then that's what "guts" in scalar context
338 will return. Otherwise, if "guts()" in list context would have
339 returned no values at all, then "guts()" in scalar context returns
340 undef. In all other cases, "guts()" in scalar context returns an
341 implicit "<div>" element node, with children consisting of whatever
342 nodes "guts()" in list context would have returned. Note that that may
343 detach those nodes from $root's tree.
344 disembowel
346 @nodes = $root->disembowel();
347 $parent_for_nodes = $root->disembowel();
348 The "disembowel()" method works just like the "guts()" method, except
350 that disembowel definitively destroys the tree above the nodes that are
351 returned. Usually when you want the guts from a tree, you're just
352 going to toss out the rest of the tree anyway, so this saves you the
353 bother. (Remember, "disembowel" means "remove the guts from".)
354
356 You should not need to call any of the following methods directly.
357 element_class
359 $classname = $h->element_class;
360 This method returns the class which will be used for new elements. It
362 defaults to HTML::Element, but can be overridden by subclassing or
363 esoteric means best left to those will will read the source and then
364 not complain when those esoteric means change. (Just subclass.)
365 comment
367 Accept a "here's a comment" signal from HTML::Parser.
368 declaration
370 Accept a "here's a markup declaration" signal from HTML::Parser.
371 done
373 TODO: document
374 end
376 Either: Acccept an end-tag signal from HTML::Parser Or: Method for
377 closing currently open elements in some fairly complex way, as used by
378 other methods in this class.
379 TODO: Why is this hidden?
381 process
383 Accept a "here's a PI" signal from HTML::Parser.
384 start
386 Accept a signal from HTML::Parser for start-tags.
387 TODO: Why is this hidden?
389 stunt
391 TODO: document
392 stunted
394 TODO: document
395 text
397 Accept a "here's a text token" signal from HTML::Parser.
398 TODO: Why is this hidden?
400 tighten_up
402 Legacy
403 Redirects to "delete_ignorable_whitespace" in HTML::Element.
405 warning
407 Wrapper for CORE::warn
408 TODO: why not just use carp?
410
412 DEBUG
413 Are we in Debug mode? This is a constant subroutine, to allow compile-
414 time optimizations. To control debug mode, set
415 $HTML::TreeBuilder::DEBUG before loading HTML::TreeBuilder.
416
418 HTML is rather harder to parse than people who write it generally
419 suspect.
420 Here's the problem: HTML is a kind of SGML that permits "minimization"
422 and "implication". In short, this means that you don't have to close
423 every tag you open (because the opening of a subsequent tag may
424 implicitly close it), and if you use a tag that can't occur in the
425 context you seem to using it in, under certain conditions the parser
426 will be able to realize you mean to leave the current context and enter
427 the new one, that being the only one that your code could correctly be
428 interpreted in.
429 Now, this would all work flawlessly and unproblematically if: 1) all
431 the rules that both prescribe and describe HTML were (and had been)
432 clearly set out, and 2) everyone was aware of these rules and wrote
433 their code in compliance to them.
434 However, it didn't happen that way, and so most HTML pages are
436 difficult if not impossible to correctly parse with nearly any set of
437 straightforward SGML rules. That's why the internals of
438 HTML::TreeBuilder consist of lots and lots of special cases -- instead
439 of being just a generic SGML parser with HTML DTD rules plugged in.
440
442 The techniques that HTML::TreeBuilder uses to perform what I consider
443 very robust parses on everyday code are not things that can work only
444 in Perl. To date, the algorithms at the center of HTML::TreeBuilder
445 have been implemented only in Perl, as far as I know; and I don't
446 foresee getting around to implementing them in any other language any
447 time soon.
448 If, however, anyone is looking for a semester project for an applied
450 programming class (or if they merely enjoy extra-curricular masochism),
451 they might do well to see about choosing as a topic the
452 implementation/adaptation of these routines to any other interesting
453 programming language that you feel currently suffers from a lack of
454 robust HTML-parsing. I welcome correspondence on this subject, and
455 point out that one can learn a great deal about languages by trying to
456 translate between them, and then comparing the result.
457 The HTML::TreeBuilder source may seem long and complex, but it is
459 rather well commented, and symbol names are generally self-explanatory.
460 (You are encouraged to read the Mozilla HTML parser source for
461 comparison.) Some of the complexity comes from little-used features,
462 and some of it comes from having the HTML tokenizer (HTML::Parser)
463 being a separate module, requiring somewhat of a different interface
464 than you'd find in a combined tokenizer and tree-builder. But most of
465 the length of the source comes from the fact that it's essentially a
466 long list of special cases, with lots and lots of sanity-checking, and
467 sanity-recovery -- because, as Roseanne Rosannadanna once said, "it's
468 always something".
469 Users looking to compare several HTML parsers should look at the source
471 for Raggett's Tidy ("<http://www.w3.org/People/Raggett/tidy/>"),
472 Mozilla ("<http://www.mozilla.org/>"), and possibly root around the
473 browsers section of Yahoo to find the various open-source ones
474 ("<http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Browsers/>").
475
477 * Framesets seem to work correctly now. Email me if you get a strange
478 parse from a document with framesets.
479 * Really bad HTML code will, often as not, make for a somewhat
481 objectionable parse tree. Regrettable, but unavoidably true.
482 * If you're running with "implicit_tags" off (God help you!), consider
484 that "$tree->content_list" probably contains the tree or grove from the
485 parse, and not $tree itself (which will, oddly enough, be an implicit
486 "<html>" element). This seems counter-intuitive and problematic; but
487 seeing as how almost no HTML ever parses correctly with "implicit_tags"
488 off, this interface oddity seems the least of your problems.
489
491 When a document parses in a way different from how you think it should,
492 I ask that you report this to me as a bug. The first thing you should
493 do is copy the document, trim out as much of it as you can while still
494 producing the bug in question, and then email me that mini-document and
495 the code you're using to parse it, to the HTML::Tree bug queue at
496 "<bug-html-tree at rt.cpan.org>".
497 Include a note as to how it parses (presumably including its
499 "$tree->dump" output), and then a careful and clear explanation of
500 where you think the parser is going astray, and how you would prefer
501 that it work instead.
502
504 For more information about the HTML-Tree distribution: HTML::Tree.
505 Modules used by HTML::TreeBuilder: HTML::Parser, HTML::Element,
507 HTML::Tagset.
508 For converting between XML::DOM::Node, HTML::Element, and XML::Element
510 trees: HTML::DOMbo.
511 For opening a HTML file with automatic charset detection: IO::HTML.
513
515 Current maintainers:
516 · Christopher J. Madsen "<perl AT cjmweb.net>"
518 · Jeff Fearn "<jfearn AT cpan.org>"
520 Original HTML-Tree author:
522 · Gisle Aas
524 Former maintainers:
526 · Sean M. Burke
528 · Andy Lester
530 · Pete Krawczyk "<petek AT cpan.org>"
532 You can follow or contribute to HTML-Tree's development at
534 <http://github.com/madsen/HTML-Tree>.
535
537 Copyright 1995-1998 Gisle Aas, 1999-2004 Sean M. Burke, 2005 Andy
538 Lester, 2006 Pete Krawczyk, 2010 Jeff Fearn, 2012 Christopher J.
539 Madsen.
540 This library is free software; you can redistribute it and/or modify it
542 under the same terms as Perl itself.
543 The programs in this library are distributed in the hope that they will
545 be useful, but without any warranty; without even the implied warranty
546 of merchantability or fitness for a particular purpose.
547perl v5.16.3 2014-06-10 HTML::TreeBuilder(3)