1HTML::TreeBuilder(3)  User Contributed Perl Documentation HTML::TreeBuilder(3)
2
3
4

NAME

6       HTML::TreeBuilder - Parser that builds a HTML syntax tree
7

VERSION

9       This document describes version 5.07 of HTML::TreeBuilder, released
10       August 31, 2017 as part of HTML-Tree.
11

SYNOPSIS

13         use HTML::TreeBuilder 5 -weak; # Ensure weak references in use
14
15         foreach my $file_name (@ARGV) {
16           my $tree = HTML::TreeBuilder->new; # empty tree
17           $tree->parse_file($file_name);
18           print "Hey, here's a dump of the parse tree of $file_name:\n";
19           $tree->dump; # a method we inherit from HTML::Element
20           print "And here it is, bizarrely rerendered as HTML:\n",
21             $tree->as_HTML, "\n";
22
23           # Now that we're done with it, we must destroy it.
24           # $tree = $tree->delete; # Not required with weak references
25         }
26

DESCRIPTION

28       (This class is part of the HTML::Tree dist.)
29
30       This class is for HTML syntax trees that get built out of HTML source.
31       The way to use it is to:
32
33       1. start a new (empty) HTML::TreeBuilder object,
34
35       2. then use one of the methods from HTML::Parser (presumably with
36       "$tree->parse_file($filename)" for files, or with
37       "$tree->parse($document_content)" and "$tree->eof" if you've got the
38       content in a string) to parse the HTML document into the tree $tree.
39
40       (You can combine steps 1 and 2 with the "new_from_file" or
41       "new_from_content" methods.)
42
43       2b. call "$root->elementify()" if you want.
44
45       3. do whatever you need to do with the syntax tree, presumably
46       involving traversing it looking for some bit of information in it,
47
48       4. previous versions of HTML::TreeBuilder required you to call
49       "$tree->delete()" to erase the contents of the tree from memory when
50       you're done with the tree.  This is not normally required anymore.  See
51       "Weak References" in HTML::Element for details.
52

ATTRIBUTES

54       Most of the following attributes native to HTML::TreeBuilder control
55       how parsing takes place; they should be set before you try parsing into
56       the given object.  You can set the attributes by passing a TRUE or
57       FALSE value as argument.  E.g., "$root->implicit_tags" returns the
58       current setting for the "implicit_tags" option,
59       "$root->implicit_tags(1)" turns that option on, and
60       "$root->implicit_tags(0)" turns it off.
61
62   implicit_tags
63       Setting this attribute to true will instruct the parser to try to
64       deduce implicit elements and implicit end tags.  If it is false you get
65       a parse tree that just reflects the text as it stands, which is
66       unlikely to be useful for anything but quick and dirty parsing.  (In
67       fact, I'd be curious to hear from anyone who finds it useful to have
68       "implicit_tags" set to false.)  Default is true.
69
70       Implicit elements have the "implicit" in HTML::Element attribute set.
71
72   implicit_body_p_tag
73       This controls an aspect of implicit element behavior, if
74       "implicit_tags" is on:  If a text element (PCDATA) or a phrasal element
75       (such as "<em>") is to be inserted under "<body>", two things can
76       happen: if "implicit_body_p_tag" is true, it's placed under a new,
77       implicit "<p>" tag.  (Past DTDs suggested this was the only correct
78       behavior, and this is how past versions of this module behaved.)  But
79       if "implicit_body_p_tag" is false, nothing is implicated -- the PCDATA
80       or phrasal element is simply placed under "<body>".  Default is false.
81
82   no_expand_entities
83       This attribute controls whether entities are decoded during the initial
84       parse of the source. Enable this if you don't want entities decoded to
85       their character value. e.g. '&amp;' is decoded to '&' by default, but
86       will be unchanged if this is enabled.  Default is false (entities will
87       be decoded.)
88
89   ignore_unknown
90       This attribute controls whether unknown tags should be represented as
91       elements in the parse tree, or whether they should be ignored.  Default
92       is true (to ignore unknown tags.)
93
94   ignore_text
95       Do not represent the text content of elements.  This saves space if all
96       you want is to examine the structure of the document.  Default is
97       false.
98
99   ignore_ignorable_whitespace
100       If set to true, TreeBuilder will try to avoid creating ignorable
101       whitespace text nodes in the tree.  Default is true.  (In fact, I'd be
102       interested in hearing if there's ever a case where you need this off,
103       or where leaving it on leads to incorrect behavior.)
104
105   no_space_compacting
106       This determines whether TreeBuilder compacts all whitespace strings in
107       the document (well, outside of PRE or TEXTAREA elements), or leaves
108       them alone.  Normally (default, value of 0), each string of contiguous
109       whitespace in the document is turned into a single space.  But that's
110       not done if "no_space_compacting" is set to 1.
111
112       Setting "no_space_compacting" to 1 might be useful if you want to read
113       in a tree just to make some minor changes to it before writing it back
114       out.
115
116       This method is experimental.  If you use it, be sure to report any
117       problems you might have with it.
118
119   p_strict
120       If set to true (and it defaults to false), TreeBuilder will take a
121       narrower than normal view of what can be under a "<p>" element; if it
122       sees a non-phrasal element about to be inserted under a "<p>", it will
123       close that "<p>".  Otherwise it will close "<p>" elements only for
124       other "<p>"'s, headings, and "<form>" (although the latter may be
125       removed in future versions).
126
127       For example, when going thru this snippet of code,
128
129         <p>stuff
130         <ul>
131
132       TreeBuilder will normally (with "p_strict" false) put the "<ul>"
133       element under the "<p>" element.  However, with "p_strict" set to true,
134       it will close the "<p>" first.
135
136       In theory, there should be strictness options like this for other/all
137       elements besides just "<p>"; but I treat this as a special case simply
138       because of the fact that "<p>" occurs so frequently and its end-tag is
139       omitted so often; and also because application of strictness rules at
140       parse-time across all elements often makes tiny errors in HTML coding
141       produce drastically bad parse-trees, in my experience.
142
143       If you find that you wish you had an option like this to enforce
144       content-models on all elements, then I suggest that what you want is
145       content-model checking as a stage after TreeBuilder has finished
146       parsing.
147
148   store_comments
149       This determines whether TreeBuilder will normally store comments found
150       while parsing content into $root.  Currently, this is off by default.
151
152   store_declarations
153       This determines whether TreeBuilder will normally store markup
154       declarations found while parsing content into $root.  This is on by
155       default.
156
157   store_pis
158       This determines whether TreeBuilder will normally store processing
159       instructions found while parsing content into $root -- assuming a
160       recent version of HTML::Parser (old versions won't parse PIs
161       correctly).  Currently, this is off (false) by default.
162
163       It is somewhat of a known bug (to be fixed one of these days, if anyone
164       needs it?) that PIs in the preamble (before the "<html>" start-tag) end
165       up actually under the "<html>" element.
166
167   warn
168       This determines whether syntax errors during parsing should generate
169       warnings, emitted via Perl's "warn" function.
170
171       This is off (false) by default.
172

METHODS

174       Objects of this class inherit the methods of both HTML::Parser and
175       HTML::Element.  The methods inherited from HTML::Parser are used for
176       building the HTML tree, and the methods inherited from HTML::Element
177       are what you use to scrutinize the tree.  Besides this
178       (HTML::TreeBuilder) documentation, you must also carefully read the
179       HTML::Element documentation, and also skim the HTML::Parser
180       documentation -- probably only its parse and parse_file methods are of
181       interest.
182
183   new_from_file
184         $root = HTML::TreeBuilder->new_from_file($filename_or_filehandle);
185
186       This "shortcut" constructor merely combines constructing a new object
187       (with the "new" method, below), and calling "$new->parse_file(...)" on
188       it.  Returns the new object.  Note that this provides no way of setting
189       any parse options like "store_comments" (for that, call "new", and then
190       set options, before calling "parse_file").  See the notes (below) on
191       parameters to "parse_file".
192
193       If HTML::TreeBuilder is unable to read the file, then "new_from_file"
194       dies.  The error can also be found in $!.  (This behavior is new in
195       HTML-Tree 5. Previous versions returned a tree with only implicit
196       elements.)
197
198   new_from_content
199         $root = HTML::TreeBuilder->new_from_content(...);
200
201       This "shortcut" constructor merely combines constructing a new object
202       (with the "new" method, below), and calling "for(...){$new->parse($_)}"
203       and "$new->eof" on it.  Returns the new object.  Note that this
204       provides no way of setting any parse options like "store_comments" (for
205       that, call "new", and then set options, before calling "parse").
206       Example usages: "HTML::TreeBuilder->new_from_content(@lines)", or
207       "HTML::TreeBuilder->new_from_content($content)".
208
209   new_from_url
210         $root = HTML::TreeBuilder->new_from_url($url)
211
212       This "shortcut" constructor combines constructing a new object (with
213       the "new" method, below), loading LWP::UserAgent, fetching the
214       specified URL, and calling "$new->parse( $response->decoded_content)"
215       and "$new->eof" on it.  Returns the new object.  Note that this
216       provides no way of setting any parse options like "store_comments".
217
218       If LWP is unable to fetch the URL, or the response is not HTML (as
219       determined by "content_is_html" in HTTP::Headers), then "new_from_url"
220       dies, and the HTTP::Response object is found in
221       $HTML::TreeBuilder::lwp_response.
222
223       You must have installed LWP::UserAgent for this method to work.  LWP is
224       not installed automatically, because it's a large set of modules and
225       you might not need it.
226
227   new
228         $root = HTML::TreeBuilder->new();
229
230       This creates a new HTML::TreeBuilder object.  This method takes no
231       attributes.
232
233   parse_file
234        $root->parse_file(...)
235
236       [An important method inherited from HTML::Parser, which see.  Current
237       versions of HTML::Parser can take a filespec, or a filehandle object,
238       like *FOO, or some object from class IO::Handle, IO::File, IO::Socket)
239       or the like.  I think you should check that a given file exists before
240       calling "$root->parse_file($filespec)".]
241
242       When you pass a filename to "parse_file", HTML::Parser opens it in
243       binary mode, which means it's interpreted as Latin-1 (ISO-8859-1).  If
244       the file is in another encoding, like UTF-8 or UTF-16, this will not do
245       the right thing.
246
247       One solution is to open the file yourself using the proper ":encoding"
248       layer, and pass the filehandle to "parse_file".  You can automate this
249       process by using "html_file" in IO::HTML, which will use the HTML5
250       encoding sniffing algorithm to automatically determine the proper
251       ":encoding" layer and apply it.
252
253       In the next major release of HTML-Tree, I plan to have it use IO::HTML
254       automatically.  If you really want your file opened in binary mode, you
255       should open it yourself and pass the filehandle to "parse_file".
256
257       The return value is "undef" if there's an error opening the file.  In
258       that case, the error will be in $!.
259
260   parse
261         $root->parse(...)
262
263       [A important method inherited from HTML::Parser, which see.  See the
264       note below for "$root->eof()".]
265
266   eof
267         $root->eof();
268
269       This signals that you're finished parsing content into this tree; this
270       runs various kinds of crucial cleanup on the tree.  This is called for
271       you when you call "$root->parse_file(...)", but not when you call
272       "$root->parse(...)".  So if you call "$root->parse(...)", then you must
273       call "$root->eof()" once you've finished feeding all the chunks to
274       "parse(...)", and before you actually start doing anything else with
275       the tree in $root.
276
277   parse_content
278         $root->parse_content(...);
279
280       Basically a handy alias for "$root->parse(...); $root->eof".  Takes the
281       exact same arguments as "$root->parse()".
282
283   delete
284         $root->delete();
285
286       [A previously important method inherited from HTML::Element, which
287       see.]
288
289   elementify
290         $root->elementify();
291
292       This changes the class of the object in $root from HTML::TreeBuilder to
293       the class used for all the rest of the elements in that tree (generally
294       HTML::Element).  Returns $root.
295
296       For most purposes, this is unnecessary, but if you call this after
297       (after!!)  you've finished building a tree, then it keeps you from
298       accidentally trying to call anything but HTML::Element methods on it.
299       (I.e., if you accidentally call "$root->parse_file(...)" on the
300       already-complete and elementified tree, then instead of charging ahead
301       and wreaking havoc, it'll throw a fatal error -- since $root is now an
302       object just of class HTML::Element which has no "parse_file" method.
303
304       Note that "elementify" currently deletes all the private attributes of
305       $root except for "_tag", "_parent", "_content", "_pos", and
306       "_implicit".  If anyone requests that I change this to leave in yet
307       more private attributes, I might do so, in future versions.
308
309   guts
310        @nodes = $root->guts();
311        $parent_for_nodes = $root->guts();
312
313       In list context (as in the first case), this method returns the topmost
314       non-implicit nodes in a tree.  This is useful when you're parsing HTML
315       code that you know doesn't expect an HTML document, but instead just a
316       fragment of an HTML document.  For example, if you wanted the parse
317       tree for a file consisting of just this:
318
319         <li>I like pie!
320
321       Then you would get that with "@nodes = $root->guts();".  It so happens
322       that in this case, @nodes will contain just one element object,
323       representing the "<li>" node (with "I like pie!" being its text child
324       node).  However, consider if you were parsing this:
325
326         <hr>Hooboy!<hr>
327
328       In that case, "$root->guts()" would return three items: an element
329       object for the first "<hr>", a text string "Hooboy!", and another
330       "<hr>" element object.
331
332       For cases where you want definitely one element (so you can treat it as
333       a "document fragment", roughly speaking), call "guts()" in scalar
334       context, as in "$parent_for_nodes = $root->guts()". That works like
335       "guts()" in list context; in fact, "guts()" in list context would have
336       returned exactly one value, and if it would have been an object (as
337       opposed to a text string), then that's what "guts" in scalar context
338       will return.  Otherwise, if "guts()" in list context would have
339       returned no values at all, then "guts()" in scalar context returns
340       undef.  In all other cases, "guts()" in scalar context returns an
341       implicit "<div>" element node, with children consisting of whatever
342       nodes "guts()" in list context would have returned.  Note that that may
343       detach those nodes from $root's tree.
344
345   disembowel
346         @nodes = $root->disembowel();
347         $parent_for_nodes = $root->disembowel();
348
349       The "disembowel()" method works just like the "guts()" method, except
350       that disembowel definitively destroys the tree above the nodes that are
351       returned.  Usually when you want the guts from a tree, you're just
352       going to toss out the rest of the tree anyway, so this saves you the
353       bother.  (Remember, "disembowel" means "remove the guts from".)
354

INTERNAL METHODS

356       You should not need to call any of the following methods directly.
357
358   element_class
359         $classname = $h->element_class;
360
361       This method returns the class which will be used for new elements.  It
362       defaults to HTML::Element, but can be overridden by subclassing or
363       esoteric means best left to those will will read the source and then
364       not complain when those esoteric means change.  (Just subclass.)
365
366   comment
367       Accept a "here's a comment" signal from HTML::Parser.
368
369   declaration
370       Accept a "here's a markup declaration" signal from HTML::Parser.
371
372   done
373       TODO: document
374
375   end
376       Either: Acccept an end-tag signal from HTML::Parser Or: Method for
377       closing currently open elements in some fairly complex way, as used by
378       other methods in this class.
379
380       TODO: Why is this hidden?
381
382   process
383       Accept a "here's a PI" signal from HTML::Parser.
384
385   start
386       Accept a signal from HTML::Parser for start-tags.
387
388       TODO: Why is this hidden?
389
390   stunt
391       TODO: document
392
393   stunted
394       TODO: document
395
396   text
397       Accept a "here's a text token" signal from HTML::Parser.
398
399       TODO: Why is this hidden?
400
401   tighten_up
402       Legacy
403
404       Redirects to "delete_ignorable_whitespace" in HTML::Element.
405
406   warning
407       Wrapper for CORE::warn
408
409       TODO: why not just use carp?
410

SUBROUTINES

412   DEBUG
413       Are we in Debug mode?  This is a constant subroutine, to allow compile-
414       time optimizations.  To control debug mode, set
415       $HTML::TreeBuilder::DEBUG before loading HTML::TreeBuilder.
416

HTML AND ITS DISCONTENTS

418       HTML is rather harder to parse than people who write it generally
419       suspect.
420
421       Here's the problem: HTML is a kind of SGML that permits "minimization"
422       and "implication".  In short, this means that you don't have to close
423       every tag you open (because the opening of a subsequent tag may
424       implicitly close it), and if you use a tag that can't occur in the
425       context you seem to using it in, under certain conditions the parser
426       will be able to realize you mean to leave the current context and enter
427       the new one, that being the only one that your code could correctly be
428       interpreted in.
429
430       Now, this would all work flawlessly and unproblematically if: 1) all
431       the rules that both prescribe and describe HTML were (and had been)
432       clearly set out, and 2) everyone was aware of these rules and wrote
433       their code in compliance to them.
434
435       However, it didn't happen that way, and so most HTML pages are
436       difficult if not impossible to correctly parse with nearly any set of
437       straightforward SGML rules.  That's why the internals of
438       HTML::TreeBuilder consist of lots and lots of special cases -- instead
439       of being just a generic SGML parser with HTML DTD rules plugged in.
440

TRANSLATIONS?

442       The techniques that HTML::TreeBuilder uses to perform what I consider
443       very robust parses on everyday code are not things that can work only
444       in Perl.  To date, the algorithms at the center of HTML::TreeBuilder
445       have been implemented only in Perl, as far as I know; and I don't
446       foresee getting around to implementing them in any other language any
447       time soon.
448
449       If, however, anyone is looking for a semester project for an applied
450       programming class (or if they merely enjoy extra-curricular masochism),
451       they might do well to see about choosing as a topic the
452       implementation/adaptation of these routines to any other interesting
453       programming language that you feel currently suffers from a lack of
454       robust HTML-parsing.  I welcome correspondence on this subject, and
455       point out that one can learn a great deal about languages by trying to
456       translate between them, and then comparing the result.
457
458       The HTML::TreeBuilder source may seem long and complex, but it is
459       rather well commented, and symbol names are generally self-explanatory.
460       (You are encouraged to read the Mozilla HTML parser source for
461       comparison.)  Some of the complexity comes from little-used features,
462       and some of it comes from having the HTML tokenizer (HTML::Parser)
463       being a separate module, requiring somewhat of a different interface
464       than you'd find in a combined tokenizer and tree-builder.  But most of
465       the length of the source comes from the fact that it's essentially a
466       long list of special cases, with lots and lots of sanity-checking, and
467       sanity-recovery -- because, as Roseanne Rosannadanna once said, "it's
468       always something".
469
470       Users looking to compare several HTML parsers should look at the source
471       for Raggett's Tidy ("<http://www.w3.org/People/Raggett/tidy/>"),
472       Mozilla ("<http://www.mozilla.org/>"), and possibly root around the
473       browsers section of Yahoo to find the various open-source ones
474       ("<http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Browsers/>").
475

BUGS

477       * Framesets seem to work correctly now.  Email me if you get a strange
478       parse from a document with framesets.
479
480       * Really bad HTML code will, often as not, make for a somewhat
481       objectionable parse tree.  Regrettable, but unavoidably true.
482
483       * If you're running with "implicit_tags" off (God help you!), consider
484       that "$tree->content_list" probably contains the tree or grove from the
485       parse, and not $tree itself (which will, oddly enough, be an implicit
486       "<html>" element).  This seems counter-intuitive and problematic; but
487       seeing as how almost no HTML ever parses correctly with "implicit_tags"
488       off, this interface oddity seems the least of your problems.
489

BUG REPORTS

491       When a document parses in a way different from how you think it should,
492       I ask that you report this to me as a bug.  The first thing you should
493       do is copy the document, trim out as much of it as you can while still
494       producing the bug in question, and then email me that mini-document and
495       the code you're using to parse it, to the HTML::Tree bug queue at
496       "<bug-html-tree at rt.cpan.org>".
497
498       Include a note as to how it parses (presumably including its
499       "$tree->dump" output), and then a careful and clear explanation of
500       where you think the parser is going astray, and how you would prefer
501       that it work instead.
502

SEE ALSO

504       For more information about the HTML-Tree distribution: HTML::Tree.
505
506       Modules used by HTML::TreeBuilder: HTML::Parser, HTML::Element,
507       HTML::Tagset.
508
509       For converting between XML::DOM::Node, HTML::Element, and XML::Element
510       trees: HTML::DOMbo.
511
512       For opening a HTML file with automatic charset detection: IO::HTML.
513

AUTHOR

515       Current maintainers:
516
517       ·   Christopher J. Madsen "<perl AT cjmweb.net>"
518
519       ·   Jeff Fearn "<jfearn AT cpan.org>"
520
521       Original HTML-Tree author:
522
523       ·   Gisle Aas
524
525       Former maintainers:
526
527       ·   Sean M. Burke
528
529       ·   Andy Lester
530
531       ·   Pete Krawczyk "<petek AT cpan.org>"
532
533       You can follow or contribute to HTML-Tree's development at
534       <https://github.com/kentfredric/HTML-Tree>.
535
537       Copyright 1995-1998 Gisle Aas, 1999-2004 Sean M. Burke, 2005 Andy
538       Lester, 2006 Pete Krawczyk, 2010 Jeff Fearn, 2012 Christopher J.
539       Madsen.
540
541       This library is free software; you can redistribute it and/or modify it
542       under the same terms as Perl itself.
543
544       The programs in this library are distributed in the hope that they will
545       be useful, but without any warranty; without even the implied warranty
546       of merchantability or fitness for a particular purpose.
547
548
549
550perl v5.30.1                      2020-01-30              HTML::TreeBuilder(3)
Impressum