1HTML::TokeParser(3)   User Contributed Perl Documentation  HTML::TokeParser(3)
2
3
4

NAME

6       HTML::TokeParser - Alternative HTML::Parser interface
7

SYNOPSIS

9        require HTML::TokeParser;
10        $p = HTML::TokeParser->new("index.html") ||
11             die "Can't open: $!";
12        $p->empty_element_tags(1);  # configure its behaviour
13
14        while (my $token = $p->get_token) {
15            #...
16        }
17

DESCRIPTION

19       The "HTML::TokeParser" is an alternative interface to the
20       "HTML::Parser" class.  It is an "HTML::PullParser" subclass with a
21       predeclared set of token types.  If you wish the tokens to be reported
22       differently you probably want to use the "HTML::PullParser" directly.
23
24       The following methods are available:
25
26       $p = HTML::TokeParser->new( $filename, %opt );
27       $p = HTML::TokeParser->new( $filehandle, %opt );
28       $p = HTML::TokeParser->new( \$document, %opt );
29           The object constructor argument is either a file name, a file
30           handle object, or the complete document to be parsed.  Extra
31           options can be provided as key/value pairs and are processed as
32           documented by the base classes.
33
34           If the argument is a plain scalar, then it is taken as the name of
35           a file to be opened and parsed.  If the file can't be opened for
36           reading, then the constructor will return "undef" and $! will tell
37           you why it failed.
38
39           If the argument is a reference to a plain scalar, then this scalar
40           is taken to be the literal document to parse.  The value of this
41           scalar should not be changed before all tokens have been extracted.
42
43           Otherwise the argument is taken to be some object that the
44           "HTML::TokeParser" can read() from when it needs more data.
45           Typically it will be a filehandle of some kind.  The stream will be
46           read() until EOF, but not closed.
47
48           A newly constructed "HTML::TokeParser" differ from its base classes
49           by having the "unbroken_text" attribute enabled by default. See
50           HTML::Parser for a description of this and other attributes that
51           influence how the document is parsed. It is often a good idea to
52           enable "empty_element_tags" behaviour.
53
54           Note that the parsing result will likely not be valid if raw
55           undecoded UTF-8 is used as a source.  When parsing UTF-8 encoded
56           files turn on UTF-8 decoding:
57
58              open(my $fh, "<:utf8", "index.html") || die "Can't open 'index.html': $!";
59              my $p = HTML::TokeParser->new( $fh );
60              # ...
61
62           If a $filename is passed to the constructor the file will be opened
63           in raw mode and the parsing result will only be valid if its
64           content is Latin-1 or pure ASCII.
65
66           If parsing from an UTF-8 encoded string buffer decode it first:
67
68              utf8::decode($document);
69              my $p = HTML::TokeParser->new( \$document );
70              # ...
71
72       $p->get_token
73           This method will return the next token found in the HTML document,
74           or "undef" at the end of the document.  The token is returned as an
75           array reference.  The first element of the array will be a string
76           denoting the type of this token: "S" for start tag, "E" for end
77           tag, "T" for text, "C" for comment, "D" for declaration, and "PI"
78           for process instructions.  The rest of the token array depend on
79           the type like this:
80
81             ["S",  $tag, $attr, $attrseq, $text]
82             ["E",  $tag, $text]
83             ["T",  $text, $is_data]
84             ["C",  $text]
85             ["D",  $text]
86             ["PI", $token0, $text]
87
88           where $attr is a hash reference, $attrseq is an array reference and
89           the rest are plain scalars.  The "Argspec" in HTML::Parser explains
90           the details.
91
92       $p->unget_token( @tokens )
93           If you find you have read too many tokens you can push them back,
94           so that they are returned the next time $p->get_token is called.
95
96       $p->get_tag
97       $p->get_tag( @tags )
98           This method returns the next start or end tag (skipping any other
99           tokens), or "undef" if there are no more tags in the document.  If
100           one or more arguments are given, then we skip tokens until one of
101           the specified tag types is found.  For example:
102
103              $p->get_tag("font", "/font");
104
105           will find the next start or end tag for a font-element.
106
107           The tag information is returned as an array reference in the same
108           form as for $p->get_token above, but the type code (first element)
109           is missing. A start tag will be returned like this:
110
111             [$tag, $attr, $attrseq, $text]
112
113           The tagname of end tags are prefixed with "/", i.e. end tag is
114           returned like this:
115
116             ["/$tag", $text]
117
118       $p->get_text
119       $p->get_text( @endtags )
120           This method returns all text found at the current position. It will
121           return a zero length string if the next token is not text. Any
122           entities will be converted to their corresponding character.
123
124           If one or more arguments are given, then we return all text
125           occurring before the first of the specified tags found. For
126           example:
127
128              $p->get_text("p", "br");
129
130           will return the text up to either a paragraph of line break
131           element.
132
133           The text might span tags that should be textified.  This is
134           controlled by the $p->{textify} attribute, which is a hash that
135           defines how certain tags can be treated as text.  If the name of a
136           start tag matches a key in this hash then this tag is converted to
137           text.  The hash value is used to specify which tag attribute to
138           obtain the text from.  If this tag attribute is missing, then the
139           upper case name of the tag enclosed in brackets is returned, e.g.
140           "[IMG]".  The hash value can also be a subroutine reference.  In
141           this case the routine is called with the start tag token content as
142           its argument and the return value is treated as the text.
143
144           The default $p->{textify} value is:
145
146             {img => "alt", applet => "alt"}
147
148           This means that <IMG> and <APPLET> tags are treated as text, and
149           that the text to substitute can be found in the ALT attribute.
150
151       $p->get_trimmed_text
152       $p->get_trimmed_text( @endtags )
153           Same as $p->get_text above, but will collapse any sequences of
154           white space to a single space character.  Leading and trailing
155           white space is removed.
156
157       $p->get_phrase
158           This will return all text found at the current position ignoring
159           any phrasal-level tags.  Text is extracted until the first non
160           phrasal-level tag.  Textification of tags is the same as for
161           get_text().  This method will collapse white space in the same way
162           as get_trimmed_text() does.
163
164           The definition of <i>phrasal-level tags</i> is obtained from the
165           HTML::Tagset module.
166

EXAMPLES

168       This example extracts all links from a document.  It will print one
169       line for each link, containing the URL and the textual description
170       between the <A>...</A> tags:
171
172         use HTML::TokeParser;
173         $p = HTML::TokeParser->new(shift||"index.html");
174
175         while (my $token = $p->get_tag("a")) {
176             my $url = $token->[1]{href} || "-";
177             my $text = $p->get_trimmed_text("/a");
178             print "$url\t$text\n";
179         }
180
181       This example extract the <TITLE> from the document:
182
183         use HTML::TokeParser;
184         $p = HTML::TokeParser->new(shift||"index.html");
185         if ($p->get_tag("title")) {
186             my $title = $p->get_trimmed_text;
187             print "Title: $title\n";
188         }
189

SEE ALSO

191       HTML::PullParser, HTML::Parser
192
194       Copyright 1998-2005 Gisle Aas.
195
196       This library is free software; you can redistribute it and/or modify it
197       under the same terms as Perl itself.
198
199
200
201perl v5.36.0                      2023-02-01               HTML::TokeParser(3)
Impressum