1HTML::TokeParser(3) User Contributed Perl Documentation HTML::TokeParser(3)
2
6 HTML::TokeParser - Alternative HTML::Parser interface
7
9 require HTML::TokeParser;
10 $p = HTML::TokeParser->new("index.html") ⎪⎪
11 die "Can't open: $!";
12 $p->empty_element_tags(1); # configure its behaviour
13 while (my $token = $p->get_token) {
15 #...
16 }
17
19 The "HTML::TokeParser" is an alternative interface to the
20 "HTML::Parser" class. It is an "HTML::PullParser" subclass with a pre‐
21 declared set of token types. If you wish the tokens to be reported
22 differently you probably want to use the "HTML::PullParser" directly.
23 The following methods are available:
25 $p = HTML::TokeParser->new( $filename, %opt );
27 $p = HTML::TokeParser->new( $filehandle, %opt );
28 $p = HTML::TokeParser->new( \$document, %opt );
29 The object constructor argument is either a file name, a file han‐
30 dle object, or the complete document to be parsed. Extra options
31 can be provided as key/value pairs and are processed as documented
32 by the base classes.
33 If the argument is a plain scalar, then it is taken as the name of
35 a file to be opened and parsed. If the file can't be opened for
36 reading, then the constructor will return "undef" and $! will tell
37 you why it failed.
38 If the argument is a reference to a plain scalar, then this scalar
40 is taken to be the literal document to parse. The value of this
41 scalar should not be changed before all tokens have been extracted.
42 Otherwise the argument is taken to be some object that the
44 "HTML::TokeParser" can read() from when it needs more data. Typi‐
45 cally it will be a filehandle of some kind. The stream will be
46 read() until EOF, but not closed.
47 A newly constructed "HTML::TokeParser" differ from its base classes
49 by having the "unbroken_text" attribute enabled by default. See
50 HTML::Parser for a description of this and other attributes that
51 influence how the document is parsed. It is often a good idea to
52 enable "empty_element_tags" behaviour.
53 Note that the parsing result will likely not be valid if raw unde‐
55 coded UTF-8 is used as a source. When parsing UTF-8 encoded files
56 turn on UTF-8 decoding:
57 open(my $fh, "<:utf8", "index.html") ⎪⎪ die "Can't open 'index.html': $!";
59 my $p = HTML::TokeParser->new( $fh );
60 # ...
61 If a $filename is passed to the constructor the file will be opened
63 in raw mode and the parsing result will only be valid if its con‐
64 tent is Latin-1 or pure ASCII.
65 If parsing from an UTF-8 encoded string buffer decode it first:
67 utf8::decode($document);
69 my $p = HTML::TokeParser->new( \$document );
70 # ...
71 $p->get_token
73 This method will return the next token found in the HTML document,
74 or "undef" at the end of the document. The token is returned as an
75 array reference. The first element of the array will be a string
76 denoting the type of this token: "S" for start tag, "E" for end
77 tag, "T" for text, "C" for comment, "D" for declaration, and "PI"
78 for process instructions. The rest of the token array depend on
79 the type like this:
80 ["S", $tag, $attr, $attrseq, $text]
82 ["E", $tag, $text]
83 ["T", $text, $is_data]
84 ["C", $text]
85 ["D", $text]
86 ["PI", $token0, $text]
87 where $attr is a hash reference, $attrseq is an array reference and
89 the rest are plain scalars. The "Argspec" in HTML::Parser explains
90 the details.
91 $p->unget_token( @tokens )
93 If you find you have read too many tokens you can push them back,
94 so that they are returned the next time $p->get_token is called.
95 $p->get_tag
97 $p->get_tag( @tags )
98 This method returns the next start or end tag (skipping any other
99 tokens), or "undef" if there are no more tags in the document. If
100 one or more arguments are given, then we skip tokens until one of
101 the specified tag types is found. For example:
102 $p->get_tag("font", "/font");
104 will find the next start or end tag for a font-element.
106 The tag information is returned as an array reference in the same
108 form as for $p->get_token above, but the type code (first element)
109 is missing. A start tag will be returned like this:
110 [$tag, $attr, $attrseq, $text]
112 The tagname of end tags are prefixed with "/", i.e. end tag is
114 returned like this:
115 ["/$tag", $text]
117 $p->get_text
119 $p->get_text( @endtags )
120 This method returns all text found at the current position. It will
121 return a zero length string if the next token is not text. Any
122 entities will be converted to their corresponding character.
123 If one or more arguments are given, then we return all text occur‐
125 ring before the first of the specified tags found. For example:
126 $p->get_text("p", "br");
128 will return the text up to either a paragraph of linebreak element.
130 The text might span tags that should be textified. This is con‐
132 trolled by the $p->{textify} attribute, which is a hash that
133 defines how certain tags can be treated as text. If the name of a
134 start tag matches a key in this hash then this tag is converted to
135 text. The hash value is used to specify which tag attribute to
136 obtain the text from. If this tag attribute is missing, then the
137 upper case name of the tag enclosed in brackets is returned, e.g.
138 "[IMG]". The hash value can also be a subroutine reference. In
139 this case the routine is called with the start tag token content as
140 its argument and the return value is treated as the text.
141 The default $p->{textify} value is:
143 {img => "alt", applet => "alt"}
145 This means that <IMG> and <APPLET> tags are treated as text, and
147 that the text to substitute can be found in the ALT attribute.
148 $p->get_trimmed_text
150 $p->get_trimmed_text( @endtags )
151 Same as $p->get_text above, but will collapse any sequences of
152 white space to a single space character. Leading and trailing
153 white space is removed.
154 $p->get_phrase
156 This will return all text found at the current position ignoring
157 any phrasal-level tags. Text is extracted until the first non
158 phrasal-level tag. Textification of tags is the same as for
159 get_text(). This method will collapse white space in the same way
160 as get_trimmed_text() does.
161 The definition of <i>phrasal-level tags</i> is obtained from the
163 HTML::Tagset module.
164
166 This example extracts all links from a document. It will print one
167 line for each link, containing the URL and the textual description
168 between the <A>...</A> tags:
169 use HTML::TokeParser;
171 $p = HTML::TokeParser->new(shift⎪⎪"index.html");
172 while (my $token = $p->get_tag("a")) {
174 my $url = $token->[1]{href} ⎪⎪ "-";
175 my $text = $p->get_trimmed_text("/a");
176 print "$url\t$text\n";
177 }
178 This example extract the <TITLE> from the document:
180 use HTML::TokeParser;
182 $p = HTML::TokeParser->new(shift⎪⎪"index.html");
183 if ($p->get_tag("title")) {
184 my $title = $p->get_trimmed_text;
185 print "Title: $title\n";
186 }
187
189 HTML::PullParser, HTML::Parser
190
192 Copyright 1998-2005 Gisle Aas.
193 This library is free software; you can redistribute it and/or modify it
195 under the same terms as Perl itself.
196perl v5.8.8 2006-04-26 HTML::TokeParser(3)