1HTML::TableExtract(3) User Contributed Perl DocumentationHTML::TableExtract(3)
2
6 HTML::TableExtract - Perl module for extracting the content contained
7 in tables within an HTML document, either as text or encoded element
8 trees.
9
11 # Matched tables are returned as table objects; tables can be matched
12 # using column headers, depth, count within a depth, table tag
13 # attributes, or some combination of the four.
14 # Example: Using column header information.
16 # Assume an HTML document with tables that have "Date", "Price", and
17 # "Cost" somewhere in a row. The columns beneath those headings are
18 # what you want to extract. They will be returned in the same order as
19 # you specified the headers since 'automap' is enabled by default.
20 use HTML::TableExtract;
22 $te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
23 $te->parse($html_string);
24 # Examine all matching tables
26 foreach $ts ($te->tables) {
27 print "Table (", join(',', $ts->coords), "):\n";
28 foreach $row ($ts->rows) {
29 print join(',', @$row), "\n";
30 }
31 }
32 # Shorthand...top level rows() method assumes the first table found in
34 # the document if no arguments are supplied.
35 foreach $row ($te->rows) {
36 print join(',', @$row), "\n";
37 }
38 # Example: Using depth and count information.
40 # Every table in the document has a unique depth and count tuple, so
41 # when both are specified it is a unique table. Depth and count both
42 # begin with 0, so in this case we are looking for a table (depth 2)
43 # within a table (depth 1) within a table (depth 0, which is the top
44 # level HTML document). In addition, it must be the third (count 2)
45 # such instance of a table at that depth.
46 $te = HTML::TableExtract->new( depth => 2, count => 2 );
48 $te->parse_file($html_file);
49 foreach $ts ($te->tables) {
50 print "Table found at ", join(',', $ts->coords), ":\n";
51 foreach $row ($ts->rows) {
52 print " ", join(',', @$row), "\n";
53 }
54 }
55 # Example: Using table tag attributes.
57 # If multiple attributes are specified, all must be present and equal
58 # for match to occur.
59 $te = HTML::TableExtract->new( attribs => { border => 1 } );
61 $te->parse($html_string);
62 foreach $ts ($te->tables) {
63 print "Table with border=1 found at ", join(',', $ts->coords), ":\n";
64 foreach $row ($ts->rows) {
65 print " ", join(',', @$row), "\n";
66 }
67 }
68 # Example: Extracting as an HTML::Element tree structure
70 # Rather than extracting raw text, the html can be converted into a
71 # tree of element objects. The HTML document is composed of
72 # HTML::Element objects and the tables are HTML::ElementTable
73 # structures. Using this, the contents of tables within a document can
74 # be edited in-place.
75 use HTML::TableExtract qw(tree);
77 $te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) );
78 $te->parse_file($html_file);
79 $table = $te->first_table_found;
80 $table_tree = $table->tree;
81 $table_tree->cell(4,4)->replace_content('Golden Goose');
82 $table_html = $table_tree->as_HTML;
83 $table_text = $table_tree->as_text;
84 $document_tree = $te->tree;
85 $document_html = $document_tree->as_HTML;
86
88 HTML::TableExtract is a subclass of HTML::Parser that serves to extract
89 the information from tables of interest contained within an HTML docu‐
90 ment. The information from each extracted table is stored in table
91 objects. Tables can be extracted as text, HTML, or HTML::ElementTable
92 structures (for in-place editing or manipulation).
93 There are currently four constraints available to specify which tables
95 you would like to extract from a document: Headers, Depth, Count, and
96 Attributes.
97 Headers, the most flexible and adaptive of the techniques, involves
99 specifying text in an array that you expect to appear above the data in
100 the tables of interest. Once all headers have been located in a row of
101 that table, all further cells beneath the columns that matched your
102 headers are extracted. All other columns are ignored: think of it as
103 vertical slices through a table. In addition, TableExtract automati‐
104 cally rearranges each row in the same order as the headers you pro‐
105 vided. If you would like to disable this, set automap to 0 during
106 object creation, and instead rely on the column_map() method to find
107 out the order in which the headers were found. Furthermore, TableEx‐
108 tract will automatically compensate for cell span issues so that col‐
109 umns are really the same columns as you would visually see in a
110 browser. This behavior can be disabled by setting the gridmap parameter
111 to 0. HTML is stripped from the entire textual content of a cell before
112 header matches are attempted -- unless the keep_html parameter was
113 enabled.
114 Depth and Count are more specific ways to specify tables in relation to
116 one another. Depth represents how deeply a table resides in other
117 tables. The depth of a top-level table in the document is 0. A table
118 within a top-level table has a depth of 1, and so on. Each depth can be
119 thought of as a layer; tables sharing the same depth are on the same
120 layer. Within each of these layers, Count represents the order in which
121 a table was seen at that depth, starting with 0. Providing both a depth
122 and a count will uniquely specify a table within a document.
123 Attributes match based on the attributes of the html <table> tag, for
125 example, boder widths or background color.
126 Each of the Headers, Depth, Count, and Attributes specifications are
128 cumulative in their effect on the overall extraction. For instance, if
129 you specify only a Depth, then you get all tables at that depth (note
130 that these could very well reside in separate higher- level tables
131 throughout the document since depth extends across tables). If you
132 specify only a Count, then the tables at that Count from all depths are
133 returned (i.e., the nth occurrence of a table at each depth). If you
134 only specify Headers, then you get all tables in the document contain‐
135 ing those column headers. If you have specified multiple constraints of
136 Headers, Depth, Count, and Attributes, then each constraint has veto
137 power over whether a particular table is extracted.
138 If no Headers, Depth, Count, or Attributes are specified, then all
140 tables match.
141 When extracting only text from tables, the text is decoded with
143 HTML::Entities by default; this can be disabled by setting the decode
144 parameter to 0.
145 Extraction Modes
147 The default mode of extraction for HTML::TableExtract is raw text or
149 HTML. In this mode, embedded tables are completely decoupled from one
150 another. In this case, HTML::TableExtract is a subclass of
151 HTML::Parser:
152 use HTML::TableExtract;
154 Alternativevly, tables can be extracted as HTML::ElementTable struc‐
156 tures, which are in turn embedded in an HTML::Element tree representing
157 the entire HTML document. Embedded tables are not decoupled from one
158 another since this tree structure must be manitained. In this case,
159 HTML::TableExtract is a subclass of HTML::TreeBuilder (itself a sub‐
160 class of HTML:::Parser):
161 use HTML::TableExtract qw(tree);
163 In either case, the basic interface for HTML::TableExtract and the
165 resulting table objects remains the same -- all that changes is what
166 you can do with the resulting data.
167 HTML::TableExtract is a subclass of HTML::Parser, and as such inherits
169 all of its basic methods such as "parse()" and "parse_file()". During
170 scans, "start()", "end()", and "text()" are utilized. Feel free to
171 override them, but if you do not eventually invoke them in the SUPER
172 class with some content, results are not guaranteed.
173 Advice
175 The main point of this module was to provide a flexible method of
177 extracting tabular information from HTML documents without relying to
178 heavily on the document layout. For that reason, I suggest using Head‐
179 ers whenever possible -- that way, you are anchoring your extraction on
180 what the document is trying to communicate rather than some feature of
181 the HTML comprising the document (other than the fact that the data is
182 contained in a table).
183
185 The following are the top-level methods of the HTML::TableExtract
186 object. Tables that have matched a query are actually returned as sepa‐
187 rate objects of type HTML::TableExtract::Table. These table objects
188 have their own methods, documented further below.
189 CONSTRUCTOR
191 new()
193 Return a new HTML::TableExtract object. Valid attributes are:
194 headers
196 Passed as an array reference, headers specify strings of inter‐
197 est at the top of columns within targeted tables. They can be
198 either strings or regular expressions (qr//). If they are
199 strings, they will eventually be passed through a non-anchored,
200 case-insensitive regular expression, so regexp special charac‐
201 ters are allowed.
202 The table row containing the headers is not returned, unless
204 "keep_headers" was specified or you are extracting into an ele‐
205 ment tree. In either case the header row can be accessed via
206 the hrow() method from within the table object.
207 Columns that are not beneath one of the provided headers will
209 be ignored unless "slice_columns" was set to 0. Columns will,
210 by default, be rearranged into the same order as the headers
211 you provide (see the automap parameter for more information)
212 unless "slice_columns" is 0.
213 Additionally, by default columns are considered what you would
215 see visually beneath that header when the table is rendered in
216 a browser. See the "gridmap" parameter for more information.
217 HTML within a header is stripped before the match is attempted,
219 unless the "keep_html" parameter was specified and
220 "strip_html_on_match" is false.
221 depth
223 Specify how embedded in other tables your tables of interest
224 should be. Top-level tables in the HTML document have a depth
225 of 0, tables within top-level tables have a depth of 1, and so
226 on.
227 count
229 Specify which table within each depth you are interested in,
230 beginning with 0.
231 attribs
233 Passed as a hash reference, attribs specify attributes of
234 interest within the HTML <table> tag itself.
235 automap
237 Automatically applies the ordering reported by column_map() to
238 the rows returned by rows(). This only makes a difference if
239 you have specified Headers and they turn out to be in a differ‐
240 ent order in the table than what you specified. Automap will
241 rearrange the columns in the same order as the headers appear.
242 To get the original ordering, you will need to take another
243 slice of each row using column_map(). automap is enabled by
244 default.
245 slice_columns
247 Enabled by default, this option controls whether vertical
248 slices are returned from under headers that match. When dis‐
249 abled, all columns of the matching table are retained, regar‐
250 dles of whether they had a matching header above them. Dis‐
251 abling this also disables "automap".
252 keep_headers
254 Disabled by default, and only applicable when header con‐
255 straints have been specified, "keep_headers" will retain the
256 matching header row as the first row of table data when
257 enabled. This option has no effect if extracting into an ele‐
258 ment tree tructure. In any case, the header row is accessible
259 from the table method "hrow()".
260 gridmap
262 Controls whether the table contents are returned as a grid or a
263 tree. ROWSPAN and COLSPAN issues are compensated for, and col‐
264 umns really are columns. Empty phantom cells are created where
265 they would have been obscured by ROWSPAN or COLSPAN settings.
266 This really becomes an issue when extracting columns beneath
267 headers. Enabled by default.
268 subtables
270 Extract all tables embedded within matched tables.
271 decode
273 Automatically decode retrieved text with HTML::Enti‐
274 ties::decode_entities(). Enabled by default. Has no effect if
275 "keep_html" was specified or if extracting into an element tree
276 structure.
277 br_translate
279 Translate <br> tags into newlines. Sometimes the remaining text
280 can be hard to parse if the <br> tag is simply dropped. Enabled
281 by default. Has no effect if keep_html is enabled or if
282 extracting into an element tree structure.
283 keep_html
285 Return the raw HTML contained in the cell, rather than just the
286 visible text. Embedded tables are not retained in the HTML
287 extracted from a cell. Patterns for header matches must take
288 into account HTML in the string if this option is enabled. This
289 option has no effect if extracting into an elment tree struc‐
290 ture.
291 strip_html_on_match
293 When "keep_html" is enabled, HTML is stripped by default during
294 attempts at matching header strings (so if
295 "strip_html_on_match" is not enabled and "keep_html" is, you
296 would have to include potential HTML tags in the regexp for
297 header matches). Stripped header tags are replaced with an
298 empty string, e.g. 'hot d<em>og</em>' would become 'hot dog'
299 before attempting a match.
300 error_handle
302 Filehandle where error messages are printed. STDERR by default.
303 debug
305 Prints some debugging information to STDERR, more for higher
306 values. If "error_handle" was provided, messages are printed
307 there rather than STDERR.
308 REGULAR METHODS
310 The following methods are invoked directly from an HTML::TableExtract
312 object.
313 depths()
315 Returns all depths that contained matched tables in the document.
316 counts($depth)
318 For a particular depth, returns all counts that contained matched
319 tables.
320 table($depth, $count)
322 For a particular depth and count, return the table object for the
323 table found, if any.
324 tables()
326 Return table objects for all tables that matched. Returns an empty
327 list if no tables matched.
328 first_table_found()
330 Return the table state object for the first table matched in the
331 document. Returns undef if no tables were matched.
332 current_table()
334 Returns the current table object while parsing the HTML. Only use‐
335 ful if you're messing around with overriding HTML::Parser methods.
336 tree()
338 If the module was invoked in tree extraction mode, returns a refer‐
339 ence to the top node of the HTML::Element tree structure for the
340 entire document (which includes, ultimately, all tables within the
341 document).
342 tables_report([$show_content, $col_sep])
344 Return a string summarizing extracted tables, along with their
345 depth and count. Optionally takes a $show_content flag which will
346 dump the extracted contents of each table as well with columns sep‐
347 arated by $col_sep. Default $col_sep is ':'.
348 tables_dump([$show_content, $col_sep])
350 Same as "tables_report()" except dump the information to STDOUT.
351 start
353 end
354 text
355 These are the hooks into HTML::Parser. If you want to subclass this
356 module and have things work, you must at some point call these with
357 content.
358 DEPRECATED METHODS
360 Tables used to be called 'table states'. Accordingly, the following
362 methods still work but have been deprecated:
363 table_state()
365 Is now table()
366 table_states()
368 Is now tables()
369 first_table_state_found()
371 Is now first_table_found()
372 TABLE METHODS
374 The following methods are invoked from an HTML::TableExtract::Table
376 object, such as those returned from the "tables()" method.
377 rows()
379 Return all rows within a matched table. Each row returned is a ref‐
380 erence to an array containing the text, HTML, or reference to the
381 HTML::Element object of each cell depending the mode of extraction.
382 Tables with rowspan or colspan attributes will have some cells con‐
383 taining undef. Returns a list or a reference to an array depending
384 on context.
385 columns()
387 Return all columns within a matched table. Each column returned is
388 a reference to an array containing the text, HTML, or reference to
389 HTML::Element object of each cell depending on the mode of extrac‐
390 tion. Tables with rowspan or colspan attributes will have some
391 cells containing undef.
392 row($row)
394 Return a particular row from within a matched table either as a
395 list or an array reference, depending on context.
396 column($col)
398 Return a particular column from within a matched table as a list or
399 an array reference, depending on context.
400 cell($row,$col)
402 Return a particular item from within a matched table, whether it be
403 the text, HTML, or reference to the HTML::Element object of that
404 cell, depending on the mode of extraction. If the cell was covered
405 due to rowspan or colspan effects, will return undef.
406 space($row,$col)
408 The same as cell(), except in cases where the given coordinates
409 were covered due to rowspan or colspan issues, in which case the
410 content of the covering cell is returned rather than undef.
411 depth()
413 Return the depth at which this table was found.
414 count()
416 Return the count for this table within the depth it was found.
417 coords()
419 Return depth and count in a list.
420 tree()
422 If the module was invoked in tree extraction mode, this accessor
423 provides a reference to the HTML::ElementTable structure encompass‐
424 ing the table.
425 hrow()
427 Returns the header row as a list when headers were specified as a
428 constraint. If "keep_headers" was specified initially, this is
429 equivalent to the first row returned by the "rows()" method.
430 column_map()
432 Return the order (via indices) in which the provided headers were
433 found. These indices can be used as slices on rows to either order
434 the rows in the same order as headers or restore the rows to their
435 natural order, depending on whether the rows have been pre-adjusted
436 using the automap parameter.
437 lineage()
439 Returns the path of matched tables that led to matching this table.
440 The path is a list of array refs containing depth, count, row, and
441 column values for each ancestor table involved. Note that corre‐
442 sponding table objects will not exist for ancestral tables that did
443 not match specified constraints.
444
446 As mentioned above, HTML::TableExtract can be invoked in 'tree' mode
447 where the resulting HTML and extracted tables are encoded in HTML::Ele‐
448 ment tree structures:
449 use HTML::TableExtract 'tree';
451 There are a number of things to take note of while using this mode. The
453 entire HTML document is encoded into an HTML::Element tree. Each table
454 is part of this structure, but nevertheless is tracked separately via
455 an HTML::ElementTable structure, which is a specialized form of
456 HTML::Element tree.
457 The HTML::ElementTable objects are accessible by invoking the tree()
459 method from within each table object returned by HTML::TableExtract.
460 The HTML::ElementTable objects have their own row(), col(), and cell()
461 methods (among others). These are not to be confused with the row() and
462 column() methods provided by the HTML::TableExtract::Table objects.
463 For example, the row() method from HTML::ElementTable will provide a
465 reference to a 'glob' of all the elements in that row. Actions (such as
466 setting attributes) performed on that row reference will affect all
467 elements within that row. On the other hand, the row() method from the
468 HTML::TableExtract::Table object will return an array (either by refer‐
469 ence or list, depending on context) of the contents of each cell within
470 the row. In tree mode, the content is represented by individual refer‐
471 ences to each cell -- these are references to the same HTML::Element
472 objects that reside in the HTML::Element tree.
473 The cell() methods provided in both cases will therefore return refer‐
475 ences to the same object. The exception to this is when a 'cell' in the
476 table grid was originally 'covered' due to rowspan or colspan issues --
477 in this case the cell content will be undef. Likewise, the row() or
478 column() methods from HTML::TableExtract::Table objects will return
479 arrays potentially containing a mixture of object references and
480 undefs. If you're going to be doing lots of manipulation of the table
481 elements, it might be more efficient to access them via the methods
482 provided by the HTML::ElementTable object instead. See HTML::Element‐
483 Table for more information on how to manipulate those objects.
484 An alternative to the cell() method in HTML::TableExtract::Table is the
486 space() method. It is largely similar to cell(), except when given
487 coordinates of a cell that was covered due to rowspan or colspan
488 effects, it will return the contents of the cell that was covering that
489 space rather than undef. So if, for example, cell (0,0) had a rowspan
490 of 2 and colspan of 2, cell(1,1) would return undef and space(1,1)
491 would return the same content as cell(0,0) or space(0,0).
492
494 HTML::Parser(3), HTML::Entities(3)
495
497 HTML::TreeBuilder(3), HTML::ElementTable(3)
498
500 Matthew P. Sisk, <sisk@mojotoad.com>
501
503 Copyright (c) 2000-2006 Matthew P. Sisk. All rights reserved. All
504 wrongs revenged. This program is free software; you can redistribute it
505 and/or modify it under the same terms as Perl itself.
506
508 HTML::Parser(3), HTML::TreeBuilder(3), HTML::ElementTable(3), perl(1).
509perl v5.8.8 2006-07-15 HTML::TableExtract(3)