1HTML::TableExtract(3) User Contributed Perl DocumentationHTML::TableExtract(3)
2
6 HTML::TableExtract - Perl module for extracting the content contained
7 in tables within an HTML document, either as text or encoded element
8 trees.
9
11 # Matched tables are returned as table objects; tables can be matched
12 # using column headers, depth, count within a depth, table tag
13 # attributes, or some combination of the four.
14 # Example: Using column header information.
16 # Assume an HTML document with tables that have "Date", "Price", and
17 # "Cost" somewhere in a row. The columns beneath those headings are
18 # what you want to extract. They will be returned in the same order as
19 # you specified the headers since 'automap' is enabled by default.
20 use HTML::TableExtract;
22 my $te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
23 $te->parse($html_string);
24 # Examine all matching tables
26 foreach my $ts ($te->tables) {
27 print "Table (", join(',', $ts->coords), "):\n";
28 foreach my $row ($ts->rows) {
29 print join(',', @$row), "\n";
30 }
31 }
32 # Shorthand...top level rows() method assumes the first table found in
34 # the document if no arguments are supplied.
35 foreach my $row ($te->rows) {
36 print join(',', @$row), "\n";
37 }
38 # Example: Using depth and count information.
40 # Every table in the document has a unique depth and count tuple, so
41 # when both are specified it is a unique table. Depth and count both
42 # begin with 0, so in this case we are looking for a table (depth 2)
43 # within a table (depth 1) within a table (depth 0, which is the top
44 # level HTML document). In addition, it must be the third (count 2)
45 # such instance of a table at that depth.
46 my $te = HTML::TableExtract->new( depth => 2, count => 2 );
48 $te->parse_file($html_file);
49 foreach my $ts ($te->tables) {
50 print "Table found at ", join(',', $ts->coords), ":\n";
51 foreach my $row ($ts->rows) {
52 print " ", join(',', @$row), "\n";
53 }
54 }
55 # Example: Using table tag attributes.
57 # If multiple attributes are specified, all must be present and equal
58 # for match to occur.
59 my $te = HTML::TableExtract->new( attribs => { border => 1 } );
61 $te->parse($html_string);
62 foreach my $ts ($te->tables) {
63 print "Table with border=1 found at ", join(',', $ts->coords), ":\n";
64 foreach my $row ($ts->rows) {
65 print " ", join(',', @$row), "\n";
66 }
67 }
68 # Example: Extracting as an HTML::Element tree structure
70 # Rather than extracting raw text, the html can be converted into a
71 # tree of element objects. The HTML document is composed of
72 # HTML::Element objects and the tables are HTML::ElementTable
73 # structures. Using this, the contents of tables within a document can
74 # be edited in-place.
75 use HTML::TableExtract qw(tree);
77 my $te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) );
78 $te->parse_file($html_file);
79 my $table = $te->first_table_found;
80 my $table_tree = $table->tree;
81 $table_tree->cell(4,4)->replace_content('Golden Goose');
82 my $table_html = $table_tree->as_HTML;
83 my $table_text = $table_tree->as_text;
84 my $document_tree = $te->tree;
85 my $document_html = $document_tree->as_HTML;
86
88 HTML::TableExtract is a subclass of HTML::Parser that serves to extract
89 the information from tables of interest contained within an HTML
90 document. The information from each extracted table is stored in table
91 objects. Tables can be extracted as text, HTML, or HTML::ElementTable
92 structures (for in-place editing or manipulation).
93 There are currently four constraints available to specify which tables
95 you would like to extract from a document: Headers, Depth, Count, and
96 Attributes.
97 Headers, the most flexible and adaptive of the techniques, involves
99 specifying text in an array that you expect to appear above the data in
100 the tables of interest. Once all headers have been located in a row of
101 that table, all further cells beneath the columns that matched your
102 headers are extracted. All other columns are ignored: think of it as
103 vertical slices through a table. In addition, TableExtract
104 automatically rearranges each row in the same order as the headers you
105 provided. If you would like to disable this, set automap to 0 during
106 object creation, and instead rely on the column_map() method to find
107 out the order in which the headers were found. Furthermore,
108 TableExtract will automatically compensate for cell span issues so that
109 columns are really the same columns as you would visually see in a
110 browser. This behavior can be disabled by setting the gridmap parameter
111 to 0. HTML is stripped from the entire textual content of a cell before
112 header matches are attempted -- unless the keep_html parameter was
113 enabled.
114 Depth and Count are more specific ways to specify tables in relation to
116 one another. Depth represents how deeply a table resides in other
117 tables. The depth of a top-level table in the document is 0. A table
118 within a top-level table has a depth of 1, and so on. Each depth can be
119 thought of as a layer; tables sharing the same depth are on the same
120 layer. Within each of these layers, Count represents the order in which
121 a table was seen at that depth, starting with 0. Providing both a depth
122 and a count will uniquely specify a table within a document.
123 Attributes match based on the attributes of the html <table> tag, for
125 example, border widths or background color.
126 Each of the Headers, Depth, Count, and Attributes specifications are
128 cumulative in their effect on the overall extraction. For instance, if
129 you specify only a Depth, then you get all tables at that depth (note
130 that these could very well reside in separate higher- level tables
131 throughout the document since depth extends across tables). If you
132 specify only a Count, then the tables at that Count from all depths are
133 returned (i.e., the nth occurrence of a table at each depth). If you
134 only specify Headers, then you get all tables in the document
135 containing those column headers. If you have specified multiple
136 constraints of Headers, Depth, Count, and Attributes, then each
137 constraint has veto power over whether a particular table is extracted.
138 If no Headers, Depth, Count, or Attributes are specified, then all
140 tables match.
141 When extracting only text from tables, the text is decoded with
143 HTML::Entities by default; this can be disabled by setting the decode
144 parameter to 0.
145 Extraction Modes
147 The default mode of extraction for HTML::TableExtract is raw text or
148 HTML. In this mode, embedded tables are completely decoupled from one
149 another. In this case, HTML::TableExtract is a subclass of
150 HTML::Parser:
151 use HTML::TableExtract;
153 Alternatively, tables can be extracted as HTML::ElementTable
155 structures, which are in turn embedded in an HTML::Element tree
156 representing the entire HTML document. Embedded tables are not
157 decoupled from one another since this tree structure must be
158 maintained. In this case, HTML::TableExtract is a subclass of
159 HTML::TreeBuilder (itself a subclass of HTML:::Parser):
160 use HTML::TableExtract qw(tree);
162 In either case, the basic interface for HTML::TableExtract and the
164 resulting table objects remains the same -- all that changes is what
165 you can do with the resulting data.
166 HTML::TableExtract is a subclass of HTML::Parser, and as such inherits
168 all of its basic methods such as "parse()" and "parse_file()". During
169 scans, "start()", "end()", and "text()" are utilized. Feel free to
170 override them, but if you do not eventually invoke them in the SUPER
171 class with some content, results are not guaranteed.
172 Advice
174 The main point of this module was to provide a flexible method of
175 extracting tabular information from HTML documents without relying to
176 heavily on the document layout. For that reason, I suggest using
177 Headers whenever possible -- that way, you are anchoring your
178 extraction on what the document is trying to communicate rather than
179 some feature of the HTML comprising the document (other than the fact
180 that the data is contained in a table).
181
183 The following are the top-level methods of the HTML::TableExtract
184 object. Tables that have matched a query are actually returned as
185 separate objects of type HTML::TableExtract::Table. These table objects
186 have their own methods, documented further below.
187 CONSTRUCTOR
189 new()
190 Return a new HTML::TableExtract object. Valid attributes are:
191 headers
193 Passed as an array reference, headers specify strings of
194 interest at the top of columns within targeted tables. They can
195 be either strings or regular expressions (qr//). If they are
196 strings, they will eventually be passed through a non-anchored,
197 case-insensitive regular expression, so regexp special
198 characters are allowed.
199 The table row containing the headers is not returned, unless
201 "keep_headers" was specified or you are extracting into an
202 element tree. In either case the header row can be accessed via
203 the hrow() method from within the table object.
204 Columns that are not beneath one of the provided headers will
206 be ignored unless "slice_columns" was set to 0. Columns will,
207 by default, be rearranged into the same order as the headers
208 you provide (see the automap parameter for more information)
209 unless "slice_columns" is 0.
210 Additionally, by default columns are considered what you would
212 see visually beneath that header when the table is rendered in
213 a browser. See the "gridmap" parameter for more information.
214 HTML within a header is stripped before the match is attempted,
216 unless the "keep_html" parameter was specified and
217 "strip_html_on_match" is false.
218 depth
220 Specify how embedded in other tables your tables of interest
221 should be. Top-level tables in the HTML document have a depth
222 of 0, tables within top-level tables have a depth of 1, and so
223 on.
224 count
226 Specify which table within each depth you are interested in,
227 beginning with 0.
228 attribs
230 Passed as a hash reference, attribs specify attributes of
231 interest within the HTML <table> tag itself.
232 automap
234 Automatically applies the ordering reported by column_map() to
235 the rows returned by rows(). This only makes a difference if
236 you have specified Headers and they turn out to be in a
237 different order in the table than what you specified. Automap
238 will rearrange the columns in the same order as the headers
239 appear. To get the original ordering, you will need to take
240 another slice of each row using column_map(). automap is
241 enabled by default.
242 slice_columns
244 Enabled by default, this option controls whether vertical
245 slices are returned from under headers that match. When
246 disabled, all columns of the matching table are retained,
247 regardles of whether they had a matching header above them.
248 Disabling this also disables "automap".
249 keep_headers
251 Disabled by default, and only applicable when header
252 constraints have been specified, "keep_headers" will retain the
253 matching header row as the first row of table data when
254 enabled. This option has no effect if extracting into an
255 element tree structure. In any case, the header row is
256 accessible from the table method "hrow()".
257 gridmap
259 Controls whether the table contents are returned as a grid or a
260 tree. ROWSPAN and COLSPAN issues are compensated for, and
261 columns really are columns. Empty phantom cells are created
262 where they would have been obscured by ROWSPAN or COLSPAN
263 settings. This really becomes an issue when extracting columns
264 beneath headers. Enabled by default.
265 subtables
267 Extract all tables embedded within matched tables.
268 decode
270 Automatically decode retrieved text with
271 HTML::Entities::decode_entities(). Enabled by default. Has no
272 effect if "keep_html" was specified or if extracting into an
273 element tree structure.
274 br_translate
276 Translate <br> tags into newlines. Sometimes the remaining text
277 can be hard to parse if the <br> tag is simply dropped. Enabled
278 by default. Has no effect if keep_html is enabled or if
279 extracting into an element tree structure.
280 keep_html
282 Return the raw HTML contained in the cell, rather than just the
283 visible text. Embedded tables are not retained in the HTML
284 extracted from a cell. Patterns for header matches must take
285 into account HTML in the string if this option is enabled. This
286 option has no effect if extracting into an elment tree
287 structure.
288 strip_html_on_match
290 When "keep_html" is enabled, HTML is stripped by default during
291 attempts at matching header strings (so if
292 "strip_html_on_match" is not enabled and "keep_html" is, you
293 would have to include potential HTML tags in the regexp for
294 header matches). Stripped header tags are replaced with an
295 empty string, e.g. 'hot d<em>og</em>' would become 'hot dog'
296 before attempting a match.
297 error_handle
299 Filehandle where error messages are printed. STDERR by default.
300 debug
302 Prints some debugging information to STDERR, more for higher
303 values. If "error_handle" was provided, messages are printed
304 there rather than STDERR.
305 REGULAR METHODS
307 The following methods are invoked directly from an HTML::TableExtract
308 object.
309 depths()
311 Returns all depths that contained matched tables in the document.
312 counts($depth)
314 For a particular depth, returns all counts that contained matched
315 tables.
316 table($depth, $count)
318 For a particular depth and count, return the table object for the
319 table found, if any.
320 tables()
322 Return table objects for all tables that matched. Returns an empty
323 list if no tables matched.
324 first_table_found()
326 Return the table state object for the first table matched in the
327 document. Returns undef if no tables were matched.
328 current_table()
330 Returns the current table object while parsing the HTML. Only
331 useful if you're messing around with overriding HTML::Parser
332 methods.
333 tree()
335 If the module was invoked in tree extraction mode, returns a
336 reference to the top node of the HTML::Element tree structure for
337 the entire document (which includes, ultimately, all tables within
338 the document).
339 tables_report([$show_content, $col_sep])
341 Return a string summarizing extracted tables, along with their
342 depth and count. Optionally takes a $show_content flag which will
343 dump the extracted contents of each table as well with columns
344 separated by $col_sep. Default $col_sep is ':'.
345 tables_dump([$show_content, $col_sep])
347 Same as "tables_report()" except dump the information to STDOUT.
348 start
350 end
351 text
352 These are the hooks into HTML::Parser. If you want to subclass this
353 module and have things work, you must at some point call these with
354 content.
355 DEPRECATED METHODS
357 Tables used to be called 'table states'. Accordingly, the following
358 methods still work but have been deprecated:
359 table_state()
361 Is now table()
362 table_states()
364 Is now tables()
365 first_table_state_found()
367 Is now first_table_found()
368 TABLE METHODS
370 The following methods are invoked from an HTML::TableExtract::Table
371 object, such as those returned from the "tables()" method.
372 rows()
374 Return all rows within a matched table. Each row returned is a
375 reference to an array containing the text, HTML, or reference to
376 the HTML::Element object of each cell depending the mode of
377 extraction. Tables with rowspan or colspan attributes will have
378 some cells containing undef. Returns a list or a reference to an
379 array depending on context.
380 columns()
382 Return all columns within a matched table. Each column returned is
383 a reference to an array containing the text, HTML, or reference to
384 HTML::Element object of each cell depending on the mode of
385 extraction. Tables with rowspan or colspan attributes will have
386 some cells containing undef.
387 row($row)
389 Return a particular row from within a matched table either as a
390 list or an array reference, depending on context.
391 column($col)
393 Return a particular column from within a matched table as a list or
394 an array reference, depending on context.
395 cell($row,$col)
397 Return a particular item from within a matched table, whether it be
398 the text, HTML, or reference to the HTML::Element object of that
399 cell, depending on the mode of extraction. If the cell was covered
400 due to rowspan or colspan effects, will return undef.
401 space($row,$col)
403 The same as cell(), except in cases where the given coordinates
404 were covered due to rowspan or colspan issues, in which case the
405 content of the covering cell is returned rather than undef.
406 depth()
408 Return the depth at which this table was found.
409 count()
411 Return the count for this table within the depth it was found.
412 coords()
414 Return depth and count in a list.
415 tree()
417 If the module was invoked in tree extraction mode, this accessor
418 provides a reference to the HTML::ElementTable structure
419 encompassing the table.
420 hrow()
422 Returns the header row as a list when headers were specified as a
423 constraint. If "keep_headers" was specified initially, this is
424 equivalent to the first row returned by the "rows()" method.
425 column_map()
427 Return the order (via indices) in which the provided headers were
428 found. These indices can be used as slices on rows to either order
429 the rows in the same order as headers or restore the rows to their
430 natural order, depending on whether the rows have been pre-adjusted
431 using the automap parameter.
432 lineage()
434 Returns the path of matched tables that led to matching this table.
435 The path is a list of array refs containing depth, count, row, and
436 column values for each ancestor table involved. Note that
437 corresponding table objects will not exist for ancestral tables
438 that did not match specified constraints.
439
441 As mentioned above, HTML::TableExtract can be invoked in 'tree' mode
442 where the resulting HTML and extracted tables are encoded in
443 HTML::Element tree structures:
444 use HTML::TableExtract 'tree';
446 There are a number of things to take note of while using this mode. The
448 entire HTML document is encoded into an HTML::Element tree. Each table
449 is part of this structure, but nevertheless is tracked separately via
450 an HTML::ElementTable structure, which is a specialized form of
451 HTML::Element tree.
452 The HTML::ElementTable objects are accessible by invoking the tree()
454 method from within each table object returned by HTML::TableExtract.
455 The HTML::ElementTable objects have their own row(), col(), and cell()
456 methods (among others). These are not to be confused with the row() and
457 column() methods provided by the HTML::TableExtract::Table objects.
458 For example, the row() method from HTML::ElementTable will provide a
460 reference to a 'glob' of all the elements in that row. Actions (such as
461 setting attributes) performed on that row reference will affect all
462 elements within that row. On the other hand, the row() method from the
463 HTML::TableExtract::Table object will return an array (either by
464 reference or list, depending on context) of the contents of each cell
465 within the row. In tree mode, the content is represented by individual
466 references to each cell -- these are references to the same
467 HTML::Element objects that reside in the HTML::Element tree.
468 The cell() methods provided in both cases will therefore return
470 references to the same object. The exception to this is when a 'cell'
471 in the table grid was originally 'covered' due to rowspan or colspan
472 issues -- in this case the cell content will be undef. Likewise, the
473 row() or column() methods from HTML::TableExtract::Table objects will
474 return arrays potentially containing a mixture of object references and
475 undefs. If you're going to be doing lots of manipulation of the table
476 elements, it might be more efficient to access them via the methods
477 provided by the HTML::ElementTable object instead. See
478 HTML::ElementTable for more information on how to manipulate those
479 objects.
480 An alternative to the cell() method in HTML::TableExtract::Table is the
482 space() method. It is largely similar to cell(), except when given
483 coordinates of a cell that was covered due to rowspan or colspan
484 effects, it will return the contents of the cell that was covering that
485 space rather than undef. So if, for example, cell (0,0) had a rowspan
486 of 2 and colspan of 2, cell(1,1) would return undef and space(1,1)
487 would return the same content as cell(0,0) or space(0,0).
488
490 HTML::Parser(3), HTML::Entities(3)
491
493 HTML::TreeBuilder(3), HTML::ElementTable(3)
494
496 Matthew P. Sisk, <sisk@mojotoad.com>
497
499 Copyright (c) 2000-2017 Matthew P. Sisk. All rights reserved. All
500 wrongs revenged. This program is free software; you can redistribute it
501 and/or modify it under the same terms as Perl itself.
502
504 HTML::Parser(3), HTML::TreeBuilder(3), HTML::ElementTable(3), perl(1).
505perl v5.30.1 2020-01-30 HTML::TableExtract(3)