1HTML::TableExtract(3) User Contributed Perl DocumentationHTML::TableExtract(3)
2
3
4

NAME

6       HTML::TableExtract - Perl module for extracting the content contained
7       in tables within an HTML document, either as text or encoded element
8       trees.
9

SYNOPSIS

11        # Matched tables are returned as table objects; tables can be matched
12        # using column headers, depth, count within a depth, table tag
13        # attributes, or some combination of the four.
14
15        # Example: Using column header information.
16        # Assume an HTML document with tables that have "Date", "Price", and
17        # "Cost" somewhere in a row. The columns beneath those headings are
18        # what you want to extract. They will be returned in the same order as
19        # you specified the headers since 'automap' is enabled by default.
20
21        use HTML::TableExtract;
22        my $te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
23        $te->parse($html_string);
24
25        # Examine all matching tables
26        foreach my $ts ($te->tables) {
27          print "Table (", join(',', $ts->coords), "):\n";
28          foreach my $row ($ts->rows) {
29             print join(',', @$row), "\n";
30          }
31        }
32
33        # Shorthand...top level rows() method assumes the first table found in
34        # the document if no arguments are supplied.
35        foreach my $row ($te->rows) {
36           print join(',', @$row), "\n";
37        }
38
39        # Example: Using depth and count information.
40        # Every table in the document has a unique depth and count tuple, so
41        # when both are specified it is a unique table. Depth and count both
42        # begin with 0, so in this case we are looking for a table (depth 2)
43        # within a table (depth 1) within a table (depth 0, which is the top
44        # level HTML document). In addition, it must be the third (count 2)
45        # such instance of a table at that depth.
46
47        my $te = HTML::TableExtract->new( depth => 2, count => 2 );
48        $te->parse_file($html_file);
49        foreach my $ts ($te->tables) {
50           print "Table found at ", join(',', $ts->coords), ":\n";
51           foreach my $row ($ts->rows) {
52              print "   ", join(',', @$row), "\n";
53           }
54        }
55
56        # Example: Using table tag attributes.
57        # If multiple attributes are specified, all must be present and equal
58        # for match to occur.
59
60        my $te = HTML::TableExtract->new( attribs => { border => 1 } );
61        $te->parse($html_string);
62        foreach my $ts ($te->tables) {
63          print "Table with border=1 found at ", join(',', $ts->coords), ":\n";
64          foreach my $row ($ts->rows) {
65             print "   ", join(',', @$row), "\n";
66          }
67        }
68
69        # Example: Extracting as an HTML::Element tree structure
70        # Rather than extracting raw text, the html can be converted into a
71        # tree of element objects. The HTML document is composed of
72        # HTML::Element objects and the tables are HTML::ElementTable
73        # structures. Using this, the contents of tables within a document can
74        # be edited in-place.
75
76        use HTML::TableExtract qw(tree);
77        my $te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) );
78        $te->parse_file($html_file);
79        my $table = $te->first_table_found;
80        my $table_tree = $table->tree;
81        $table_tree->cell(4,4)->replace_content('Golden Goose');
82        my $table_html = $table_tree->as_HTML;
83        my $table_text = $table_tree->as_text;
84        my $document_tree = $te->tree;
85        my $document_html = $document_tree->as_HTML;
86

DESCRIPTION

88       HTML::TableExtract is a subclass of HTML::Parser that serves to extract
89       the information from tables of interest contained within an HTML
90       document. The information from each extracted table is stored in table
91       objects. Tables can be extracted as text, HTML, or HTML::ElementTable
92       structures (for in-place editing or manipulation).
93
94       There are currently four constraints available to specify which tables
95       you would like to extract from a document: Headers, Depth, Count, and
96       Attributes.
97
98       Headers, the most flexible and adaptive of the techniques, involves
99       specifying text in an array that you expect to appear above the data in
100       the tables of interest. Once all headers have been located in a row of
101       that table, all further cells beneath the columns that matched your
102       headers are extracted. All other columns are ignored: think of it as
103       vertical slices through a table. In addition, TableExtract
104       automatically rearranges each row in the same order as the headers you
105       provided. If you would like to disable this, set automap to 0 during
106       object creation, and instead rely on the column_map() method to find
107       out the order in which the headers were found. Furthermore,
108       TableExtract will automatically compensate for cell span issues so that
109       columns are really the same columns as you would visually see in a
110       browser. This behavior can be disabled by setting the gridmap parameter
111       to 0. HTML is stripped from the entire textual content of a cell before
112       header matches are attempted -- unless the keep_html parameter was
113       enabled.
114
115       Depth and Count are more specific ways to specify tables in relation to
116       one another. Depth represents how deeply a table resides in other
117       tables. The depth of a top-level table in the document is 0. A table
118       within a top-level table has a depth of 1, and so on. Each depth can be
119       thought of as a layer; tables sharing the same depth are on the same
120       layer. Within each of these layers, Count represents the order in which
121       a table was seen at that depth, starting with 0. Providing both a depth
122       and a count will uniquely specify a table within a document.
123
124       Attributes match based on the attributes of the html <table> tag, for
125       example, border widths or background color.
126
127       Each of the Headers, Depth, Count, and Attributes specifications are
128       cumulative in their effect on the overall extraction.  For instance, if
129       you specify only a Depth, then you get all tables at that depth (note
130       that these could very well reside in separate higher- level tables
131       throughout the document since depth extends across tables).  If you
132       specify only a Count, then the tables at that Count from all depths are
133       returned (i.e., the nth occurrence of a table at each depth). If you
134       only specify Headers, then you get all tables in the document
135       containing those column headers. If you have specified multiple
136       constraints of Headers, Depth, Count, and Attributes, then each
137       constraint has veto power over whether a particular table is extracted.
138
139       If no Headers, Depth, Count, or Attributes are specified, then all
140       tables match.
141
142       When extracting only text from tables, the text is decoded with
143       HTML::Entities by default; this can be disabled by setting the decode
144       parameter to 0.
145
146   Extraction Modes
147       The default mode of extraction for HTML::TableExtract is raw text or
148       HTML. In this mode, embedded tables are completely decoupled from one
149       another. In this case, HTML::TableExtract is a subclass of
150       HTML::Parser:
151
152         use HTML::TableExtract;
153
154       Alternatively, tables can be extracted as HTML::ElementTable
155       structures, which are in turn embedded in an HTML::Element tree
156       representing the entire HTML document. Embedded tables are not
157       decoupled from one another since this tree structure must be
158       maintained. In this case, HTML::TableExtract is a subclass of
159       HTML::TreeBuilder (itself a subclass of HTML:::Parser):
160
161         use HTML::TableExtract qw(tree);
162
163       In either case, the basic interface for HTML::TableExtract and the
164       resulting table objects remains the same -- all that changes is what
165       you can do with the resulting data.
166
167       HTML::TableExtract is a subclass of HTML::Parser, and as such inherits
168       all of its basic methods such as "parse()" and "parse_file()". During
169       scans, "start()", "end()", and "text()" are utilized. Feel free to
170       override them, but if you do not eventually invoke them in the SUPER
171       class with some content, results are not guaranteed.
172
173   Advice
174       The main point of this module was to provide a flexible method of
175       extracting tabular information from HTML documents without relying to
176       heavily on the document layout. For that reason, I suggest using
177       Headers whenever possible -- that way, you are anchoring your
178       extraction on what the document is trying to communicate rather than
179       some feature of the HTML comprising the document (other than the fact
180       that the data is contained in a table).
181

METHODS

183       The following are the top-level methods of the HTML::TableExtract
184       object. Tables that have matched a query are actually returned as
185       separate objects of type HTML::TableExtract::Table. These table objects
186       have their own methods, documented further below.
187
188   CONSTRUCTOR
189       new()
190           Return a new HTML::TableExtract object. Valid attributes are:
191
192           headers
193               Passed as an array reference, headers specify strings of
194               interest at the top of columns within targeted tables. They can
195               be either strings or regular expressions (qr//). If they are
196               strings, they will eventually be passed through a non-anchored,
197               case-insensitive regular expression, so regexp special
198               characters are allowed.
199
200               The table row containing the headers is not returned, unless
201               "keep_headers" was specified or you are extracting into an
202               element tree. In either case the header row can be accessed via
203               the hrow() method from within the table object.
204
205               Columns that are not beneath one of the provided headers will
206               be ignored unless "slice_columns" was set to 0. Columns will,
207               by default, be rearranged into the same order as the headers
208               you provide (see the automap parameter for more information)
209               unless "slice_columns" is 0.
210
211               Additionally, by default columns are considered what you would
212               see visually beneath that header when the table is rendered in
213               a browser.  See the "gridmap" parameter for more information.
214
215               HTML within a header is stripped before the match is attempted,
216               unless the "keep_html" parameter was specified and
217               "strip_html_on_match" is false.
218
219           depth
220               Specify how embedded in other tables your tables of interest
221               should be.  Top-level tables in the HTML document have a depth
222               of 0, tables within top-level tables have a depth of 1, and so
223               on.
224
225           count
226               Specify which table within each depth you are interested in,
227               beginning with 0.
228
229           attribs
230               Passed as a hash reference, attribs specify attributes of
231               interest within the HTML <table> tag itself.
232
233           automap
234               Automatically applies the ordering reported by column_map() to
235               the rows returned by rows(). This only makes a difference if
236               you have specified Headers and they turn out to be in a
237               different order in the table than what you specified. Automap
238               will rearrange the columns in the same order as the headers
239               appear. To get the original ordering, you will need to take
240               another slice of each row using column_map(). automap is
241               enabled by default.
242
243           slice_columns
244               Enabled by default, this option controls whether vertical
245               slices are returned from under headers that match. When
246               disabled, all columns of the matching table are retained,
247               regardles of whether they had a matching header above them.
248               Disabling this also disables "automap".
249
250           keep_headers
251               Disabled by default, and only applicable when header
252               constraints have been specified, "keep_headers" will retain the
253               matching header row as the first row of table data when
254               enabled. This option has no effect if extracting into an
255               element tree structure. In any case, the header row is
256               accessible from the table method "hrow()".
257
258           gridmap
259               Controls whether the table contents are returned as a grid or a
260               tree.  ROWSPAN and COLSPAN issues are compensated for, and
261               columns really are columns. Empty phantom cells are created
262               where they would have been obscured by ROWSPAN or COLSPAN
263               settings. This really becomes an issue when extracting columns
264               beneath headers. Enabled by default.
265
266           subtables
267               Extract all tables embedded within matched tables.
268
269           decode
270               Automatically decode retrieved text with
271               HTML::Entities::decode_entities(). Enabled by default. Has no
272               effect if "keep_html" was specified or if extracting into an
273               element tree structure.
274
275           br_translate
276               Translate <br> tags into newlines. Sometimes the remaining text
277               can be hard to parse if the <br> tag is simply dropped. Enabled
278               by default. Has no effect if keep_html is enabled or if
279               extracting into an element tree structure.
280
281           keep_html
282               Return the raw HTML contained in the cell, rather than just the
283               visible text. Embedded tables are not retained in the HTML
284               extracted from a cell. Patterns for header matches must take
285               into account HTML in the string if this option is enabled. This
286               option has no effect if extracting into an elment tree
287               structure.
288
289           strip_html_on_match
290               When "keep_html" is enabled, HTML is stripped by default during
291               attempts at matching header strings (so if
292               "strip_html_on_match" is not enabled and "keep_html" is, you
293               would have to include potential HTML tags in the regexp for
294               header matches). Stripped header tags are replaced with an
295               empty string, e.g. 'hot d<em>og</em>' would become 'hot dog'
296               before attempting a match.
297
298           error_handle
299               Filehandle where error messages are printed. STDERR by default.
300
301           debug
302               Prints some debugging information to STDERR, more for higher
303               values.  If "error_handle" was provided, messages are printed
304               there rather than STDERR.
305
306   REGULAR METHODS
307       The following methods are invoked directly from an HTML::TableExtract
308       object.
309
310       depths()
311           Returns all depths that contained matched tables in the document.
312
313       counts($depth)
314           For a particular depth, returns all counts that contained matched
315           tables.
316
317       table($depth, $count)
318           For a particular depth and count, return the table object for the
319           table found, if any.
320
321       tables()
322           Return table objects for all tables that matched. Returns an empty
323           list if no tables matched.
324
325       first_table_found()
326           Return the table state object for the first table matched in the
327           document. Returns undef if no tables were matched.
328
329       current_table()
330           Returns the current table object while parsing the HTML. Only
331           useful if you're messing around with overriding HTML::Parser
332           methods.
333
334       tree()
335           If the module was invoked in tree extraction mode, returns a
336           reference to the top node of the HTML::Element tree structure for
337           the entire document (which includes, ultimately, all tables within
338           the document).
339
340       tables_report([$show_content, $col_sep])
341           Return a string summarizing extracted tables, along with their
342           depth and count. Optionally takes a $show_content flag which will
343           dump the extracted contents of each table as well with columns
344           separated by $col_sep. Default $col_sep is ':'.
345
346       tables_dump([$show_content, $col_sep])
347           Same as "tables_report()" except dump the information to STDOUT.
348
349       start
350       end
351       text
352           These are the hooks into HTML::Parser. If you want to subclass this
353           module and have things work, you must at some point call these with
354           content.
355
356   DEPRECATED METHODS
357       Tables used to be called 'table states'. Accordingly, the following
358       methods still work but have been deprecated:
359
360       table_state()
361           Is now table()
362
363       table_states()
364           Is now tables()
365
366       first_table_state_found()
367           Is now first_table_found()
368
369   TABLE METHODS
370       The following methods are invoked from an HTML::TableExtract::Table
371       object, such as those returned from the "tables()" method.
372
373       rows()
374           Return all rows within a matched table. Each row returned is a
375           reference to an array containing the text, HTML, or reference to
376           the HTML::Element object of each cell depending the mode of
377           extraction. Tables with rowspan or colspan attributes will have
378           some cells containing undef.  Returns a list or a reference to an
379           array depending on context.
380
381       columns()
382           Return all columns within a matched table. Each column returned is
383           a reference to an array containing the text, HTML, or reference to
384           HTML::Element object of each cell depending on the mode of
385           extraction.  Tables with rowspan or colspan attributes will have
386           some cells containing undef.
387
388       row($row)
389           Return a particular row from within a matched table either as a
390           list or an array reference, depending on context.
391
392       column($col)
393           Return a particular column from within a matched table as a list or
394           an array reference, depending on context.
395
396       cell($row,$col)
397           Return a particular item from within a matched table, whether it be
398           the text, HTML, or reference to the HTML::Element object of that
399           cell, depending on the mode of extraction. If the cell was covered
400           due to rowspan or colspan effects, will return undef.
401
402       space($row,$col)
403           The same as cell(), except in cases where the given coordinates
404           were covered due to rowspan or colspan issues, in which case the
405           content of the covering cell is returned rather than undef.
406
407       depth()
408           Return the depth at which this table was found.
409
410       count()
411           Return the count for this table within the depth it was found.
412
413       coords()
414           Return depth and count in a list.
415
416       tree()
417           If the module was invoked in tree extraction mode, this accessor
418           provides a reference to the HTML::ElementTable structure
419           encompassing the table.
420
421       hrow()
422           Returns the header row as a list when headers were specified as a
423           constraint. If "keep_headers" was specified initially, this is
424           equivalent to the first row returned by the "rows()" method.
425
426       column_map()
427           Return the order (via indices) in which the provided headers were
428           found.  These indices can be used as slices on rows to either order
429           the rows in the same order as headers or restore the rows to their
430           natural order, depending on whether the rows have been pre-adjusted
431           using the automap parameter.
432
433       lineage()
434           Returns the path of matched tables that led to matching this table.
435           The path is a list of array refs containing depth, count, row, and
436           column values for each ancestor table involved. Note that
437           corresponding table objects will not exist for ancestral tables
438           that did not match specified constraints.
439

NOTES ON TREE EXTRACTION MODE

441       As mentioned above, HTML::TableExtract can be invoked in 'tree' mode
442       where the resulting HTML and extracted tables are encoded in
443       HTML::Element tree structures:
444
445         use HTML::TableExtract 'tree';
446
447       There are a number of things to take note of while using this mode. The
448       entire HTML document is encoded into an HTML::Element tree. Each table
449       is part of this structure, but nevertheless is tracked separately via
450       an HTML::ElementTable structure, which is a specialized form of
451       HTML::Element tree.
452
453       The HTML::ElementTable objects are accessible by invoking the tree()
454       method from within each table object returned by HTML::TableExtract.
455       The HTML::ElementTable objects have their own row(), col(), and cell()
456       methods (among others). These are not to be confused with the row() and
457       column() methods provided by the HTML::TableExtract::Table objects.
458
459       For example, the row() method from HTML::ElementTable will provide a
460       reference to a 'glob' of all the elements in that row. Actions (such as
461       setting attributes) performed on that row reference will affect all
462       elements within that row. On the other hand, the row() method from the
463       HTML::TableExtract::Table object will return an array (either by
464       reference or list, depending on context) of the contents of each cell
465       within the row. In tree mode, the content is represented by individual
466       references to each cell -- these are references to the same
467       HTML::Element objects that reside in the HTML::Element tree.
468
469       The cell() methods provided in both cases will therefore return
470       references to the same object. The exception to this is when a 'cell'
471       in the table grid was originally 'covered' due to rowspan or colspan
472       issues -- in this case the cell content will be undef. Likewise, the
473       row() or column() methods from HTML::TableExtract::Table objects will
474       return arrays potentially containing a mixture of object references and
475       undefs.  If you're going to be doing lots of manipulation of the table
476       elements, it might be more efficient to access them via the methods
477       provided by the HTML::ElementTable object instead. See
478       HTML::ElementTable for more information on how to manipulate those
479       objects.
480
481       An alternative to the cell() method in HTML::TableExtract::Table is the
482       space() method. It is largely similar to cell(), except when given
483       coordinates of a cell that was covered due to rowspan or colspan
484       effects, it will return the contents of the cell that was covering that
485       space rather than undef. So if, for example, cell (0,0) had a rowspan
486       of 2 and colspan of 2, cell(1,1) would return undef and space(1,1)
487       would return the same content as cell(0,0) or space(0,0).
488

REQUIRES

490       HTML::Parser(3), HTML::Entities(3)
491

OPTIONALLY REQUIRES

493       HTML::TreeBuilder(3), HTML::ElementTable(3)
494

AUTHOR

496       Matthew P. Sisk, <sisk@mojotoad.com>
497
499       Copyright (c) 2000-2017 Matthew P. Sisk.  All rights reserved. All
500       wrongs revenged. This program is free software; you can redistribute it
501       and/or modify it under the same terms as Perl itself.
502

SEE ALSO

504       HTML::Parser(3), HTML::TreeBuilder(3), HTML::ElementTable(3), perl(1).
505
506
507
508perl v5.32.0                      2020-07-28             HTML::TableExtract(3)
Impressum