1Statistics::ContingencyU(s3e)r Contributed Perl DocumentaSttiaotnistics::Contingency(3)
2
6 Statistics::Contingency - Calculate precision, recall, F1, accuracy,
7 etc.
8
10 version 0.09
11
13 use Statistics::Contingency;
14 my $s = new Statistics::Contingency(categories => \@all_categories);
15 while (...something...) {
17 ...
18 $s->add_result($assigned_categories, $correct_categories);
19 }
20 print "Micro F1: ", $s->micro_F1, "\n"; # Access a single statistic
22 print $s->stats_table; # Show several stats in table form
23
25 The "Statistics::Contingency" class helps you calculate several useful
26 statistical measures based on 2x2 "contingency tables". I use these
27 measures to help judge the results of automatic text categorization
28 experiments, but they are useful in other situations as well.
29 The general usage flow is to tally a whole bunch of results in the
31 "Statistics::Contingency" object, then query that object to obtain the
32 measures you are interested in. When all results have been collected,
33 you can get a report on accuracy, precision, recall, F1, and so on,
34 with both macro-averaging and micro-averaging over categories.
35 Macro vs. Micro Statistics
37 All of the statistics offered by this module can be calculated for each
38 category and then averaged, or can be calculated over all decisions and
39 then averaged. The former is called macro-averaging (specifically,
40 macro-averaging with respect to category), and the latter is called
41 micro-averaging. The two procedures bias the results differently -
42 micro-averaging tends to over-emphasize the performance on the largest
43 categories, while macro-averaging over-emphasizes the performance on
44 the smallest. It's often best to look at both of them to get a good
45 idea of how your data distributes across categories.
46 Statistics available
48 All of the statistics are calculated based on a so-called "contingency
49 table", which looks like this:
50 Correct=Y Correct=N
52 +-----------+-----------+
53 Assigned=Y | a | b |
54 +-----------+-----------+
55 Assigned=N | c | d |
56 +-----------+-----------+
57 a, b, c, and d are counts that reflect how the assigned categories
59 matched the correct categories. Depending on whether a macro-statistic
60 or a micro-statistic is being calculated, these numbers will be tallied
61 per-category or for the entire result set.
62 The following statistics are available:
64 • accuracy
66 This measures the portion of all decisions that were correct
68 decisions. It is defined as "(a+d)/(a+b+c+d)". It falls in the
69 range from 0 to 1, with 1 being the best score.
70 Note that macro-accuracy and micro-accuracy will always give the
72 same number.
73 • error
75 This measures the portion of all decisions that were incorrect
77 decisions. It is defined as "(b+c)/(a+b+c+d)". It falls in the
78 range from 0 to 1, with 0 being the best score.
79 Note that macro-error and micro-error will always give the same
81 number.
82 • precision
84 This measures the portion of the assigned categories that were
86 correct. It is defined as "a/(a+b)". It falls in the range from 0
87 to 1, with 1 being the best score.
88 • recall
90 This measures the portion of the correct categories that were
92 assigned. It is defined as "a/(a+c)". It falls in the range from
93 0 to 1, with 1 being the best score.
94 • F1
96 This measures an even combination of precision and recall. It is
98 defined as "2*p*r/(p+r)". In terms of a, b, and c, it may be
99 expressed as "2a/(2a+b+c)". It falls in the range from 0 to 1,
100 with 1 being the best score.
101 The F1 measure is often the only simple measure that is worth trying to
103 maximize on its own - consider the fact that you can get a perfect
104 precision score by always assigning zero categories, or a perfect
105 recall score by always assigning every category. A truly smart system
106 will assign the correct categories and only the correct categories,
107 maximizing precision and recall at the same time, and therefore
108 maximizing the F1 score.
109 Sometimes it's worth trying to maximize the accuracy score, but
111 accuracy (and its counterpart error) are considered fairly crude scores
112 that don't give much information about the performance of a
113 categorizer.
114
116 The general execution flow when using this class is to create a
117 "Statistics::Contingency" object, add a bunch of results to it, and
118 then report on the results.
119 • $e = Statistics::Contingency->new()
121 Returns a new "Statistics::Contingency" object. Expects a
123 "categories" parameter specifying the entire set of categories that
124 may be assigned during this experiment. Also accepts a "verbose"
125 parameter - if true, some diagnostic status information will be
126 displayed when certain actions are performed.
127 • $e->add_result($assigned_categories, $correct_categories, $name)
129 Adds a new result to the experiment. The lists of assigned and
131 correct categories can be given as an array of category names
132 (strings), as a hash whose keys are the category names and whose
133 values are anything logically true, or as a single string if there
134 is only one category.
135 If you've already got the lists in hash form, this will be the
137 fastest way to pass them. Otherwise, the current implementation
138 will convert them to hash form internally in order to make its
139 calculations efficient.
140 The $name parameter is an optional name for this result. It will
142 only be used in error messages or debugging/progress output.
143 In the current implementation, we only store the contingency tables
145 per category, as well as a table for the entire result set. This
146 means that you can't recover information about any particular
147 single result from the "Statistics::Contingency" object.
148 • $e->set_entries($a, $b, $c, $d)
150 If you don't wish to use the c<add_result()> interface, but still
152 take advantage of the calculation methods and the various edge
153 cases they handle, you can directly set the four elements of the
154 contingency table with this method.
155 • $e->micro_accuracy
157 Returns the micro-averaged accuracy for the data set.
159 • $e->micro_error
161 Returns the micro-averaged error for the data set.
163 • $e->micro_precision
165 Returns the micro-averaged precision for the data set.
167 • $e->micro_recall
169 Returns the micro-averaged recall for the data set.
171 • $e->micro_F1
173 Returns the micro-averaged F1 for the data set.
175 • $e->macro_accuracy
177 Returns the macro-averaged accuracy for the data set.
179 • $e->macro_error
181 Returns the macro-averaged error for the data set.
183 • $e->macro_precision
185 Returns the macro-averaged precision for the data set.
187 • $e->macro_recall
189 Returns the macro-averaged recall for the data set.
191 • $e->macro_F1
193 Returns the macro-averaged F1 for the data set.
195 • $e->stats_table
197 Returns a string combining several statistics in one graphic table.
199 Since accuracy is 1 minus error, we only report error since it
200 takes less space to print. An optional argument specifies the
201 number of significant digits to show in the data - the default is 3
202 significant digits.
203 • $e->category_stats
205 Returns a hash reference whose keys are the names of each category,
207 and whose values contain the various statistical measures
208 (accuracy, error, precision, recall, or F1) about each category as
209 a hash reference. For example, to print a single statistic:
210 print $e->category_stats->{sports}{recall}, "\n";
212 Or to print certain statistics for all categtories:
214 my $stats = $e->category_stats;
216 while (my ($cat, $value) = each %$stats) {
217 print "Category '$cat': \n";
218 print " Accuracy: $value->{accuracy}\n";
219 print " Precision: $value->{precision}\n";
220 print " F1: $value->{F1}\n";
221 }
222
224 Ken Williams <kwilliams@cpan.org>
225
227 Copyright 2002-2008 Ken Williams. All rights reserved.
228 This distribution is free software; you can redistribute it and/or
230 modify it under the same terms as Perl itself.
231perl v5.32.1 2021-01-27 Statistics::Contingency(3)