Lingua::Stem::Snowball(3pm)

1Lingua::Stem::Snowball(U3s)er Contributed Perl DocumentatLiionngua::Stem::Snowball(3)
2
3
4

NAME

6       Lingua::Stem::Snowball - Perl interface to Snowball stemmers.
7

SYNOPSIS

9           my @words = qw( horse hooves );
10
11           # OO interface:
12           my $stemmer = Lingua::Stem::Snowball->new( lang => 'en' );
13           $stemmer->stem_in_place( \@words ); # qw( hors hoov )
14
15           # Functional interface:
16           my @stems = stem( 'en', \@words );
17

DESCRIPTION

19       Stemming reduces related words to a common root form -- for instance,
20       "horse", "horses", and "horsing" all become "hors".  Most commonly,
21       stemming is deployed as part of a search application, allowing searches
22       for a given term to match documents which contain other forms of that
23       term.
24
25       This module is very similar to Lingua::Stem -- however, Lingua::Stem is
26       pure Perl, while Lingua::Stem::Snowball is an XS module which provides
27       a Perl interface to the C version of the Snowball stemmers.
28       (<http://snowball.tartarus.org>).
29
30   Supported Languages
31       The following stemmers are available (as of Lingua::Stem::Snowball
32       0.95):
33
34           |-----------------------------------------------------------|
35           | Language   | ISO code | default encoding | also available |
36           |-----------------------------------------------------------|
37           | Danish     | da       | ISO-8859-1       | UTF-8          |
38           | Dutch      | nl       | ISO-8859-1       | UTF-8          |
39           | English    | en       | ISO-8859-1       | UTF-8          |
40           | Finnish    | fi       | ISO-8859-1       | UTF-8          |
41           | French     | fr       | ISO-8859-1       | UTF-8          |
42           | German     | de       | ISO-8859-1       | UTF-8          |
43           | Hungarian  | hu       | ISO-8859-1       | UTF-8          |
44           | Italian    | it       | ISO-8859-1       | UTF-8          |
45           | Norwegian  | no       | ISO-8859-1       | UTF-8          |
46           | Portuguese | pt       | ISO-8859-1       | UTF-8          |
47           | Romanian   | ro       | ISO-8859-2       | UTF-8          |
48           | Russian    | ru       | KOI8-R           | UTF-8          |
49           | Spanish    | es       | ISO-8859-1       | UTF-8          |
50           | Swedish    | sv       | ISO-8859-1       | UTF-8          |
51           | Turkish    | tr       | UTF-8            |                |
52           |-----------------------------------------------------------|
53
54   Benchmarks
55       Here is a comparison of Lingua::Stem::Snowball and Lingua::Stem, using
56       The Works of Edgar Allen Poe, volumes 1-5 (via Project Gutenberg) as
57       source material.  It was produced on a 3.2GHz Pentium 4 running FreeBSD
58       5.3 and Perl 5.8.7.  (The benchmarking script is included in this
59       distribution: devel/benchmark_stemmers.plx.)
60
61           |--------------------------------------------------------------------|
62           | total words: 454285 | unique words: 22748                          |
63           |--------------------------------------------------------------------|
64           | module                        | config        | avg secs | rate    |
65           |--------------------------------------------------------------------|
66           | Lingua::Stem 0.81             | no cache      | 2.029    | 223881  |
67           | Lingua::Stem 0.81             | cache level 2 | 1.280    | 355025  |
68           | Lingua::Stem::Snowball 0.94   | stem          | 1.426    | 318636  |
69           | Lingua::Stem::Snowball 0.94   | stem_in_place | 0.641    | 708495  |
70           |--------------------------------------------------------------------|
71

METHODS / FUNCTIONS

73   new
74           my $stemmer = Lingua::Stem::Snowball->new(
75               lang     => 'es',
76               encoding => 'UTF-8',
77           );
78           die $@ if $@;
79
80       Create a Lingua::Stem::Snowball object.  new() accepts the following
81       hash style parameters:
82
83       ·   lang: An ISO code taken from the table of supported languages,
84           above.
85
86       ·   encoding: A supported character encoding.
87
88       Be careful with the values you supply to new(). If "lang" is invalid,
89       Lingua::Stem::Snowball does not throw an exception, but instead sets
90       $@.  Also, if you supply an invalid combination of values for "lang"
91       and "encoding", Lingua::Stem::Snowball will not warn you, but the
92       behavior will change: stem() will always return undef, and
93       stem_in_place() will be a no-op.
94
95   stem
96           @stemmed = $stemmer->stem( WORDS, [IS_STEMMED] );
97           @stemmed = stem( ISO_CODE, WORDS, [LOCALE], [IS_STEMMED] );
98
99       Return lowercased and stemmed output.  WORDS may be either an array of
100       words or a single scalar word.
101
102       In a scalar context, stem() returns the first item in the array of
103       stems:
104
105           $stem       = $stemmer->stem($word);
106           $first_stem = $stemmer->stem(\@words); # probably wrong
107
108       LOCALE has no effect; it is only there as a placeholder for backwards
109       compatibility (see Changes).  IS_STEMMED must be a reference to a
110       scalar; if it is supplied, it will be set to 1 if the output differs
111       from the input in some way, 0 otherwise.
112
113   stem_in_place
114           $stemmer->stem_in_place(\@words);
115
116       This is a high-performance, streamlined version of stem() (in fact,
117       stem() calls stem_in_place() internally). It has no return value,
118       instead modifying each item in an existing array of words.  The words
119       must already be in lower case.
120
121   lang
122           my $lang = $stemmer->lang;
123           $stemmer->lang($iso_language_code);
124
125       Accessor/mutator for the lang parameter. If there is no stemmer for the
126       supplied ISO code, the language is not changed (but $@ is set).
127
128   encoding
129           my $encoding = $stemmer->encoding;
130           $stemmer->encoding($encoding);
131
132       Accessor/mutator for the encoding parameter.
133
134   stemmers
135           my @iso_codes = stemmers();
136           my @iso_codes = $stemmer->stemmers();
137
138       Returns a list of all valid language codes.
139

REQUESTS & BUGS

141       Please report any requests, suggestions or bugs via the RT bug-tracking
142       system at http://rt.cpan.org/ or email to
143       bug-Lingua-Stem-Snowball@rt.cpan.org.
144
145       http://rt.cpan.org/NoAuth/Bugs.html?Dist=Lingua-Stem-Snowball is the RT
146       queue for Lingua::Stem::Snowball.  Please check to see if your bug has
147       already been reported.
148

AUTHORS

150       Lingua::Stem::Snowball was originally developed to provide access to
151       stemming algorithms for the OpenFTS (full text search engine) project
152       (<http://openfts.sourceforge.net>), by Oleg Bartunov, <oleg at sai dot
153       msu dot su> and Teodor Sigaev, <teodor at stack dot net>.
154
155       Currently maintained by Marvin Humphrey <marvin at rectangular dot
156       com>.  Previously maintained by Fabien Potencier <fabpot at cpan dot
157       org>.
158

COPYRIGHT AND LICENSE

160       Perl bindings copyright 2004-2008 by Marvin Humphrey, Fabien Potencier,
161       Oleg Bartunov and Teodor Sigaev.
162
163       This software may be freely copied and distributed under the same terms
164       and conditions as Perl.
165
166       Snowball files and stemmers are covered by the BSD license.
167