Encode::Guess(3pm)

1Encode::Guess(3pm)     Perl Programmers Reference Guide     Encode::Guess(3pm)
2
3
4

NAME

6       Encode::Guess -- Guesses encoding from data
7

SYNOPSIS

9         # if you are sure $data won't contain anything bogus
10
11         use Encode;
12         use Encode::Guess qw/euc-jp shiftjis 7bit-jis/;
13         my $utf8 = decode("Guess", $data);
14         my $data = encode("Guess", $utf8);   # this doesn't work!
15
16         # more elaborate way
17         use Encode::Guess;
18         my $enc = guess_encoding($data, qw/euc-jp shiftjis 7bit-jis/);
19         ref($enc) or die "Can't guess: $enc"; # trap error this way
20         $utf8 = $enc->decode($data);
21         # or
22         $utf8 = decode($enc->name, $data)
23

ABSTRACT

25       Encode::Guess enables you to guess in what encoding a given data is
26       encoded, or at least tries to.
27

DESCRIPTION

29       By default, it checks only ascii, utf8 and UTF-16/32 with BOM.
30
31         use Encode::Guess; # ascii/utf8/BOMed UTF
32
33       To use it more practically, you have to give the names of encodings to
34       check (suspects as follows).  The name of suspects can either be canon‐
35       ical names or aliases.
36
37       CAVEAT: Unlike UTF-(16⎪32), BOM in utf8 is NOT AUTOMATICALLY STRIPPED.
38
39        # tries all major Japanese Encodings as well
40         use Encode::Guess qw/euc-jp shiftjis 7bit-jis/;
41
42       If the $Encode::Guess::NoUTFAutoGuess variable is set to a true value,
43       no heuristics will be applied to UTF8/16/32, and the result will be
44       limited to the suspects and "ascii".
45
46       Encode::Guess->set_suspects
47           You can also change the internal suspects list via "set_suspects"
48           method.
49
50             use Encode::Guess;
51             Encode::Guess->set_suspects(qw/euc-jp shiftjis 7bit-jis/);
52
53       Encode::Guess->add_suspects
54           Or you can use "add_suspects" method.  The difference is that
55           "set_suspects" flushes the current suspects list while "add_sus‐
56           pects" adds.
57
58             use Encode::Guess;
59             Encode::Guess->add_suspects(qw/euc-jp shiftjis 7bit-jis/);
60             # now the suspects are euc-jp,shiftjis,7bit-jis, AND
61             # euc-kr,euc-cn, and big5-eten
62             Encode::Guess->add_suspects(qw/euc-kr euc-cn big5-eten/);
63
64       Encode::decode("Guess" ...)
65           When you are content with suspects list, you can now
66
67             my $utf8 = Encode::decode("Guess", $data);
68
69       Encode::Guess->guess($data)
70           But it will croak if:
71
72           *   Two or more suspects remain
73
74           *   No suspects left
75
76           So you should instead try this;
77
78             my $decoder = Encode::Guess->guess($data);
79
80           On success, $decoder is an object that is documented in
81           Encode::Encoding.  So you can now do this;
82
83             my $utf8 = $decoder->decode($data);
84
85           On failure, $decoder now contains an error message so the whole
86           thing would be as follows;
87
88             my $decoder = Encode::Guess->guess($data);
89             die $decoder unless ref($decoder);
90             my $utf8 = $decoder->decode($data);
91
92       guess_encoding($data, [, list of suspects])
93           You can also try "guess_encoding" function which is exported by
94           default.  It takes $data to check and it also takes the list of
95           suspects by option.  The optional suspect list is not reflected to
96           the internal suspects list.
97
98             my $decoder = guess_encoding($data, qw/euc-jp euc-kr euc-cn/);
99             die $decoder unless ref($decoder);
100             my $utf8 = $decoder->decode($data);
101             # check only ascii and utf8
102             my $decoder = guess_encoding($data);
103

CAVEATS

105       ·   Because of the algorithm used, ISO-8859 series and other single-
106           byte encodings do not work well unless either one of ISO-8859 is
107           the only one suspect (besides ascii and utf8).
108
109             use Encode::Guess;
110             # perhaps ok
111             my $decoder = guess_encoding($data, 'latin1');
112             # definitely NOT ok
113             my $decoder = guess_encoding($data, qw/latin1 greek/);
114
115           The reason is that Encode::Guess guesses encoding by trial and
116           error.  It first splits $data into lines and tries to decode the
117           line for each suspect.  It keeps it going until all but one encod‐
118           ing is eliminated out of suspects list.  ISO-8859 series is just
119           too successful for most cases (because it fills almost all code
120           points in \x00-\xff).
121
122       ·   Do not mix national standard encodings and the corresponding vendor
123           encodings.
124
125             # a very bad idea
126             my $decoder
127                = guess_encoding($data, qw/shiftjis MacJapanese cp932/);
128
129           The reason is that vendor encoding is usually a superset of
130           national standard so it becomes too ambiguous for most cases.
131
132       ·   On the other hand, mixing various national standard encodings
133           automagically works unless $data is too short to allow for guess‐
134           ing.
135
136            # This is ok if $data is long enough
137            my $decoder =
138             guess_encoding($data, qw/euc-cn
139                                      euc-jp shiftjis 7bit-jis
140                                      euc-kr
141                                      big5-eten/);
142
143       ·   DO NOT PUT TOO MANY SUSPECTS!  Don't you try something like this!
144
145             my $decoder = guess_encoding($data,
146                                          Encode->encodings(":all"));
147
148       It is, after all, just a guess.  You should alway be explicit when it
149       comes to encodings.  But there are some, especially Japanese, environ‐
150       ment that guess-coding is a must.  Use this module with care.
151

TO DO

153       Encode::Guess does not work on EBCDIC platforms.
154

NAME

SYNOPSIS

ABSTRACT

DESCRIPTION

CAVEATS

TO DO

SEE ALSO