1Encode(3pm) Perl Programmers Reference Guide Encode(3pm)
2
6 Encode - character encodings
7
9 use Encode;
10 Table of Contents
12 Encode consists of a collection of modules whose details are too big to
13 fit in one document. This POD itself explains the top-level APIs and
14 general topics at a glance. For other topics and more details, see the
15 PODs below:
16 Name Description
18 --------------------------------------------------------
19 Encode::Alias Alias definitions to encodings
20 Encode::Encoding Encode Implementation Base Class
21 Encode::Supported List of Supported Encodings
22 Encode::CN Simplified Chinese Encodings
23 Encode::JP Japanese Encodings
24 Encode::KR Korean Encodings
25 Encode::TW Traditional Chinese Encodings
26 --------------------------------------------------------
27
29 The "Encode" module provides the interfaces between Perl's strings and
30 the rest of the system. Perl strings are sequences of characters.
31 The repertoire of characters that Perl can represent is at least that
33 defined by the Unicode Consortium. On most platforms the ordinal values
34 of the characters (as returned by "ord(ch)") is the "Unicode codepoint"
35 for the character (the exceptions are those platforms where the legacy
36 encoding is some variant of EBCDIC rather than a super-set of ASCII -
37 see perlebcdic).
38 Traditionally, computer data has been moved around in 8-bit chunks
40 often called "bytes". These chunks are also known as "octets" in
41 networking standards. Perl is widely used to manipulate data of many
42 types - not only strings of characters representing human or computer
43 languages but also "binary" data being the machine's representation of
44 numbers, pixels in an image - or just about anything.
45 When Perl is processing "binary data", the programmer wants Perl to
47 process "sequences of bytes". This is not a problem for Perl - as a
48 byte has 256 possible values, it easily fits in Perl's much larger
49 "logical character".
50 TERMINOLOGY
52 · character: a character in the range 0..(2**32-1) (or more). (What
53 Perl's strings are made of.)
54 · byte: a character in the range 0..255 (A special case of a Perl
56 character.)
57 · octet: 8 bits of data, with ordinal values 0..255 (Term for bytes
59 passed to or from a non-Perl context, e.g. a disk file.)
60
62 $octets = encode(ENCODING, $string [, CHECK])
63 Encodes a string from Perl's internal form into ENCODING and returns
64 a sequence of octets. ENCODING can be either a canonical name or an
65 alias. For encoding names and aliases, see "Defining Aliases". For
66 CHECK, see "Handling Malformed Data".
67 For example, to convert a string from Perl's internal format to
69 iso-8859-1 (also known as Latin1),
70 $octets = encode("iso-8859-1", $string);
72 CAVEAT: When you run "$octets = encode("utf8", $string)", then
74 $octets may not be equal to $string. Though they both contain the
75 same data, the UTF8 flag for $octets is always off. When you encode
76 anything, UTF8 flag of the result is always off, even when it
77 contains completely valid utf8 string. See "The UTF8 flag" below.
78 If the $string is "undef" then "undef" is returned.
80 $string = decode(ENCODING, $octets [, CHECK])
82 Decodes a sequence of octets assumed to be in ENCODING into Perl's
83 internal form and returns the resulting string. As in encode(),
84 ENCODING can be either a canonical name or an alias. For encoding
85 names and aliases, see "Defining Aliases". For CHECK, see "Handling
86 Malformed Data".
87 For example, to convert ISO-8859-1 data to a string in Perl's
89 internal format:
90 $string = decode("iso-8859-1", $octets);
92 CAVEAT: When you run "$string = decode("utf8", $octets)", then
94 $string may not be equal to $octets. Though they both contain the
95 same data, the UTF8 flag for $string is on unless $octets entirely
96 consists of ASCII data (or EBCDIC on EBCDIC machines). See "The UTF8
97 flag" below.
98 If the $string is "undef" then "undef" is returned.
100 [$obj =] find_encoding(ENCODING)
102 Returns the encoding object corresponding to ENCODING. Returns undef
103 if no matching ENCODING is find.
104 This object is what actually does the actual (en|de)coding.
106 $utf8 = decode($name, $bytes);
108 is in fact
110 $utf8 = do{
112 $obj = find_encoding($name);
113 croak qq(encoding "$name" not found) unless ref $obj;
114 $obj->decode($bytes)
115 };
116 with more error checking.
118 Therefore you can save time by reusing this object as follows;
120 my $enc = find_encoding("iso-8859-1");
122 while(<>){
123 my $utf8 = $enc->decode($_);
124 # and do someting with $utf8;
125 }
126 Besides "->decode" and "->encode", other methods are available as
128 well. For instance, "-> name" returns the canonical name of the
129 encoding object.
130 find_encoding("latin1")->name; # iso-8859-1
132 See Encode::Encoding for details.
134 [$length =] from_to($octets, FROM_ENC, TO_ENC [, CHECK])
136 Converts in-place data between two encodings. The data in $octets
137 must be encoded as octets and not as characters in Perl's internal
138 format. For example, to convert ISO-8859-1 data to Microsoft's CP1250
139 encoding:
140 from_to($octets, "iso-8859-1", "cp1250");
142 and to convert it back:
144 from_to($octets, "cp1250", "iso-8859-1");
146 Note that because the conversion happens in place, the data to be
148 converted cannot be a string constant; it must be a scalar variable.
149 from_to() returns the length of the converted string in octets on
151 success, undef on error.
152 CAVEAT: The following operations look the same but are not quite so;
154 from_to($data, "iso-8859-1", "utf8"); #1
156 $data = decode("iso-8859-1", $data); #2
157 Both #1 and #2 make $data consist of a completely valid UTF-8 string
159 but only #2 turns UTF8 flag on. #1 is equivalent to
160 $data = encode("utf8", decode("iso-8859-1", $data));
162 See "The UTF8 flag" below.
164 Also note that
166 from_to($octets, $from, $to, $check);
168 is equivalent to
170 $octets = encode($to, decode($from, $octets), $check);
172 Yes, it does not respect the $check during decoding. It is
174 deliberately done that way. If you need minute control, "decode"
175 then "encode" as follows;
176 $octets = encode($to, decode($from, $octets, $check_from), $check_to);
178 $octets = encode_utf8($string);
180 Equivalent to "$octets = encode("utf8", $string);" The characters
181 that comprise $string are encoded in Perl's internal format and the
182 result is returned as a sequence of octets. All possible characters
183 have a UTF-8 representation so this function cannot fail.
184 $string = decode_utf8($octets [, CHECK]);
186 equivalent to "$string = decode("utf8", $octets [, CHECK])". The
187 sequence of octets represented by $octets is decoded from UTF-8 into
188 a sequence of logical characters. Not all sequences of octets form
189 valid UTF-8 encodings, so it is possible for this call to fail. For
190 CHECK, see "Handling Malformed Data".
191 Listing available encodings
193 use Encode;
194 @list = Encode->encodings();
195 Returns a list of the canonical names of the available encodings that
197 are loaded. To get a list of all available encodings including the
198 ones that are not loaded yet, say
199 @all_encodings = Encode->encodings(":all");
201 Or you can give the name of a specific module.
203 @with_jp = Encode->encodings("Encode::JP");
205 When "::" is not in the name, "Encode::" is assumed.
207 @ebcdic = Encode->encodings("EBCDIC");
209 To find out in detail which encodings are supported by this package,
211 see Encode::Supported.
212 Defining Aliases
214 To add a new alias to a given encoding, use:
215 use Encode;
217 use Encode::Alias;
218 define_alias(newName => ENCODING);
219 After that, newName can be used as an alias for ENCODING. ENCODING may
221 be either the name of an encoding or an encoding object
222 But before you do so, make sure the alias is nonexistent with
224 "resolve_alias()", which returns the canonical name thereof. i.e.
225 Encode::resolve_alias("latin1") eq "iso-8859-1" # true
227 Encode::resolve_alias("iso-8859-12") # false; nonexistent
228 Encode::resolve_alias($name) eq $name # true if $name is canonical
229 resolve_alias() does not need "use Encode::Alias"; it can be exported
231 via "use Encode qw(resolve_alias)".
232 See Encode::Alias for details.
234 Finding IANA Character Set Registry names
236 The canonical name of a given encoding does not necessarily agree with
237 IANA IANA Character Set Registry, commonly seen as "Content-Type:
238 text/plain; charset=I<whatever>". For most cases canonical names work
239 but sometimes it does not (notably 'utf-8-strict').
240 Therefore as of Encode version 2.21, a new method "mime_name()" is
242 added.
243 use Encode;
245 my $enc = find_encoding('UTF-8');
246 warn $enc->name; # utf-8-strict
247 warn $enc->mime_name; # UTF-8
248 See also: Encode::Encoding
250
252 If your perl supports PerlIO (which is the default), you can use a
253 PerlIO layer to decode and encode directly via a filehandle. The
254 following two examples are totally identical in their functionality.
255 # via PerlIO
257 open my $in, "<:encoding(shiftjis)", $infile or die;
258 open my $out, ">:encoding(euc-jp)", $outfile or die;
259 while(<$in>){ print $out $_; }
260 # via from_to
262 open my $in, "<", $infile or die;
263 open my $out, ">", $outfile or die;
264 while(<$in>){
265 from_to($_, "shiftjis", "euc-jp", 1);
266 print $out $_;
267 }
268 Unfortunately, it may be that encodings are PerlIO-savvy. You can
270 check if your encoding is supported by PerlIO by calling the
271 "perlio_ok" method.
272 Encode::perlio_ok("hz"); # False
274 find_encoding("euc-cn")->perlio_ok; # True where PerlIO is available
275 use Encode qw(perlio_ok); # exported upon request
277 perlio_ok("euc-jp")
278 Fortunately, all encodings that come with Encode core are PerlIO-savvy
280 except for hz and ISO-2022-kr. For gory details, see Encode::Encoding
281 and Encode::PerlIO.
282
284 The optional CHECK argument tells Encode what to do when it encounters
285 malformed data. Without CHECK, Encode::FB_DEFAULT ( == 0 ) is assumed.
286 As of version 2.12 Encode supports coderef values for CHECK. See
288 below.
289 NOTE: Not all encoding support this feature
291 Some encodings ignore CHECK argument. For example, Encode::Unicode
292 ignores CHECK and it always croaks on error.
293 Now here is the list of CHECK values available
295 CHECK = Encode::FB_DEFAULT ( == 0)
297 If CHECK is 0, (en|de)code will put a substitution character in place
298 of a malformed character. When you encode, <subchar> will be used.
299 When you decode the code point 0xFFFD is used. If the data is
300 supposed to be UTF-8, an optional lexical warning (category utf8) is
301 given.
302 CHECK = Encode::FB_CROAK ( == 1)
304 If CHECK is 1, methods will die on error immediately with an error
305 message. Therefore, when CHECK is set to 1, you should trap the
306 error with eval{} unless you really want to let it die.
307 CHECK = Encode::FB_QUIET
309 If CHECK is set to Encode::FB_QUIET, (en|de)code will immediately
310 return the portion of the data that has been processed so far when an
311 error occurs. The data argument will be overwritten with everything
312 after that point (that is, the unprocessed part of data). This is
313 handy when you have to call decode repeatedly in the case where your
314 source data may contain partial multi-byte character sequences, (i.e.
315 you are reading with a fixed-width buffer). Here is a sample code
316 that does exactly this:
317 my $buffer = ''; my $string = '';
319 while(read $fh, $buffer, 256, length($buffer)){
320 $string .= decode($encoding, $buffer, Encode::FB_QUIET);
321 # $buffer now contains the unprocessed partial character
322 }
323 CHECK = Encode::FB_WARN
325 This is the same as above, except that it warns on error. Handy when
326 you are debugging the mode above.
327 perlqq mode (CHECK = Encode::FB_PERLQQ)
329 HTML charref mode (CHECK = Encode::FB_HTMLCREF)
330 XML charref mode (CHECK = Encode::FB_XMLCREF)
331 For encodings that are implemented by Encode::XS, CHECK ==
332 Encode::FB_PERLQQ turns (en|de)code into "perlqq" fallback mode.
333 When you decode, "\xHH" will be inserted for a malformed character,
335 where HH is the hex representation of the octet that could not be
336 decoded to utf8. And when you encode, "\x{HHHH}" will be inserted,
337 where HHHH is the Unicode ID of the character that cannot be found in
338 the character repertoire of the encoding.
339 HTML/XML character reference modes are about the same, in place of
341 "\x{HHHH}", HTML uses "&#NNN;" where NNN is a decimal number and XML
342 uses "&#xHHHH;" where HHHH is the hexadecimal number.
343 In Encode 2.10 or later, "LEAVE_SRC" is also implied.
345 The bitmask
347 These modes are actually set via a bitmask. Here is how the FB_XX
348 constants are laid out. You can import the FB_XX constants via "use
349 Encode qw(:fallbacks)"; you can import the generic bitmask constants
350 via "use Encode qw(:fallback_all)".
351 FB_DEFAULT FB_CROAK FB_QUIET FB_WARN FB_PERLQQ
353 DIE_ON_ERR 0x0001 X
354 WARN_ON_ERR 0x0002 X
355 RETURN_ON_ERR 0x0004 X X
356 LEAVE_SRC 0x0008 X
357 PERLQQ 0x0100 X
358 HTMLCREF 0x0200
359 XMLCREF 0x0400
360 Encode::LEAVE_SRC
362 If the "Encode::LEAVE_SRC" bit is not set, but CHECK is, then the
363 second argument to "encode()" or "decode()" may be assigned to by the
364 functions. If you're not interested in this, then bitwise-or the
365 bitmask with it.
366 coderef for CHECK
368 As of Encode 2.12 CHECK can also be a code reference which takes the
369 ord value of unmapped caharacter as an argument and returns a string
370 that represents the fallback character. For instance,
371 $ascii = encode("ascii", $utf8, sub{ sprintf "<U+%04X>", shift });
373 Acts like FB_PERLQQ but <U+XXXX> is used instead of \x{XXXX}.
375
377 To define a new encoding, use:
378 use Encode qw(define_encoding);
380 define_encoding($object, 'canonicalName' [, alias...]);
381 canonicalName will be associated with $object. The object should
383 provide the interface described in Encode::Encoding. If more than two
384 arguments are provided then additional arguments are taken as aliases
385 for $object.
386 See Encode::Encoding for more details.
388
390 Before the introduction of Unicode support in perl, The "eq" operator
391 just compared the strings represented by two scalars. Beginning with
392 perl 5.8, "eq" compares two strings with simultaneous consideration of
393 the UTF8 flag. To explain why we made it so, I will quote page 402 of
394 "Programming Perl, 3rd ed."
395 Goal #1:
397 Old byte-oriented programs should not spontaneously break on the old
398 byte-oriented data they used to work on.
399 Goal #2:
401 Old byte-oriented programs should magically start working on the new
402 character-oriented data when appropriate.
403 Goal #3:
405 Programs should run just as fast in the new character-oriented mode
406 as in the old byte-oriented mode.
407 Goal #4:
409 Perl should remain one language, rather than forking into a byte-
410 oriented Perl and a character-oriented Perl.
411 Back when "Programming Perl, 3rd ed." was written, not even Perl 5.6.0
413 was born and many features documented in the book remained
414 unimplemented for a long time. Perl 5.8 corrected this and the
415 introduction of the UTF8 flag is one of them. You can think of this
416 perl notion as of a byte-oriented mode (UTF8 flag off) and a character-
417 oriented mode (UTF8 flag on).
418 Here is how Encode takes care of the UTF8 flag.
420 · When you encode, the resulting UTF8 flag is always off.
422 · When you decode, the resulting UTF8 flag is on unless you can
424 unambiguously represent data. Here is the definition of dis-
425 ambiguity.
426 After "$utf8 = decode('foo', $octet);",
428 When $octet is... The UTF8 flag in $utf8 is
430 ---------------------------------------------
431 In ASCII only (or EBCDIC only) OFF
432 In ISO-8859-1 ON
433 In any other Encoding ON
434 ---------------------------------------------
435 As you see, there is one exception, In ASCII. That way you can
437 assume Goal #1. And with Encode Goal #2 is assumed but you still
438 have to be careful in such cases mentioned in CAVEAT paragraphs.
439 This UTF8 flag is not visible in perl scripts, exactly for the same
441 reason you cannot (or you don't have to) see if a scalar contains a
442 string, integer, or floating point number. But you can still peek
443 and poke these if you will. See the section below.
444 Messing with Perl's Internals
446 The following API uses parts of Perl's internals in the current
447 implementation. As such, they are efficient but may change.
448 is_utf8(STRING [, CHECK])
450 [INTERNAL] Tests whether the UTF8 flag is turned on in the STRING.
451 If CHECK is true, also checks the data in STRING for being well-
452 formed UTF-8. Returns true if successful, false otherwise.
453 As of perl 5.8.1, utf8 also has utf8::is_utf8().
455 _utf8_on(STRING)
457 [INTERNAL] Turns on the UTF8 flag in STRING. The data in STRING is
458 not checked for being well-formed UTF-8. Do not use unless you know
459 that the STRING is well-formed UTF-8. Returns the previous state of
460 the UTF8 flag (so please don't treat the return value as indicating
461 success or failure), or "undef" if STRING is not a string.
462 This function does not work on tainted values.
464 _utf8_off(STRING)
466 [INTERNAL] Turns off the UTF8 flag in STRING. Do not use
467 frivolously. Returns the previous state of the UTF8 flag (so please
468 don't treat the return value as indicating success or failure), or
469 "undef" if STRING is not a string.
470 This function does not work on tainted values.
472
474 ....We now view strings not as sequences of bytes, but as sequences
475 of numbers in the range 0 .. 2**32-1 (or in the case of 64-bit
476 computers, 0 .. 2**64-1) -- Programming Perl, 3rd ed.
477 That has been the perl's notion of UTF-8 but official UTF-8 is more
479 strict; Its ranges is much narrower (0 .. 10FFFF), some sequences are
480 not allowed (i.e. Those used in the surrogate pair, 0xFFFE, et al).
481 Now that is overruled by Larry Wall himself.
483 From: Larry Wall <larry@wall.org>
485 Date: December 04, 2004 11:51:58 JST
486 To: perl-unicode@perl.org
487 Subject: Re: Make Encode.pm support the real UTF-8
488 Message-Id: <20041204025158.GA28754@wall.org>
489 On Fri, Dec 03, 2004 at 10:12:12PM +0000, Tim Bunce wrote:
491 : I've no problem with 'utf8' being perl's unrestricted uft8 encoding,
492 : but "UTF-8" is the name of the standard and should give the
493 : corresponding behaviour.
494 For what it's worth, that's how I've always kept them straight in my
496 head.
497 Also for what it's worth, Perl 6 will mostly default to strict but
499 make it easy to switch back to lax.
500 Larry
502 Do you copy? As of Perl 5.8.7, UTF-8 means strict, official UTF-8
504 while utf8 means liberal, lax, version thereof. And Encode version
505 2.10 or later thus groks the difference between "UTF-8" and C"utf8".
506 encode("utf8", "\x{FFFF_FFFF}", 1); # okay
508 encode("UTF-8", "\x{FFFF_FFFF}", 1); # croaks
509 "UTF-8" in Encode is actually a canonical name for "utf-8-strict".
511 Yes, the hyphen between "UTF" and "8" is important. Without it Encode
512 goes "liberal"
513 find_encoding("UTF-8")->name # is 'utf-8-strict'
515 find_encoding("utf-8")->name # ditto. names are case insensitive
516 find_encoding("utf_8")->name # ditto. "_" are treated as "-"
517 find_encoding("UTF8")->name # is 'utf8'.
518 The UTF8 flag is internally called UTF8, without a hyphen. It indicates
520 whether a string is internally encoded as utf8, also without a hypen.
521
523 Encode::Encoding, Encode::Supported, Encode::PerlIO, encoding,
524 perlebcdic, "open" in perlfunc, perlunicode, perluniintro, perlunifaq,
525 perlunitut utf8, the Perl Unicode Mailing List <perl-unicode@perl.org>
526
528 This project was originated by Nick Ing-Simmons and later maintained by
529 Dan Kogai <dankogai@dan.co.jp>. See AUTHORS for a full list of people
530 involved. For any questions, use <perl-unicode@perl.org> so we can all
531 share.
532 While Dan Kogai retains the copyright as a maintainer, the credit
534 should go to all those involoved. See AUTHORS for those submitted
535 codes.
536
538 Copyright 2002-2006 Dan Kogai <dankogai@dan.co.jp>
539 This library is free software; you can redistribute it and/or modify it
541 under the same terms as Perl itself.
542perl v5.10.1 2009-07-13 Encode(3pm)