1PERLUNITUT(1)          Perl Programmers Reference Guide          PERLUNITUT(1)
2
3
4

NAME

6       perlunitut - Perl Unicode Tutorial
7

DESCRIPTION

9       The days of just flinging strings around are over. It's well
10       established that modern programs need to be capable of communicating
11       funny accented letters, and things like euro symbols. This means that
12       programmers need new habits. It's easy to program Unicode capable
13       software, but it does require discipline to do it right.
14
15       There's a lot to know about character sets, and text encodings. It's
16       probably best to spend a full day learning all this, but the basics can
17       be learned in minutes.
18
19       These are not the very basics, though. It is assumed that you already
20       know the difference between bytes and characters, and realise (and
21       accept!)  that there are many different character sets and encodings,
22       and that your program has to be explicit about them. Recommended
23       reading is "The Absolute Minimum Every Software Developer Absolutely,
24       Positively Must Know About Unicode and Character Sets (No Excuses!)" by
25       Joel Spolsky, at <http://joelonsoftware.com/articles/Unicode.html>.
26
27       This tutorial speaks in rather absolute terms, and provides only a
28       limited view of the wealth of character string related features that
29       Perl has to offer. For most projects, this information will probably
30       suffice.
31
32   Definitions
33       It's important to set a few things straight first. This is the most
34       important part of this tutorial. This view may conflict with other
35       information that you may have found on the web, but that's mostly
36       because many sources are wrong.
37
38       You may have to re-read this entire section a few times...
39
40       Unicode
41
42       Unicode is a character set with room for lots of characters. The
43       ordinal value of a character is called a code point.   (But in
44       practice, the distinction between code point and character is blurred,
45       so the terms often are used interchangeably.)
46
47       There are many, many code points, but computers work with bytes, and a
48       byte has room for only 256 values.  Unicode has many more characters
49       than that, so you need a method to make these accessible.
50
51       Unicode is encoded using several competing encodings, of which UTF-8 is
52       the most used. In a Unicode encoding, multiple subsequent bytes can be
53       used to store a single code point, or simply: character.
54
55       UTF-8
56
57       UTF-8 is a Unicode encoding. Many people think that Unicode and UTF-8
58       are the same thing, but they're not. There are more Unicode encodings,
59       but much of the world has standardized on UTF-8.
60
61       UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They
62       take only one byte per character. All other characters are encoded as
63       two or more (up to six) bytes using a complex scheme. Fortunately, Perl
64       handles this for us, so we don't have to worry about this.
65
66       Text strings (character strings)
67
68       Text strings, or character strings are made of characters. Bytes are
69       irrelevant here, and so are encodings. Each character is just that: the
70       character.
71
72       On a text string, you would do things like:
73
74           $text =~ s/foo/bar/;
75           if ($string =~ /^\d+$/) { ... }
76           $text = ucfirst $text;
77           my $character_count = length $text;
78
79       The value of a character ("ord", "chr") is the corresponding Unicode
80       code point.
81
82       Binary strings (byte strings)
83
84       Binary strings, or byte strings are made of bytes. Here, you don't have
85       characters, just bytes. All communication with the outside world
86       (anything outside of your current Perl process) is done in binary.
87
88       On a binary string, you would do things like:
89
90           my (@length_content) = unpack "(V/a)*", $binary;
91           $binary =~ s/\x00\x0F/\xFF\xF0/;  # for the brave :)
92           print {$fh} $binary;
93           my $byte_count = length $binary;
94
95       Encoding
96
97       Encoding (as a verb) is the conversion from text to binary. To encode,
98       you have to supply the target encoding, for example "iso-8859-1" or
99       "UTF-8".  Some encodings, like the "iso-8859" ("latin") range, do not
100       support the full Unicode standard; characters that can't be represented
101       are lost in the conversion.
102
103       Decoding
104
105       Decoding is the conversion from binary to text. To decode, you have to
106       know what encoding was used during the encoding phase. And most of all,
107       it must be something decodable. It doesn't make much sense to decode a
108       PNG image into a text string.
109
110       Internal format
111
112       Perl has an internal format, an encoding that it uses to encode text
113       strings so it can store them in memory. All text strings are in this
114       internal format.  In fact, text strings are never in any other format!
115
116       You shouldn't worry about what this format is, because conversion is
117       automatically done when you decode or encode.
118
119   Your new toolkit
120       Add to your standard heading the following line:
121
122           use Encode qw(encode decode);
123
124       Or, if you're lazy, just:
125
126           use Encode;
127
128   I/O flow (the actual 5 minute tutorial)
129       The typical input/output flow of a program is:
130
131           1. Receive and decode
132           2. Process
133           3. Encode and output
134
135       If your input is binary, and is supposed to remain binary, you
136       shouldn't decode it to a text string, of course. But in all other
137       cases, you should decode it.
138
139       Decoding can't happen reliably if you don't know how the data was
140       encoded. If you get to choose, it's a good idea to standardize on
141       UTF-8.
142
143           my $foo   = decode('UTF-8', get 'http://example.com/');
144           my $bar   = decode('ISO-8859-1', readline STDIN);
145           my $xyzzy = decode('Windows-1251', $cgi->param('foo'));
146
147       Processing happens as you knew before. The only difference is that
148       you're now using characters instead of bytes. That's very useful if you
149       use things like "substr", or "length".
150
151       It's important to realize that there are no bytes in a text string. Of
152       course, Perl has its internal encoding to store the string in memory,
153       but ignore that.  If you have to do anything with the number of bytes,
154       it's probably best to move that part to step 3, just after you've
155       encoded the string. Then you know exactly how many bytes it will be in
156       the destination string.
157
158       The syntax for encoding text strings to binary strings is as simple as
159       decoding:
160
161           $body = encode('UTF-8', $body);
162
163       If you needed to know the length of the string in bytes, now's the
164       perfect time for that. Because $body is now a byte string, "length"
165       will report the number of bytes, instead of the number of characters.
166       The number of characters is no longer known, because characters only
167       exist in text strings.
168
169           my $byte_count = length $body;
170
171       And if the protocol you're using supports a way of letting the
172       recipient know which character encoding you used, please help the
173       receiving end by using that feature! For example, E-mail and HTTP
174       support MIME headers, so you can use the "Content-Type" header. They
175       can also have "Content-Length" to indicate the number of bytes, which
176       is always a good idea to supply if the number is known.
177
178           "Content-Type: text/plain; charset=UTF-8",
179           "Content-Length: $byte_count"
180

SUMMARY

182       Decode everything you receive, encode everything you send out. (If it's
183       text data.)
184

Q and A (or FAQ)

186       After reading this document, you ought to read perlunifaq too.
187

ACKNOWLEDGEMENTS

189       Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants
190       during the Amsterdam Perl Mongers meetings got me interested and
191       determined to find out how to use character encodings in Perl in ways
192       that don't break easily.
193
194       Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild"
195       (Dutch Perl Workshop 2006) inspired me to publish my thoughts and write
196       this tutorial.
197
198       Thanks to the people who asked about this kind of stuff in several Perl
199       IRC channels, and have constantly reminded me that a simpler
200       explanation was needed.
201
202       Thanks to the people who reviewed this document for me, before it went
203       public.  They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans,
204       Lukas Mai, Nathan Gray.
205

AUTHOR

207       Juerd Waalboer <#####@juerd.nl>
208

SEE ALSO

210       perlunifaq, perlunicode, perluniintro, Encode
211
212
213
214perl v5.16.3                      2013-03-04                     PERLUNITUT(1)
Impressum