1PERLUNITUT(1) Perl Programmers Reference Guide PERLUNITUT(1)
2
6 perlunitut - Perl Unicode Tutorial
7
9 The days of just flinging strings around are over. It's well
10 established that modern programs need to be capable of communicating
11 funny accented letters, and things like euro symbols. This means that
12 programmers need new habits. It's easy to program Unicode capable
13 software, but it does require discipline to do it right.
14 There's a lot to know about character sets, and text encodings. It's
16 probably best to spend a full day learning all this, but the basics can
17 be learned in minutes.
18 These are not the very basics, though. It is assumed that you already
20 know the difference between bytes and characters, and realise (and
21 accept!) that there are many different character sets and encodings,
22 and that your program has to be explicit about them. Recommended
23 reading is "The Absolute Minimum Every Software Developer Absolutely,
24 Positively Must Know About Unicode and Character Sets (No Excuses!)" by
25 Joel Spolsky, at <http://joelonsoftware.com/articles/Unicode.html>.
26 This tutorial speaks in rather absolute terms, and provides only a
28 limited view of the wealth of character string related features that
29 Perl has to offer. For most projects, this information will probably
30 suffice.
31 Definitions
33 It's important to set a few things straight first. This is the most
34 important part of this tutorial. This view may conflict with other
35 information that you may have found on the web, but that's mostly
36 because many sources are wrong.
37 You may have to re-read this entire section a few times...
39 Unicode
41 Unicode is a character set with room for lots of characters. The
43 ordinal value of a character is called a code point.
44 There are many, many code points, but computers work with bytes, and a
46 byte can have only 256 values. Unicode has many more characters, so you
47 need a method to make these accessible.
48 Unicode is encoded using several competing encodings, of which UTF-8 is
50 the most used. In a Unicode encoding, multiple subsequent bytes can be
51 used to store a single code point, or simply: character.
52 UTF-8
54 UTF-8 is a Unicode encoding. Many people think that Unicode and UTF-8
56 are the same thing, but they're not. There are more Unicode encodings,
57 but much of the world has standardized on UTF-8.
58 UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They
60 take only one byte per character. All other characters are encoded as
61 two or more (up to six) bytes using a complex scheme. Fortunately, Perl
62 handles this for us, so we don't have to worry about this.
63 Text strings (character strings)
65 Text strings, or character strings are made of characters. Bytes are
67 irrelevant here, and so are encodings. Each character is just that: the
68 character.
69 Text strings are also called Unicode strings, because in Perl, every
71 text string is a Unicode string.
72 On a text string, you would do things like:
74 $text =~ s/foo/bar/;
76 if ($string =~ /^\d+$/) { ... }
77 $text = ucfirst $text;
78 my $character_count = length $text;
79 The value of a character ("ord", "chr") is the corresponding Unicode
81 code point.
82 Binary strings (byte strings)
84 Binary strings, or byte strings are made of bytes. Here, you don't have
86 characters, just bytes. All communication with the outside world
87 (anything outside of your current Perl process) is done in binary.
88 On a binary string, you would do things like:
90 my (@length_content) = unpack "(V/a)*", $binary;
92 $binary =~ s/\x00\x0F/\xFF\xF0/; # for the brave :)
93 print {$fh} $binary;
94 my $byte_count = length $binary;
95 Encoding
97 Encoding (as a verb) is the conversion from text to binary. To encode,
99 you have to supply the target encoding, for example "iso-8859-1" or
100 "UTF-8". Some encodings, like the "iso-8859" ("latin") range, do not
101 support the full Unicode standard; characters that can't be represented
102 are lost in the conversion.
103 Decoding
105 Decoding is the conversion from binary to text. To decode, you have to
107 know what encoding was used during the encoding phase. And most of all,
108 it must be something decodable. It doesn't make much sense to decode a
109 PNG image into a text string.
110 Internal format
112 Perl has an internal format, an encoding that it uses to encode text
114 strings so it can store them in memory. All text strings are in this
115 internal format. In fact, text strings are never in any other format!
116 You shouldn't worry about what this format is, because conversion is
118 automatically done when you decode or encode.
119 Your new toolkit
121 Add to your standard heading the following line:
122 use Encode qw(encode decode);
124 Or, if you're lazy, just:
126 use Encode;
128 I/O flow (the actual 5 minute tutorial)
130 The typical input/output flow of a program is:
131 1. Receive and decode
133 2. Process
134 3. Encode and output
135 If your input is binary, and is supposed to remain binary, you
137 shouldn't decode it to a text string, of course. But in all other
138 cases, you should decode it.
139 Decoding can't happen reliably if you don't know how the data was
141 encoded. If you get to choose, it's a good idea to standardize on
142 UTF-8.
143 my $foo = decode('UTF-8', get 'http://example.com/');
145 my $bar = decode('ISO-8859-1', readline STDIN);
146 my $xyzzy = decode('Windows-1251', $cgi->param('foo'));
147 Processing happens as you knew before. The only difference is that
149 you're now using characters instead of bytes. That's very useful if you
150 use things like "substr", or "length".
151 It's important to realize that there are no bytes in a text string. Of
153 course, Perl has its internal encoding to store the string in memory,
154 but ignore that. If you have to do anything with the number of bytes,
155 it's probably best to move that part to step 3, just after you've
156 encoded the string. Then you know exactly how many bytes it will be in
157 the destination string.
158 The syntax for encoding text strings to binary strings is as simple as
160 decoding:
161 $body = encode('UTF-8', $body);
163 If you needed to know the length of the string in bytes, now's the
165 perfect time for that. Because $body is now a byte string, "length"
166 will report the number of bytes, instead of the number of characters.
167 The number of characters is no longer known, because characters only
168 exist in text strings.
169 my $byte_count = length $body;
171 And if the protocol you're using supports a way of letting the
173 recipient know which character encoding you used, please help the
174 receiving end by using that feature! For example, E-mail and HTTP
175 support MIME headers, so you can use the "Content-Type" header. They
176 can also have "Content-Length" to indicate the number of bytes, which
177 is always a good idea to supply if the number is known.
178 "Content-Type: text/plain; charset=UTF-8",
180 "Content-Length: $byte_count"
181
183 Decode everything you receive, encode everything you send out. (If it's
184 text data.)
185
187 After reading this document, you ought to read perlunifaq too.
188
190 Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants
191 during the Amsterdam Perl Mongers meetings got me interested and
192 determined to find out how to use character encodings in Perl in ways
193 that don't break easily.
194 Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild"
196 (Dutch Perl Workshop 2006) inspired me to publish my thoughts and write
197 this tutorial.
198 Thanks to the people who asked about this kind of stuff in several Perl
200 IRC channels, and have constantly reminded me that a simpler
201 explanation was needed.
202 Thanks to the people who reviewed this document for me, before it went
204 public. They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans,
205 Lukas Mai, Nathan Gray.
206
208 Juerd Waalboer <#####@juerd.nl>
209
211 perlunifaq, perlunicode, perluniintro, Encode
212perl v5.10.1 2009-02-12 PERLUNITUT(1)