1PERLUNIFAQ(1)          Perl Programmers Reference Guide          PERLUNIFAQ(1)
2
3
4

NAME

6       perlunifaq - Perl Unicode FAQ
7

Q and A

9       This is a list of questions and answers about Unicode in Perl, intended
10       to be read after perlunitut.
11
12   perlunitut isn't really a Unicode tutorial, is it?
13       No, and this isn't really a Unicode FAQ.
14
15       Perl has an abstracted interface for all supported character encodings,
16       so this is actually a generic "Encode" tutorial and "Encode" FAQ. But
17       many people think that Unicode is special and magical, and I didn't
18       want to disappoint them, so I decided to call the document a Unicode
19       tutorial.
20
21   What character encodings does Perl support?
22       To find out which character encodings your Perl supports, run:
23
24           perl -MEncode -le "print for Encode->encodings(':all')"
25
26   Which version of perl should I use?
27       Well, if you can, upgrade to the most recent, but certainly 5.8.1 or
28       newer.  The tutorial and FAQ assume the latest release.
29
30       You should also check your modules, and upgrade them if necessary. For
31       example, HTML::Entities requires version >= 1.32 to function correctly,
32       even though the changelog is silent about this.
33
34   What about binary data, like images?
35       Well, apart from a bare "binmode $fh", you shouldn't treat them
36       specially.  (The binmode is needed because otherwise Perl may convert
37       line endings on Win32 systems.)
38
39       Be careful, though, to never combine text strings with binary strings.
40       If you need text in a binary stream, encode your text strings first
41       using the appropriate encoding, then join them with binary strings. See
42       also: "What if I don't encode?".
43
44   When should I decode or encode?
45       Whenever you're communicating text with anything that is external to
46       your perl process, like a database, a text file, a socket, or another
47       program. Even if the thing you're communicating with is also written in
48       Perl.
49
50   What if I don't decode?
51       Whenever your encoded, binary string is used together with a text
52       string, Perl will assume that your binary string was encoded with
53       ISO-8859-1, also known as latin-1. If it wasn't latin-1, then your data
54       is unpleasantly converted. For example, if it was UTF-8, the individual
55       bytes of multibyte characters are seen as separate characters, and then
56       again converted to UTF-8. Such double encoding can be compared to
57       double HTML encoding (">"), or double URI encoding (%253E).
58
59       This silent implicit decoding is known as "upgrading". That may sound
60       positive, but it's best to avoid it.
61
62   What if I don't encode?
63       It depends on what you output and how you output it.
64
65       Output via a filehandle
66
67       •   If the string's characters are all code point 255 or lower, Perl
68           outputs bytes that match those code points. This is what happens
69           with encoded strings. It can also, though, happen with unencoded
70           strings that happen to be all code point 255 or lower.
71
72       •   Otherwise, Perl outputs the string encoded as UTF-8. This only
73           happens with strings you neglected to encode. Since that should not
74           happen, Perl also throws a "wide character" warning in this case.
75
76       Other output mechanisms (e.g., "exec", "chdir", ..)
77
78       Your text string will be sent using the bytes in Perl's internal
79       format.
80
81       Because the internal format is often UTF-8, these bugs are hard to
82       spot, because UTF-8 is usually the encoding you wanted! But don't be
83       lazy, and don't use the fact that Perl's internal format is UTF-8 to
84       your advantage. Encode explicitly to avoid weird bugs, and to show to
85       maintenance programmers that you thought this through.
86
87   Is there a way to automatically decode or encode?
88       If all data that comes from a certain handle is encoded in exactly the
89       same way, you can tell the PerlIO system to automatically decode
90       everything, with the "encoding" layer. If you do this, you can't
91       accidentally forget to decode or encode anymore, on things that use the
92       layered handle.
93
94       You can provide this layer when "open"ing the file:
95
96         open my $fh, '>:encoding(UTF-8)', $filename;  # auto encoding on write
97         open my $fh, '<:encoding(UTF-8)', $filename;  # auto decoding on read
98
99       Or if you already have an open filehandle:
100
101         binmode $fh, ':encoding(UTF-8)';
102
103       Some database drivers for DBI can also automatically encode and decode,
104       but that is sometimes limited to the UTF-8 encoding.
105
106   What if I don't know which encoding was used?
107       Do whatever you can to find out, and if you have to: guess. (Don't
108       forget to document your guess with a comment.)
109
110       You could open the document in a web browser, and change the character
111       set or character encoding until you can visually confirm that all
112       characters look the way they should.
113
114       There is no way to reliably detect the encoding automatically, so if
115       people keep sending you data without charset indication, you may have
116       to educate them.
117
118   Can I use Unicode in my Perl sources?
119       Yes, you can! If your sources are UTF-8 encoded, you can indicate that
120       with the "use utf8" pragma.
121
122           use utf8;
123
124       This doesn't do anything to your input, or to your output. It only
125       influences the way your sources are read. You can use Unicode in string
126       literals, in identifiers (but they still have to be "word characters"
127       according to "\w"), and even in custom delimiters.
128
129   Data::Dumper doesn't restore the UTF8 flag; is it broken?
130       No, Data::Dumper's Unicode abilities are as they should be. There have
131       been some complaints that it should restore the UTF8 flag when the data
132       is read again with "eval". However, you should really not look at the
133       flag, and nothing indicates that Data::Dumper should break this rule.
134
135       Here's what happens: when Perl reads in a string literal, it sticks to
136       8 bit encoding as long as it can. (But perhaps originally it was
137       internally encoded as UTF-8, when you dumped it.) When it has to give
138       that up because other characters are added to the text string, it
139       silently upgrades the string to UTF-8.
140
141       If you properly encode your strings for output, none of this is of your
142       concern, and you can just "eval" dumped data as always.
143
144   Why do regex character classes sometimes match only in the ASCII range?
145       Starting in Perl 5.14 (and partially in Perl 5.12), just put a "use
146       feature 'unicode_strings'" near the beginning of your program.  Within
147       its lexical scope you shouldn't have this problem.  It also is
148       automatically enabled under "use feature ':5.12'" or "use v5.12" or
149       using "-E" on the command line for Perl 5.12 or higher.
150
151       The rationale for requiring this is to not break older programs that
152       rely on the way things worked before Unicode came along.  Those older
153       programs knew only about the ASCII character set, and so may not work
154       properly for additional characters.  When a string is encoded in UTF-8,
155       Perl assumes that the program is prepared to deal with Unicode, but
156       when the string isn't, Perl assumes that only ASCII is wanted, and so
157       those characters that are not ASCII characters aren't recognized as to
158       what they would be in Unicode.  "use feature 'unicode_strings'" tells
159       Perl to treat all characters as Unicode, whether the string is encoded
160       in UTF-8 or not, thus avoiding the problem.
161
162       However, on earlier Perls, or if you pass strings to subroutines
163       outside the feature's scope, you can force Unicode rules by changing
164       the encoding to UTF-8 by doing utf8::upgrade($string). This can be used
165       safely on any string, as it checks and does not change strings that
166       have already been upgraded.
167
168       For a more detailed discussion, see Unicode::Semantics on CPAN.
169
170   Why do some characters not uppercase or lowercase correctly?
171       See the answer to the previous question.
172
173   How can I determine if a string is a text string or a binary string?
174       You can't. Some use the UTF8 flag for this, but that's misuse, and
175       makes well behaved modules like Data::Dumper look bad. The flag is
176       useless for this purpose, because it's off when an 8 bit encoding (by
177       default ISO-8859-1) is used to store the string.
178
179       This is something you, the programmer, has to keep track of; sorry. You
180       could consider adopting a kind of "Hungarian notation" to help with
181       this.
182
183   How do I convert from encoding FOO to encoding BAR?
184       By first converting the FOO-encoded byte string to a text string, and
185       then the text string to a BAR-encoded byte string:
186
187           my $text_string = decode('FOO', $foo_string);
188           my $bar_string  = encode('BAR', $text_string);
189
190       or by skipping the text string part, and going directly from one binary
191       encoding to the other:
192
193           use Encode qw(from_to);
194           from_to($string, 'FOO', 'BAR');  # changes contents of $string
195
196       or by letting automatic decoding and encoding do all the work:
197
198           open my $foofh, '<:encoding(FOO)', 'example.foo.txt';
199           open my $barfh, '>:encoding(BAR)', 'example.bar.txt';
200           print { $barfh } $_ while <$foofh>;
201
202   What are "decode_utf8" and "encode_utf8"?
203       These are alternate syntaxes for "decode('utf8', ...)" and
204       "encode('utf8', ...)". Do not use these functions for data exchange.
205       Instead use "decode('UTF-8', ...)" and "encode('UTF-8', ...)"; see
206       "What's the difference between UTF-8 and utf8?" below.
207
208   What is a "wide character"?
209       This is a term used for characters occupying more than one byte.
210
211       The Perl warning "Wide character in ..." is caused by such a character.
212       With no specified encoding layer, Perl tries to fit things into a
213       single byte.  When it can't, it emits this warning (if warnings are
214       enabled), and uses UTF-8 encoded data instead.
215
216       To avoid this warning and to avoid having different output encodings in
217       a single stream, always specify an encoding explicitly, for example
218       with a PerlIO layer:
219
220           binmode STDOUT, ":encoding(UTF-8)";
221

INTERNALS

223   What is "the UTF8 flag"?
224       Please, unless you're hacking the internals, or debugging weirdness,
225       don't think about the UTF8 flag at all. That means that you very
226       probably shouldn't use "is_utf8", "_utf8_on" or "_utf8_off" at all.
227
228       The UTF8 flag, also called SvUTF8, is an internal flag that indicates
229       that the current internal representation is UTF-8. Without the flag, it
230       is assumed to be ISO-8859-1. Perl converts between these automatically.
231       (Actually Perl usually assumes the representation is ASCII; see "Why do
232       regex character classes sometimes match only in the ASCII range?"
233       above.)
234
235       One of Perl's internal formats happens to be UTF-8. Unfortunately, Perl
236       can't keep a secret, so everyone knows about this. That is the source
237       of much confusion. It's better to pretend that the internal format is
238       some unknown encoding, and that you always have to encode and decode
239       explicitly.
240
241   What about the "use bytes" pragma?
242       Don't use it. It makes no sense to deal with bytes in a text string,
243       and it makes no sense to deal with characters in a byte string. Do the
244       proper conversions (by decoding/encoding), and things will work out
245       well: you get character counts for decoded data, and byte counts for
246       encoded data.
247
248       "use bytes" is usually a failed attempt to do something useful. Just
249       forget about it.
250
251   What about the "use encoding" pragma?
252       Don't use it. Unfortunately, it assumes that the programmer's
253       environment and that of the user will use the same encoding. It will
254       use the same encoding for the source code and for STDIN and STDOUT.
255       When a program is copied to another machine, the source code does not
256       change, but the STDIO environment might.
257
258       If you need non-ASCII characters in your source code, make it a UTF-8
259       encoded file and "use utf8".
260
261       If you need to set the encoding for STDIN, STDOUT, and STDERR, for
262       example based on the user's locale, "use open".
263
264   What is the difference between ":encoding" and ":utf8"?
265       Because UTF-8 is one of Perl's internal formats, you can often just
266       skip the encoding or decoding step, and manipulate the UTF8 flag
267       directly.
268
269       Instead of :encoding(UTF-8), you can simply use ":utf8", which skips
270       the encoding step if the data was already represented as UTF8
271       internally. This is widely accepted as good behavior when you're
272       writing, but it can be dangerous when reading, because it causes
273       internal inconsistency when you have invalid byte sequences. Using
274       ":utf8" for input can sometimes result in security breaches, so please
275       use :encoding(UTF-8) instead.
276
277       Instead of "decode" and "encode", you could use "_utf8_on" and
278       "_utf8_off", but this is considered bad style. Especially "_utf8_on"
279       can be dangerous, for the same reason that ":utf8" can.
280
281       There are some shortcuts for oneliners; see -C in perlrun.
282
283   What's the difference between "UTF-8" and "utf8"?
284       "UTF-8" is the official standard. "utf8" is Perl's way of being liberal
285       in what it accepts. If you have to communicate with things that aren't
286       so liberal, you may want to consider using "UTF-8". If you have to
287       communicate with things that are too liberal, you may have to use
288       "utf8". The full explanation is in "UTF-8 vs. utf8 vs. UTF8" in Encode.
289
290       "UTF-8" is internally known as "utf-8-strict". The tutorial uses UTF-8
291       consistently, even where utf8 is actually used internally, because the
292       distinction can be hard to make, and is mostly irrelevant.
293
294       For example, utf8 can be used for code points that don't exist in
295       Unicode, like 9999999, but if you encode that to UTF-8, you get a
296       substitution character (by default; see "Handling Malformed Data" in
297       Encode for more ways of dealing with this.)
298
299       Okay, if you insist: the "internal format" is utf8, not UTF-8. (When
300       it's not some other encoding.)
301
302   I lost track; what encoding is the internal format really?
303       It's good that you lost track, because you shouldn't depend on the
304       internal format being any specific encoding. But since you asked: by
305       default, the internal format is either ISO-8859-1 (latin-1), or utf8,
306       depending on the history of the string. On EBCDIC platforms, this may
307       be different even.
308
309       Perl knows how it stored the string internally, and will use that
310       knowledge when you "encode". In other words: don't try to find out what
311       the internal encoding for a certain string is, but instead just encode
312       it into the encoding that you want.
313

AUTHOR

315       Juerd Waalboer <#####@juerd.nl>
316

SEE ALSO

318       perlunicode, perluniintro, Encode
319
320
321
322perl v5.38.2                      2023-11-30                     PERLUNIFAQ(1)
Impressum