1PERLUNICODE(1) Perl Programmers Reference Guide PERLUNICODE(1)
2
6 perlunicode - Unicode support in Perl
7
9 Important Caveats
10 Unicode support is an extensive requirement. While Perl does not imple‐
12 ment the Unicode standard or the accompanying technical reports from
13 cover to cover, Perl does support many Unicode features.
14 Input and Output Layers
16 Perl knows when a filehandle uses Perl's internal Unicode encodings
17 (UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened
18 with the ":utf8" layer. Other encodings can be converted to Perl's
19 encoding on input or from Perl's encoding on output by use of the
20 ":encoding(...)" layer. See open.
21 To indicate that Perl source itself is using a particular encoding,
23 see encoding.
24 Regular Expressions
26 The regular expression compiler produces polymorphic opcodes. That
27 is, the pattern adapts to the data and automatically switches to
28 the Unicode character scheme when presented with Unicode data--or
29 instead uses a traditional byte scheme when presented with byte
30 data.
31 "use utf8" still needed to enable UTF-8/UTF-EBCDIC in scripts
33 As a compatibility measure, the "use utf8" pragma must be explic‐
34 itly included to enable recognition of UTF-8 in the Perl scripts
35 themselves (in string or regular expression literals, or in identi‐
36 fier names) on ASCII-based machines or to recognize UTF-EBCDIC on
37 EBCDIC-based machines. These are the only times when an explicit
38 "use utf8" is needed. See utf8.
39 You can also use the "encoding" pragma to change the default encod‐
41 ing of the data in your script; see encoding.
42 BOM-marked scripts and UTF-16 scripts autodetected
44 If a Perl script begins marked with the Unicode BOM (UTF-16LE,
45 UTF16-BE, or UTF-8), or if the script looks like non-BOM-marked
46 UTF-16 of either endianness, Perl will correctly read in the script
47 as Unicode. (BOMless UTF-8 cannot be effectively recognized or
48 differentiated from ISO 8859-1 or other eight-bit encodings.)
49 "use encoding" needed to upgrade non-Latin-1 byte strings
51 By default, there is a fundamental asymmetry in Perl's unicode
52 model: implicit upgrading from byte strings to Unicode strings
53 assumes that they were encoded in ISO 8859-1 (Latin-1), but Unicode
54 strings are downgraded with UTF-8 encoding. This happens because
55 the first 256 codepoints in Unicode happens to agree with Latin-1.
56 If you wish to interpret byte strings as UTF-8 instead, use the
58 "encoding" pragma:
59 use encoding 'utf8';
61 See "Byte and Character Semantics" for more details.
63 Byte and Character Semantics
65 Beginning with version 5.6, Perl uses logically-wide characters to rep‐
67 resent strings internally.
68 In future, Perl-level operations will be expected to work with charac‐
70 ters rather than bytes.
71 However, as an interim compatibility measure, Perl aims to provide a
73 safe migration path from byte semantics to character semantics for pro‐
74 grams. For operations where Perl can unambiguously decide that the
75 input data are characters, Perl switches to character semantics. For
76 operations where this determination cannot be made without additional
77 information from the user, Perl decides in favor of compatibility and
78 chooses to use byte semantics.
79 This behavior preserves compatibility with earlier versions of Perl,
81 which allowed byte semantics in Perl operations only if none of the
82 program's inputs were marked as being as source of Unicode character
83 data. Such data may come from filehandles, from calls to external pro‐
84 grams, from information provided by the system (such as %ENV), or from
85 literals and constants in the source text.
86 The "bytes" pragma will always, regardless of platform, force byte
88 semantics in a particular lexical scope. See bytes.
89 The "utf8" pragma is primarily a compatibility device that enables
91 recognition of UTF-(8⎪EBCDIC) in literals encountered by the parser.
92 Note that this pragma is only required while Perl defaults to byte
93 semantics; when character semantics become the default, this pragma may
94 become a no-op. See utf8.
95 Unless explicitly stated, Perl operators use character semantics for
97 Unicode data and byte semantics for non-Unicode data. The decision to
98 use character semantics is made transparently. If input data comes
99 from a Unicode source--for example, if a character encoding layer is
100 added to a filehandle or a literal Unicode string constant appears in a
101 program--character semantics apply. Otherwise, byte semantics are in
102 effect. The "bytes" pragma should be used to force byte semantics on
103 Unicode data.
104 If strings operating under byte semantics and strings with Unicode
106 character data are concatenated, the new string will be created by
107 decoding the byte strings as ISO 8859-1 (Latin-1), even if the old Uni‐
108 code string used EBCDIC. This translation is done without regard to
109 the system's native 8-bit encoding. To change this for systems with
110 non-Latin-1 and non-EBCDIC native encodings, use the "encoding" pragma.
111 See encoding.
112 Under character semantics, many operations that formerly operated on
114 bytes now operate on characters. A character in Perl is logically just
115 a number ranging from 0 to 2**31 or so. Larger characters may encode
116 into longer sequences of bytes internally, but this internal detail is
117 mostly hidden for Perl code. See perluniintro for more.
118 Effects of Character Semantics
120 Character semantics have the following effects:
122 · Strings--including hash keys--and regular expression patterns may
124 contain characters that have an ordinal value larger than 255.
125 If you use a Unicode editor to edit your program, Unicode charac‐
127 ters may occur directly within the literal strings in one of the
128 various Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but
129 will be recognized as such and converted to Perl's internal repre‐
130 sentation only if the appropriate encoding is specified.
131 Unicode characters can also be added to a string by using the
133 "\x{...}" notation. The Unicode code for the desired character, in
134 hexadecimal, should be placed in the braces. For instance, a smiley
135 face is "\x{263A}". This encoding scheme only works for characters
136 with a code of 0x100 or above.
137 Additionally, if you
139 use charnames ':full';
141 you can use the "\N{...}" notation and put the official Unicode
143 character name within the braces, such as "\N{WHITE SMILING FACE}".
144 · If an appropriate encoding is specified, identifiers within the
146 Perl script may contain Unicode alphanumeric characters, including
147 ideographs. Perl does not currently attempt to canonicalize vari‐
148 able names.
149 · Regular expressions match characters instead of bytes. "." matches
151 a character instead of a byte. The "\C" pattern is provided to
152 force a match a single byte--a "char" in C, hence "\C".
153 · Character classes in regular expressions match characters instead
155 of bytes and match against the character properties specified in
156 the Unicode properties database. "\w" can be used to match a Japa‐
157 nese ideograph, for instance.
158 (However, and as a limitation of the current implementation, using
160 "\w" or "\W" inside a "[...]" character class will still match with
161 byte semantics.)
162 · Named Unicode properties, scripts, and block ranges may be used
164 like character classes via the "\p{}" "matches property" construct
165 and the "\P{}" negation, "doesn't match property".
166 For instance, "\p{Lu}" matches any character with the Unicode "Lu"
168 (Letter, uppercase) property, while "\p{M}" matches any character
169 with an "M" (mark--accents and such) property. Brackets are not
170 required for single letter properties, so "\p{M}" is equivalent to
171 "\pM". Many predefined properties are available, such as "\p{Mir‐
172 rored}" and "\p{Tibetan}".
173 The official Unicode script and block names have spaces and dashes
175 as separators, but for convenience you can use dashes, spaces, or
176 underbars, and case is unimportant. It is recommended, however,
177 that for consistency you use the following naming: the official
178 Unicode script, property, or block name (see below for the addi‐
179 tional rules that apply to block names) with whitespace and dashes
180 removed, and the words "uppercase-first-lowercase-rest". "Latin-1
181 Supplement" thus becomes "Latin1Supplement".
182 You can also use negation in both "\p{}" and "\P{}" by introducing
184 a caret (^) between the first brace and the property name:
185 "\p{^Tamil}" is equal to "\P{Tamil}".
186 NOTE: the properties, scripts, and blocks listed here are as of
188 Unicode 3.2.0, March 2002, or Perl 5.8.0, July 2002. Unicode 4.0.0
189 came out in April 2003, and Perl 5.8.1 in September 2003.
190 Here are the basic Unicode General Category properties, followed by
192 their long form. You can use either; "\p{Lu}" and "\p{Uppercase‐
193 Letter}", for instance, are identical.
194 Short Long
196 L Letter
198 LC CasedLetter
199 Lu UppercaseLetter
200 Ll LowercaseLetter
201 Lt TitlecaseLetter
202 Lm ModifierLetter
203 Lo OtherLetter
204 M Mark
206 Mn NonspacingMark
207 Mc SpacingMark
208 Me EnclosingMark
209 N Number
211 Nd DecimalNumber
212 Nl LetterNumber
213 No OtherNumber
214 P Punctuation
216 Pc ConnectorPunctuation
217 Pd DashPunctuation
218 Ps OpenPunctuation
219 Pe ClosePunctuation
220 Pi InitialPunctuation
221 (may behave like Ps or Pe depending on usage)
222 Pf FinalPunctuation
223 (may behave like Ps or Pe depending on usage)
224 Po OtherPunctuation
225 S Symbol
227 Sm MathSymbol
228 Sc CurrencySymbol
229 Sk ModifierSymbol
230 So OtherSymbol
231 Z Separator
233 Zs SpaceSeparator
234 Zl LineSeparator
235 Zp ParagraphSeparator
236 C Other
238 Cc Control
239 Cf Format
240 Cs Surrogate (not usable)
241 Co PrivateUse
242 Cn Unassigned
243 Single-letter properties match all characters in any of the two-
245 letter sub-properties starting with the same letter. "LC" and "L&"
246 are special cases, which are aliases for the set of "Ll", "Lu", and
247 "Lt".
248 Because Perl hides the need for the user to understand the internal
250 representation of Unicode characters, there is no need to implement
251 the somewhat messy concept of surrogates. "Cs" is therefore not
252 supported.
253 Because scripts differ in their directionality--Hebrew is written
255 right to left, for example--Unicode supplies these properties in
256 the BidiClass class:
257 Property Meaning
259 L Left-to-Right
261 LRE Left-to-Right Embedding
262 LRO Left-to-Right Override
263 R Right-to-Left
264 AL Right-to-Left Arabic
265 RLE Right-to-Left Embedding
266 RLO Right-to-Left Override
267 PDF Pop Directional Format
268 EN European Number
269 ES European Number Separator
270 ET European Number Terminator
271 AN Arabic Number
272 CS Common Number Separator
273 NSM Non-Spacing Mark
274 BN Boundary Neutral
275 B Paragraph Separator
276 S Segment Separator
277 WS Whitespace
278 ON Other Neutrals
279 For example, "\p{BidiClass:R}" matches characters that are normally
281 written right to left.
282 Scripts
284 The script names which can be used by "\p{...}" and "\P{...}", such as
286 in "\p{Latin}" or "\p{Cyrillic}", are as follows:
287 Arabic
289 Armenian
290 Bengali
291 Bopomofo
292 Buhid
293 CanadianAboriginal
294 Cherokee
295 Cyrillic
296 Deseret
297 Devanagari
298 Ethiopic
299 Georgian
300 Gothic
301 Greek
302 Gujarati
303 Gurmukhi
304 Han
305 Hangul
306 Hanunoo
307 Hebrew
308 Hiragana
309 Inherited
310 Kannada
311 Katakana
312 Khmer
313 Lao
314 Latin
315 Malayalam
316 Mongolian
317 Myanmar
318 Ogham
319 OldItalic
320 Oriya
321 Runic
322 Sinhala
323 Syriac
324 Tagalog
325 Tagbanwa
326 Tamil
327 Telugu
328 Thaana
329 Thai
330 Tibetan
331 Yi
332 Extended property classes can supplement the basic properties, defined
334 by the PropList Unicode database:
335 ASCIIHexDigit
337 BidiControl
338 Dash
339 Deprecated
340 Diacritic
341 Extender
342 GraphemeLink
343 HexDigit
344 Hyphen
345 Ideographic
346 IDSBinaryOperator
347 IDSTrinaryOperator
348 JoinControl
349 LogicalOrderException
350 NoncharacterCodePoint
351 OtherAlphabetic
352 OtherDefaultIgnorableCodePoint
353 OtherGraphemeExtend
354 OtherLowercase
355 OtherMath
356 OtherUppercase
357 QuotationMark
358 Radical
359 SoftDotted
360 TerminalPunctuation
361 UnifiedIdeograph
362 WhiteSpace
363 and there are further derived properties:
365 Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
367 Lowercase Ll + OtherLowercase
368 Uppercase Lu + OtherUppercase
369 Math Sm + OtherMath
370 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
372 ID_Continue ID_Start + Mn + Mc + Nd + Pc
373 Any Any character
375 Assigned Any non-Cn character (i.e. synonym for \P{Cn})
376 Unassigned Synonym for \p{Cn}
377 Common Any character (or unassigned code point)
378 not explicitly assigned to a script
379 For backward compatibility (with Perl 5.6), all properties mentioned so
381 far may have "Is" prepended to their name, so "\P{IsLu}", for example,
382 is equal to "\P{Lu}".
383 Blocks
385 In addition to scripts, Unicode also defines blocks of characters. The
387 difference between scripts and blocks is that the concept of scripts is
388 closer to natural languages, while the concept of blocks is more of an
389 artificial grouping based on groups of 256 Unicode characters. For
390 example, the "Latin" script contains letters from many blocks but does
391 not contain all the characters from those blocks. It does not, for
392 example, contain digits, because digits are shared across many scripts.
393 Digits and similar groups, like punctuation, are in a category called
394 "Common".
395 For more about scripts, see the UTR #24:
397 http://www.unicode.org/unicode/reports/tr24/
399 For more about blocks, see:
401 http://www.unicode.org/Public/UNIDATA/Blocks.txt
403 Block names are given with the "In" prefix. For example, the Katakana
405 block is referenced via "\p{InKatakana}". The "In" prefix may be omit‐
406 ted if there is no naming conflict with a script or any other property,
407 but it is recommended that "In" always be used for block tests to avoid
408 confusion.
409 These block names are supported:
411 InAlphabeticPresentationForms
413 InArabic
414 InArabicPresentationFormsA
415 InArabicPresentationFormsB
416 InArmenian
417 InArrows
418 InBasicLatin
419 InBengali
420 InBlockElements
421 InBopomofo
422 InBopomofoExtended
423 InBoxDrawing
424 InBraillePatterns
425 InBuhid
426 InByzantineMusicalSymbols
427 InCJKCompatibility
428 InCJKCompatibilityForms
429 InCJKCompatibilityIdeographs
430 InCJKCompatibilityIdeographsSupplement
431 InCJKRadicalsSupplement
432 InCJKSymbolsAndPunctuation
433 InCJKUnifiedIdeographs
434 InCJKUnifiedIdeographsExtensionA
435 InCJKUnifiedIdeographsExtensionB
436 InCherokee
437 InCombiningDiacriticalMarks
438 InCombiningDiacriticalMarksforSymbols
439 InCombiningHalfMarks
440 InControlPictures
441 InCurrencySymbols
442 InCyrillic
443 InCyrillicSupplementary
444 InDeseret
445 InDevanagari
446 InDingbats
447 InEnclosedAlphanumerics
448 InEnclosedCJKLettersAndMonths
449 InEthiopic
450 InGeneralPunctuation
451 InGeometricShapes
452 InGeorgian
453 InGothic
454 InGreekExtended
455 InGreekAndCoptic
456 InGujarati
457 InGurmukhi
458 InHalfwidthAndFullwidthForms
459 InHangulCompatibilityJamo
460 InHangulJamo
461 InHangulSyllables
462 InHanunoo
463 InHebrew
464 InHighPrivateUseSurrogates
465 InHighSurrogates
466 InHiragana
467 InIPAExtensions
468 InIdeographicDescriptionCharacters
469 InKanbun
470 InKangxiRadicals
471 InKannada
472 InKatakana
473 InKatakanaPhoneticExtensions
474 InKhmer
475 InLao
476 InLatin1Supplement
477 InLatinExtendedA
478 InLatinExtendedAdditional
479 InLatinExtendedB
480 InLetterlikeSymbols
481 InLowSurrogates
482 InMalayalam
483 InMathematicalAlphanumericSymbols
484 InMathematicalOperators
485 InMiscellaneousMathematicalSymbolsA
486 InMiscellaneousMathematicalSymbolsB
487 InMiscellaneousSymbols
488 InMiscellaneousTechnical
489 InMongolian
490 InMusicalSymbols
491 InMyanmar
492 InNumberForms
493 InOgham
494 InOldItalic
495 InOpticalCharacterRecognition
496 InOriya
497 InPrivateUseArea
498 InRunic
499 InSinhala
500 InSmallFormVariants
501 InSpacingModifierLetters
502 InSpecials
503 InSuperscriptsAndSubscripts
504 InSupplementalArrowsA
505 InSupplementalArrowsB
506 InSupplementalMathematicalOperators
507 InSupplementaryPrivateUseAreaA
508 InSupplementaryPrivateUseAreaB
509 InSyriac
510 InTagalog
511 InTagbanwa
512 InTags
513 InTamil
514 InTelugu
515 InThaana
516 InThai
517 InTibetan
518 InUnifiedCanadianAboriginalSyllabics
519 InVariationSelectors
520 InYiRadicals
521 InYiSyllables
522 · The special pattern "\X" matches any extended Unicode sequence--"a
524 combining character sequence" in Standardese--where the first char‐
525 acter is a base character and subsequent characters are mark char‐
526 acters that apply to the base character. "\X" is equivalent to
527 "(?:\PM\pM*)".
528 · The "tr///" operator translates characters instead of bytes. Note
530 that the "tr///CU" functionality has been removed. For similar
531 functionality see pack('U0', ...) and pack('C0', ...).
532 · Case translation operators use the Unicode case translation tables
534 when character input is provided. Note that "uc()", or "\U" in
535 interpolated strings, translates to uppercase, while "ucfirst", or
536 "\u" in interpolated strings, translates to titlecase in languages
537 that make the distinction.
538 · Most operators that deal with positions or lengths in a string will
540 automatically switch to using character positions, including
541 "chop()", "chomp()", "substr()", "pos()", "index()", "rindex()",
542 "sprintf()", "write()", and "length()". Operators that specifi‐
543 cally do not switch include "vec()", "pack()", and "unpack()".
544 Operators that really don't care include operators that treats
545 strings as a bucket of bits such as "sort()", and operators dealing
546 with filenames.
547 · The "pack()"/"unpack()" letters "c" and "C" do not change, since
549 they are often used for byte-oriented formats. Again, think "char"
550 in the C language.
551 There is a new "U" specifier that converts between Unicode charac‐
553 ters and code points.
554 · The "chr()" and "ord()" functions work on characters, similar to
556 "pack("U")" and "unpack("U")", not "pack("C")" and "unpack("C")".
557 "pack("C")" and "unpack("C")" are methods for emulating byte-ori‐
558 ented "chr()" and "ord()" on Unicode strings. While these methods
559 reveal the internal encoding of Unicode strings, that is not some‐
560 thing one normally needs to care about at all.
561 · The bit string operators, "& ⎪ ^ ~", can operate on character data.
563 However, for backward compatibility, such as when using bit string
564 operations when characters are all less than 256 in ordinal value,
565 one should not use "~" (the bit complement) with characters of both
566 values less than 256 and values greater than 256. Most impor‐
567 tantly, DeMorgan's laws ("~($x⎪$y) eq ~$x&~$y" and "~($x&$y) eq
568 ~$x⎪~$y") will not hold. The reason for this mathematical faux pas
569 is that the complement cannot return both the 8-bit (byte-wide) bit
570 complement and the full character-wide bit complement.
571 · lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
573 · the case mapping is from a single Unicode character to
575 another single Unicode character, or
576 · the case mapping is from a single Unicode character to more
578 than one Unicode character.
579 Things to do with locales (Lithuanian, Turkish, Azeri) do not work
581 since Perl does not understand the concept of Unicode locales.
582 See the Unicode Technical Report #21, Case Mappings, for more
584 details.
585 · And finally, "scalar reverse()" reverses by character rather than
587 by byte.
588 User-Defined Character Properties
590 You can define your own character properties by defining subroutines
592 whose names begin with "In" or "Is". The subroutines can be defined in
593 any package. The user-defined properties can be used in the regular
594 expression "\p" and "\P" constructs; if you are using a user-defined
595 property from a package other than the one you are in, you must specify
596 its package in the "\p" or "\P" construct.
597 # assuming property IsForeign defined in Lang::
599 package main; # property package name required
600 if ($txt =~ /\p{Lang::IsForeign}+/) { ... }
601 package Lang; # property package name not required
603 if ($txt =~ /\p{IsForeign}+/) { ... }
604 Note that the effect is compile-time and immutable once defined.
606 The subroutines must return a specially-formatted string, with one or
608 more newline-separated lines. Each line must be one of the following:
609 · Two hexadecimal numbers separated by horizontal whitespace (space
611 or tabular characters) denoting a range of Unicode code points to
612 include.
613 · Something to include, prefixed by "+": a built-in character prop‐
615 erty (prefixed by "utf8::") or a user-defined character property,
616 to represent all the characters in that property; two hexadecimal
617 code points for a range; or a single hexadecimal code point.
618 · Something to exclude, prefixed by "-": an existing character prop‐
620 erty (prefixed by "utf8::") or a user-defined character property,
621 to represent all the characters in that property; two hexadecimal
622 code points for a range; or a single hexadecimal code point.
623 · Something to negate, prefixed "!": an existing character property
625 (prefixed by "utf8::") or a user-defined character property, to
626 represent all the characters in that property; two hexadecimal code
627 points for a range; or a single hexadecimal code point.
628 · Something to intersect with, prefixed by "&": an existing character
630 property (prefixed by "utf8::") or a user-defined character prop‐
631 erty, for all the characters except the characters in the property;
632 two hexadecimal code points for a range; or a single hexadecimal
633 code point.
634 For example, to define a property that covers both the Japanese syl‐
636 labaries (hiragana and katakana), you can define
637 sub InKana {
639 return <<END;
640 3040\t309F
641 30A0\t30FF
642 END
643 }
644 Imagine that the here-doc end marker is at the beginning of the line.
646 Now you can use "\p{InKana}" and "\P{InKana}".
647 You could also have used the existing block property names:
649 sub InKana {
651 return <<'END';
652 +utf8::InHiragana
653 +utf8::InKatakana
654 END
655 }
656 Suppose you wanted to match only the allocated characters, not the raw
658 block ranges: in other words, you want to remove the non-characters:
659 sub InKana {
661 return <<'END';
662 +utf8::InHiragana
663 +utf8::InKatakana
664 -utf8::IsCn
665 END
666 }
667 The negation is useful for defining (surprise!) negated classes.
669 sub InNotKana {
671 return <<'END';
672 !utf8::InHiragana
673 -utf8::InKatakana
674 +utf8::IsCn
675 END
676 }
677 Intersection is useful for getting the common characters matched by two
679 (or more) classes.
680 sub InFooAndBar {
682 return <<'END';
683 +main::Foo
684 &main::Bar
685 END
686 }
687 It's important to remember not to use "&" for the first set -- that
689 would be intersecting with nothing (resulting in an empty set).
690 You can also define your own mappings to be used in the lc(),
692 lcfirst(), uc(), and ucfirst() (or their string-inlined versions). The
693 principle is the same: define subroutines in the "main" package with
694 names like "ToLower" (for lc() and lcfirst()), "ToTitle" (for the first
695 character in ucfirst()), and "ToUpper" (for uc(), and the rest of the
696 characters in ucfirst()).
697 The string returned by the subroutines needs now to be three hexadeci‐
699 mal numbers separated by tabulators: start of the source range, end of
700 the source range, and start of the destination range. For example:
701 sub ToUpper {
703 return <<END;
704 0061\t0063\t0041
705 END
706 }
707 defines an uc() mapping that causes only the characters "a", "b", and
709 "c" to be mapped to "A", "B", "C", all other characters will remain
710 unchanged.
711 If there is no source range to speak of, that is, the mapping is from a
713 single character to another single character, leave the end of the
714 source range empty, but the two tabulator characters are still needed.
715 For example:
716 sub ToLower {
718 return <<END;
719 0041\t\t0061
720 END
721 }
722 defines a lc() mapping that causes only "A" to be mapped to "a", all
724 other characters will remain unchanged.
725 (For serious hackers only) If you want to introspect the default map‐
727 pings, you can find the data in the directory $Config{privlib}/uni‐
728 core/To/. The mapping data is returned as the here-document, and the
729 "utf8::ToSpecFoo" are special exception mappings derived from <$Con‐
730 fig{privlib}>/unicore/SpecialCasing.txt. The "Digit" and "Fold" map‐
731 pings that one can see in the directory are not directly user-accessi‐
732 ble, one can use either the "Unicode::UCD" module, or just match case-
733 insensitively (that's when the "Fold" mapping is used).
734 A final note on the user-defined property tests and mappings: they will
736 be used only if the scalar has been marked as having Unicode charac‐
737 ters. Old byte-style strings will not be affected.
738 Character Encodings for Input and Output
740 See Encode.
742 Unicode Regular Expression Support Level
744 The following list of Unicode support for regular expressions describes
746 all the features currently supported. The references to "Level N" and
747 the section numbers refer to the Unicode Technical Report 18, "Unicode
748 Regular Expression Guidelines", version 6 (Unicode 3.2.0, Perl 5.8.0).
749 · Level 1 - Basic Unicode Support
751 2.1 Hex Notation - done [1]
753 Named Notation - done [2]
754 2.2 Categories - done [3][4]
755 2.3 Subtraction - MISSING [5][6]
756 2.4 Simple Word Boundaries - done [7]
757 2.5 Simple Loose Matches - done [8]
758 2.6 End of Line - MISSING [9][10]
759 [ 1] \x{...}
761 [ 2] \N{...}
762 [ 3] . \p{...} \P{...}
763 [ 4] support for scripts (see UTR#24 Script Names), blocks,
764 binary properties, enumerated non-binary properties, and
765 numeric properties (as listed in UTR#18 Other Properties)
766 [ 5] have negation
767 [ 6] can use regular expression look-ahead [a]
768 or user-defined character properties [b] to emulate subtraction
769 [ 7] include Letters in word characters
770 [ 8] note that Perl does Full case-folding in matching, not Simple:
771 for example U+1F88 is equivalent with U+1F00 U+03B9,
772 not with 1F80. This difference matters for certain Greek
773 capital letters with certain modifiers: the Full case-folding
774 decomposes the letter, while the Simple case-folding would map
775 it to a single character.
776 [ 9] see UTR #13 Unicode Newline Guidelines
777 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029}
778 (should also affect <>, $., and script line numbers)
779 (the \x{85}, \x{2028} and \x{2029} do match \s)
780 [a] You can mimic class subtraction using lookahead. For example,
782 what UTR #18 might write as
783 [{Greek}-[{UNASSIGNED}]]
785 in Perl can be written as:
787 (?!\p{Unassigned})\p{InGreekAndCoptic}
789 (?=\p{Assigned})\p{InGreekAndCoptic}
790 But in this particular example, you probably really want
792 \p{GreekAndCoptic}
794 which will match assigned characters known to be part of the Greek
796 script.
797 Also see the Unicode::Regex::Set module, it does implement the full
799 UTR #18 grouping, intersection, union, and removal (subtraction)
800 syntax.
801 [b] See "User-Defined Character Properties".
803 · Level 2 - Extended Unicode Support
805 3.1 Surrogates - MISSING [11]
807 3.2 Canonical Equivalents - MISSING [12][13]
808 3.3 Locale-Independent Graphemes - MISSING [14]
809 3.4 Locale-Independent Words - MISSING [15]
810 3.5 Locale-Independent Loose Matches - MISSING [16]
811 [11] Surrogates are solely a UTF-16 concept and Perl's internal
813 representation is UTF-8. The Encode module does UTF-16, though.
814 [12] see UTR#15 Unicode Normalization
815 [13] have Unicode::Normalize but not integrated to regexes
816 [14] have \X but at this level . should equal that
817 [15] need three classes, not just \w and \W
818 [16] see UTR#21 Case Mappings
819 · Level 3 - Locale-Sensitive Support
821 4.1 Locale-Dependent Categories - MISSING
823 4.2 Locale-Dependent Graphemes - MISSING [16][17]
824 4.3 Locale-Dependent Words - MISSING
825 4.4 Locale-Dependent Loose Matches - MISSING
826 4.5 Locale-Dependent Ranges - MISSING
827 [16] see UTR#10 Unicode Collation Algorithms
829 [17] have Unicode::Collate but not integrated to regexes
830 Unicode Encodings
832 Unicode characters are assigned to code points, which are abstract num‐
834 bers. To use these numbers, various encodings are needed.
835 · UTF-8
837 UTF-8 is a variable-length (1 to 6 bytes, current character alloca‐
839 tions require 4 bytes), byte-order independent encoding. For ASCII
840 (and we really do mean 7-bit ASCII, not another 8-bit encoding),
841 UTF-8 is transparent.
842 The following table is from Unicode 3.2.
844 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
846 U+0000..U+007F 00..7F
848 U+0080..U+07FF C2..DF 80..BF
849 U+0800..U+0FFF E0 A0..BF 80..BF
850 U+1000..U+CFFF E1..EC 80..BF 80..BF
851 U+D000..U+D7FF ED 80..9F 80..BF
852 U+D800..U+DFFF ******* ill-formed *******
853 U+E000..U+FFFF EE..EF 80..BF 80..BF
854 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
855 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
856 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
857 Note the "A0..BF" in "U+0800..U+0FFF", the "80..9F" in
859 "U+D000...U+D7FF", the "90..B"F in "U+10000..U+3FFFF", and the
860 "80...8F" in "U+100000..U+10FFFF". The "gaps" are caused by legal
861 UTF-8 avoiding non-shortest encodings: it is technically possible
862 to UTF-8-encode a single code point in different ways, but that is
863 explicitly forbidden, and the shortest possible encoding should
864 always be used. So that's what Perl does.
865 Another way to look at it is via bits:
867 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
869 0aaaaaaa 0aaaaaaa
871 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
872 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
873 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
874 As you can see, the continuation bytes all begin with 10, and the
876 leading bits of the start byte tell how many bytes the are in the
877 encoded character.
878 · UTF-EBCDIC
880 Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
882 · UTF-16, UTF-16BE, UTF-16LE, Surrogates, and BOMs (Byte Order Marks)
884 The followings items are mostly for reference and general Unicode
886 knowledge, Perl doesn't use these constructs internally.
887 UTF-16 is a 2 or 4 byte encoding. The Unicode code points
889 "U+0000..U+FFFF" are stored in a single 16-bit unit, and the code
890 points "U+10000..U+10FFFF" in two 16-bit units. The latter case is
891 using surrogates, the first 16-bit unit being the high surrogate,
892 and the second being the low surrogate.
893 Surrogates are code points set aside to encode the
895 "U+10000..U+10FFFF" range of Unicode code points in pairs of 16-bit
896 units. The high surrogates are the range "U+D800..U+DBFF", and the
897 low surrogates are the range "U+DC00..U+DFFF". The surrogate
898 encoding is
899 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
901 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
902 and the decoding is
904 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
906 If you try to generate surrogates (for example by using chr()), you
908 will get a warning if warnings are turned on, because those code
909 points are not valid for a Unicode character.
910 Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
912 itself can be used for in-memory computations, but if storage or
913 transfer is required either UTF-16BE (big-endian) or UTF-16LE (lit‐
914 tle-endian) encodings must be chosen.
915 This introduces another problem: what if you just know that your
917 data is UTF-16, but you don't know which endianness? Byte Order
918 Marks, or BOMs, are a solution to this. A special character has
919 been reserved in Unicode to function as a byte order marker: the
920 character with the code point "U+FEFF" is the BOM.
921 The trick is that if you read a BOM, you will know the byte order,
923 since if it was written on a big-endian platform, you will read the
924 bytes "0xFE 0xFF", but if it was written on a little-endian plat‐
925 form, you will read the bytes "0xFF 0xFE". (And if the originating
926 platform was writing in UTF-8, you will read the bytes "0xEF 0xBB
927 0xBF".)
928 The way this trick works is that the character with the code point
930 "U+FFFE" is guaranteed not to be a valid Unicode character, so the
931 sequence of bytes "0xFF 0xFE" is unambiguously "BOM, represented in
932 little-endian format" and cannot be "U+FFFE", represented in big-
933 endian format".
934 · UTF-32, UTF-32BE, UTF-32LE
936 The UTF-32 family is pretty much like the UTF-16 family, expect
938 that the units are 32-bit, and therefore the surrogate scheme is
939 not needed. The BOM signatures will be "0x00 0x00 0xFE 0xFF" for
940 BE and "0xFF 0xFE 0x00 0x00" for LE.
941 · UCS-2, UCS-4
943 Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
945 encoding. Unlike UTF-16, UCS-2 is not extensible beyond "U+FFFF",
946 because it does not use surrogates. UCS-4 is a 32-bit encoding,
947 functionally identical to UTF-32.
948 · UTF-7
950 A seven-bit safe (non-eight-bit) encoding, which is useful if the
952 transport or storage is not eight-bit safe. Defined by RFC 2152.
953 Security Implications of Unicode
955 · Malformed UTF-8
957 Unfortunately, the specification of UTF-8 leaves some room for
959 interpretation of how many bytes of encoded output one should gen‐
960 erate from one input Unicode character. Strictly speaking, the
961 shortest possible sequence of UTF-8 bytes should be generated,
962 because otherwise there is potential for an input buffer overflow
963 at the receiving end of a UTF-8 connection. Perl always generates
964 the shortest length UTF-8, and with warnings on Perl will warn
965 about non-shortest length UTF-8 along with other malformations,
966 such as the surrogates, which are not real Unicode code points.
967 · Regular expressions behave slightly differently between byte data
969 and character (Unicode) data. For example, the "word character"
970 character class "\w" will work differently depending on if data is
971 eight-bit bytes or Unicode.
972 In the first case, the set of "\w" characters is either small--the
974 default set of alphabetic characters, digits, and the "_"--or, if
975 you are using a locale (see perllocale), the "\w" might contain a
976 few more letters according to your language and country.
977 In the second case, the "\w" set of characters is much, much
979 larger. Most importantly, even in the set of the first 256 charac‐
980 ters, it will probably match different characters: unlike most
981 locales, which are specific to a language and country pair, Unicode
982 classifies all the characters that are letters somewhere as "\w".
983 For example, your locale might not think that LATIN SMALL LETTER
984 ETH is a letter (unless you happen to speak Icelandic), but Unicode
985 does.
986 As discussed elsewhere, Perl has one foot (two hooves?) planted in
988 each of two worlds: the old world of bytes and the new world of
989 characters, upgrading from bytes to characters when necessary. If
990 your legacy code does not explicitly use Unicode, no automatic
991 switch-over to characters should happen. Characters shouldn't get
992 downgraded to bytes, either. It is possible to accidentally mix
993 bytes and characters, however (see perluniintro), in which case
994 "\w" in regular expressions might start behaving differently.
995 Review your code. Use warnings and the "strict" pragma.
996 Unicode in Perl on EBCDIC
998 The way Unicode is handled on EBCDIC platforms is still experimental.
1000 On such platforms, references to UTF-8 encoding in this document and
1001 elsewhere should be read as meaning the UTF-EBCDIC specified in Unicode
1002 Technical Report 16, unless ASCII vs. EBCDIC issues are specifically
1003 discussed. There is no "utfebcdic" pragma or ":utfebcdic" layer;
1004 rather, "utf8" and ":utf8" are reused to mean the platform's "natural"
1005 8-bit encoding of Unicode. See perlebcdic for more discussion of the
1006 issues.
1007 Locales
1009 Usually locale settings and Unicode do not affect each other, but there
1011 are a couple of exceptions:
1012 · You can enable automatic UTF-8-ification of your standard file han‐
1014 dles, default "open()" layer, and @ARGV by using either the "-C"
1015 command line switch or the "PERL_UNICODE" environment variable, see
1016 perlrun for the documentation of the "-C" switch.
1017 · Perl tries really hard to work both with Unicode and the old byte-
1019 oriented world. Most often this is nice, but sometimes Perl's
1020 straddling of the proverbial fence causes problems.
1021 When Unicode Does Not Happen
1023 While Perl does have extensive ways to input and output in Unicode, and
1025 few other 'entry points' like the @ARGV which can be interpreted as
1026 Unicode (UTF-8), there still are many places where Unicode (in some
1027 encoding or another) could be given as arguments or received as
1028 results, or both, but it is not.
1029 The following are such interfaces. For all of these interfaces Perl
1031 currently (as of 5.8.3) simply assumes byte strings both as arguments
1032 and results, or UTF-8 strings if the "encoding" pragma has been used.
1033 One reason why Perl does not attempt to resolve the role of Unicode in
1035 this cases is that the answers are highly dependent on the operating
1036 system and the file system(s). For example, whether filenames can be
1037 in Unicode, and in exactly what kind of encoding, is not exactly a por‐
1038 table concept. Similarly for the qx and system: how well will the
1039 'command line interface' (and which of them?) handle Unicode?
1040 · chdir, chmod, chown, chroot, exec, link, lstat, mkdir, rename,
1042 rmdir, stat, symlink, truncate, unlink, utime, -X
1043 · %ENV
1045 · glob (aka the <*>)
1047 · open, opendir, sysopen
1049 · qx (aka the backtick operator), system
1051 · readdir, readlink
1053 Forcing Unicode in Perl (Or Unforcing Unicode in Perl)
1055 Sometimes (see "When Unicode Does Not Happen") there are situations
1057 where you simply need to force Perl to believe that a byte string is
1058 UTF-8, or vice versa. The low-level calls utf8::upgrade($bytestring)
1059 and utf8::downgrade($utf8string) are the answers.
1060 Do not use them without careful thought, though: Perl may easily get
1062 very confused, angry, or even crash, if you suddenly change the
1063 'nature' of scalar like that. Especially careful you have to be if you
1064 use the utf8::upgrade(): any random byte string is not valid UTF-8.
1065 Using Unicode in XS
1067 If you want to handle Perl Unicode in XS extensions, you may find the
1069 following C APIs useful. See also "Unicode Support" in perlguts for an
1070 explanation about Unicode at the XS level, and perlapi for the API
1071 details.
1072 · "DO_UTF8(sv)" returns true if the "UTF8" flag is on and the bytes
1074 pragma is not in effect. "SvUTF8(sv)" returns true is the "UTF8"
1075 flag is on; the bytes pragma is ignored. The "UTF8" flag being on
1076 does not mean that there are any characters of code points greater
1077 than 255 (or 127) in the scalar or that there are even any charac‐
1078 ters in the scalar. What the "UTF8" flag means is that the
1079 sequence of octets in the representation of the scalar is the
1080 sequence of UTF-8 encoded code points of the characters of a
1081 string. The "UTF8" flag being off means that each octet in this
1082 representation encodes a single character with code point 0..255
1083 within the string. Perl's Unicode model is not to use UTF-8 until
1084 it is absolutely necessary.
1085 · "uvuni_to_utf8(buf, chr)" writes a Unicode character code point
1087 into a buffer encoding the code point as UTF-8, and returns a
1088 pointer pointing after the UTF-8 bytes.
1089 · "utf8_to_uvuni(buf, lenp)" reads UTF-8 encoded bytes from a buffer
1091 and returns the Unicode character code point and, optionally, the
1092 length of the UTF-8 byte sequence.
1093 · "utf8_length(start, end)" returns the length of the UTF-8 encoded
1095 buffer in characters. "sv_len_utf8(sv)" returns the length of the
1096 UTF-8 encoded scalar.
1097 · "sv_utf8_upgrade(sv)" converts the string of the scalar to its
1099 UTF-8 encoded form. "sv_utf8_downgrade(sv)" does the opposite, if
1100 possible. "sv_utf8_encode(sv)" is like sv_utf8_upgrade except that
1101 it does not set the "UTF8" flag. "sv_utf8_decode()" does the oppo‐
1102 site of "sv_utf8_encode()". Note that none of these are to be used
1103 as general-purpose encoding or decoding interfaces: "use Encode"
1104 for that. "sv_utf8_upgrade()" is affected by the encoding pragma
1105 but "sv_utf8_downgrade()" is not (since the encoding pragma is
1106 designed to be a one-way street).
1107 · is_utf8_char(s) returns true if the pointer points to a valid UTF-8
1109 character.
1110 · "is_utf8_string(buf, len)" returns true if "len" bytes of the buf‐
1112 fer are valid UTF-8.
1113 · "UTF8SKIP(buf)" will return the number of bytes in the UTF-8
1115 encoded character in the buffer. "UNISKIP(chr)" will return the
1116 number of bytes required to UTF-8-encode the Unicode character code
1117 point. "UTF8SKIP()" is useful for example for iterating over the
1118 characters of a UTF-8 encoded buffer; "UNISKIP()" is useful, for
1119 example, in computing the size required for a UTF-8 encoded buffer.
1120 · "utf8_distance(a, b)" will tell the distance in characters between
1122 the two pointers pointing to the same UTF-8 encoded buffer.
1123 · "utf8_hop(s, off)" will return a pointer to an UTF-8 encoded buffer
1125 that is "off" (positive or negative) Unicode characters displaced
1126 from the UTF-8 buffer "s". Be careful not to overstep the buffer:
1127 "utf8_hop()" will merrily run off the end or the beginning of the
1128 buffer if told to do so.
1129 · "pv_uni_display(dsv, spv, len, pvlim, flags)" and "sv_uni_dis‐
1131 play(dsv, ssv, pvlim, flags)" are useful for debugging the output
1132 of Unicode strings and scalars. By default they are useful only
1133 for debugging--they display all characters as hexadecimal code
1134 points--but with the flags "UNI_DISPLAY_ISPRINT", "UNI_DIS‐
1135 PLAY_BACKSLASH", and "UNI_DISPLAY_QQ" you can make the output more
1136 readable.
1137 · "ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2)" can be used to
1139 compare two strings case-insensitively in Unicode. For case-sensi‐
1140 tive comparisons you can just use "memEQ()" and "memNE()" as usual.
1141 For more information, see perlapi, and utf8.c and utf8.h in the Perl
1143 source code distribution.
1144
1146 Interaction with Locales
1147 Use of locales with Unicode data may lead to odd results. Currently,
1149 Perl attempts to attach 8-bit locale info to characters in the range
1150 0..255, but this technique is demonstrably incorrect for locales that
1151 use characters above that range when mapped into Unicode. Perl's Uni‐
1152 code support will also tend to run slower. Use of locales with Unicode
1153 is discouraged.
1154 Interaction with Extensions
1156 When Perl exchanges data with an extension, the extension should be
1158 able to understand the UTF-8 flag and act accordingly. If the extension
1159 doesn't know about the flag, it's likely that the extension will return
1160 incorrectly-flagged data.
1161 So if you're working with Unicode data, consult the documentation of
1163 every module you're using if there are any issues with Unicode data
1164 exchange. If the documentation does not talk about Unicode at all, sus‐
1165 pect the worst and probably look at the source to learn how the module
1166 is implemented. Modules written completely in Perl shouldn't cause
1167 problems. Modules that directly or indirectly access code written in
1168 other programming languages are at risk.
1169 For affected functions, the simple strategy to avoid data corruption is
1171 to always make the encoding of the exchanged data explicit. Choose an
1172 encoding that you know the extension can handle. Convert arguments
1173 passed to the extensions to that encoding and convert results back from
1174 that encoding. Write wrapper functions that do the conversions for you,
1175 so you can later change the functions when the extension catches up.
1176 To provide an example, let's say the popular Foo::Bar::escape_html
1178 function doesn't deal with Unicode data yet. The wrapper function would
1179 convert the argument to raw UTF-8 and convert the result back to Perl's
1180 internal representation like so:
1181 sub my_escape_html ($) {
1183 my($what) = shift;
1184 return unless defined $what;
1185 Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
1186 }
1187 Sometimes, when the extension does not convert data but just stores and
1189 retrieves them, you will be in a position to use the otherwise danger‐
1190 ous Encode::_utf8_on() function. Let's say the popular "Foo::Bar"
1191 extension, written in C, provides a "param" method that lets you store
1192 and retrieve data according to these prototypes:
1193 $self->param($name, $value); # set a scalar
1195 $value = $self->param($name); # retrieve a scalar
1196 If it does not yet provide support for any encoding, one could write a
1198 derived class with such a "param" method:
1199 sub param {
1201 my($self,$name,$value) = @_;
1202 utf8::upgrade($name); # make sure it is UTF-8 encoded
1203 if (defined $value)
1204 utf8::upgrade($value); # make sure it is UTF-8 encoded
1205 return $self->SUPER::param($name,$value);
1206 } else {
1207 my $ret = $self->SUPER::param($name);
1208 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
1209 return $ret;
1210 }
1211 }
1212 Some extensions provide filters on data entry/exit points, such as
1214 DB_File::filter_store_key and family. Look out for such filters in the
1215 documentation of your extensions, they can make the transition to Uni‐
1216 code data much easier.
1217 Speed
1219 Some functions are slower when working on UTF-8 encoded strings than on
1221 byte encoded strings. All functions that need to hop over characters
1222 such as length(), substr() or index(), or matching regular expressions
1223 can work much faster when the underlying data are byte-encoded.
1224 In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1 a
1226 caching scheme was introduced which will hopefully make the slowness
1227 somewhat less spectacular, at least for some operations. In general,
1228 operations with UTF-8 encoded strings are still slower. As an example,
1229 the Unicode properties (character classes) like "\p{Nd}" are known to
1230 be quite a bit slower (5-20 times) than their simpler counterparts like
1231 "\d" (then again, there 268 Unicode characters matching "Nd" compared
1232 with the 10 ASCII characters matching "d").
1233 Porting code from perl-5.6.X
1235 Perl 5.8 has a different Unicode model from 5.6. In 5.6 the programmer
1237 was required to use the "utf8" pragma to declare that a given scope
1238 expected to deal with Unicode data and had to make sure that only Uni‐
1239 code data were reaching that scope. If you have code that is working
1240 with 5.6, you will need some of the following adjustments to your code.
1241 The examples are written such that the code will continue to work under
1242 5.6, so you should be safe to try them out.
1243 · A filehandle that should read or write UTF-8
1245 if ($] > 5.007) {
1247 binmode $fh, ":utf8";
1248 }
1249 · A scalar that is going to be passed to some extension
1251 Be it Compress::Zlib, Apache::Request or any extension that has no
1253 mention of Unicode in the manpage, you need to make sure that the
1254 UTF-8 flag is stripped off. Note that at the time of this writing
1255 (October 2002) the mentioned modules are not UTF-8-aware. Please
1256 check the documentation to verify if this is still true.
1257 if ($] > 5.007) {
1259 require Encode;
1260 $val = Encode::encode_utf8($val); # make octets
1261 }
1262 · A scalar we got back from an extension
1264 If you believe the scalar comes back as UTF-8, you will most likely
1266 want the UTF-8 flag restored:
1267 if ($] > 5.007) {
1269 require Encode;
1270 $val = Encode::decode_utf8($val);
1271 }
1272 · Same thing, if you are really sure it is UTF-8
1274 if ($] > 5.007) {
1276 require Encode;
1277 Encode::_utf8_on($val);
1278 }
1279 · A wrapper for fetchrow_array and fetchrow_hashref
1281 When the database contains only UTF-8, a wrapper function or method
1283 is a convenient way to replace all your fetchrow_array and
1284 fetchrow_hashref calls. A wrapper function will also make it easier
1285 to adapt to future enhancements in your database driver. Note that
1286 at the time of this writing (October 2002), the DBI has no stan‐
1287 dardized way to deal with UTF-8 data. Please check the documenta‐
1288 tion to verify if that is still true.
1289 sub fetchrow {
1291 my($self, $sth, $what) = @_; # $what is one of fetchrow_{array,hashref}
1292 if ($] < 5.007) {
1293 return $sth->$what;
1294 } else {
1295 require Encode;
1296 if (wantarray) {
1297 my @arr = $sth->$what;
1298 for (@arr) {
1299 defined && /[^\000-\177]/ && Encode::_utf8_on($_);
1300 }
1301 return @arr;
1302 } else {
1303 my $ret = $sth->$what;
1304 if (ref $ret) {
1305 for my $k (keys %$ret) {
1306 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret->{$k};
1307 }
1308 return $ret;
1309 } else {
1310 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
1311 return $ret;
1312 }
1313 }
1314 }
1315 }
1316 · A large scalar that you know can only contain ASCII
1318 Scalars that contain only ASCII and are marked as UTF-8 are some‐
1320 times a drag to your program. If you recognize such a situation,
1321 just remove the UTF-8 flag:
1322 utf8::downgrade($val) if $] > 5.007;
1324
1326 perluniintro, encoding, Encode, open, utf8, bytes, perlretut, "${^UNI‐
1327 CODE}" in perlvar
1328perl v5.8.8 2006-01-07 PERLUNICODE(1)