1Normalize(3) User Contributed Perl Documentation Normalize(3)
2
6 Unicode::Normalize - Unicode Normalization Forms
7
9 (1) using function names exported by default:
10 use Unicode::Normalize;
12 $NFD_string = NFD($string); # Normalization Form D
14 $NFC_string = NFC($string); # Normalization Form C
15 $NFKD_string = NFKD($string); # Normalization Form KD
16 $NFKC_string = NFKC($string); # Normalization Form KC
17 (2) using function names exported on request:
19 use Unicode::Normalize 'normalize';
21 $NFD_string = normalize('D', $string); # Normalization Form D
23 $NFC_string = normalize('C', $string); # Normalization Form C
24 $NFKD_string = normalize('KD', $string); # Normalization Form KD
25 $NFKC_string = normalize('KC', $string); # Normalization Form KC
26
28 Parameters:
29 $string is used as a string under character semantics (see
31 perlunicode).
32 $code_point should be an unsigned integer representing a Unicode code
34 point.
35 Note: Between XSUB and pure Perl, there is an incompatibility about the
37 interpretation of $code_point as a decimal number. XSUB converts
38 $code_point to an unsigned integer, but pure Perl does not. Do not use
39 a floating point nor a negative sign in $code_point.
40 Normalization Forms
42 "$NFD_string = NFD($string)"
43 It returns the Normalization Form D (formed by canonical
44 decomposition).
45 "$NFC_string = NFC($string)"
47 It returns the Normalization Form C (formed by canonical
48 decomposition followed by canonical composition).
49 "$NFKD_string = NFKD($string)"
51 It returns the Normalization Form KD (formed by compatibility
52 decomposition).
53 "$NFKC_string = NFKC($string)"
55 It returns the Normalization Form KC (formed by compatibility
56 decomposition followed by canonical composition).
57 "$FCD_string = FCD($string)"
59 If the given string is in FCD ("Fast C or D" form; cf. UTN #5), it
60 returns the string without modification; otherwise it returns an
61 FCD string.
62 Note: FCD is not always unique, then plural forms may be equivalent
64 each other. "FCD()" will return one of these equivalent forms.
65 "$FCC_string = FCC($string)"
67 It returns the FCC form ("Fast C Contiguous"; cf. UTN #5).
68 Note: FCC is unique, as well as four normalization forms (NF*).
70 "$normalized_string = normalize($form_name, $string)"
72 It returns the normalization form of $form_name.
73 As $form_name, one of the following names must be given.
75 'C' or 'NFC' for Normalization Form C (UAX #15)
77 'D' or 'NFD' for Normalization Form D (UAX #15)
78 'KC' or 'NFKC' for Normalization Form KC (UAX #15)
79 'KD' or 'NFKD' for Normalization Form KD (UAX #15)
80 'FCD' for "Fast C or D" Form (UTN #5)
82 'FCC' for "Fast C Contiguous" (UTN #5)
83 Decomposition and Composition
85 "$decomposed_string = decompose($string [, $useCompatMapping])"
86 It returns the concatenation of the decomposition of each character
87 in the string.
88 If the second parameter (a boolean) is omitted or false, the
90 decomposition is canonical decomposition; if the second parameter
91 (a boolean) is true, the decomposition is compatibility
92 decomposition.
93 The string returned is not always in NFD/NFKD. Reordering may be
95 required.
96 $NFD_string = reorder(decompose($string)); # eq. to NFD()
98 $NFKD_string = reorder(decompose($string, TRUE)); # eq. to NFKD()
99 "$reordered_string = reorder($string)"
101 It returns the result of reordering the combining characters
102 according to Canonical Ordering Behavior.
103 For example, when you have a list of NFD/NFKD strings, you can get
105 the concatenated NFD/NFKD string from them, by saying
106 $concat_NFD = reorder(join '', @NFD_strings);
108 $concat_NFKD = reorder(join '', @NFKD_strings);
109 "$composed_string = compose($string)"
111 It returns the result of canonical composition without applying any
112 decomposition.
113 For example, when you have a NFD/NFKD string, you can get its
115 NFC/NFKC string, by saying
116 $NFC_string = compose($NFD_string);
118 $NFKC_string = compose($NFKD_string);
119 "($processed, $unprocessed) = splitOnLastStarter($normalized)"
121 It returns two strings: the first one, $processed, is a part before
122 the last starter, and the second one, $unprocessed is another part
123 after the first part. A starter is a character having a combining
124 class of zero (see UAX #15).
125 Note that $processed may be empty (when $normalized contains no
127 starter or starts with the last starter), and then $unprocessed
128 should be equal to the entire $normalized.
129 When you have a $normalized string and an $unnormalized string
131 following it, a simple concatenation is wrong:
132 $concat = $normalized . normalize($form, $unnormalized); # wrong!
134 Instead of it, do like this:
136 ($processed, $unprocessed) = splitOnLastStarter($normalized);
138 $concat = $processed . normalize($form, $unprocessed.$unnormalized);
139 "splitOnLastStarter()" should be called with a pre-normalized
141 parameter $normalized, that is in the same form as $form you want.
142 If you have an array of @string that should be concatenated and
144 then normalized, you can do like this:
145 my $result = "";
147 my $unproc = "";
148 foreach my $str (@string) {
149 $unproc .= $str;
150 my $n = normalize($form, $unproc);
151 my($p, $u) = splitOnLastStarter($n);
152 $result .= $p;
153 $unproc = $u;
154 }
155 $result .= $unproc;
156 # instead of normalize($form, join('', @string))
157 "$processed = normalize_partial($form, $unprocessed)"
159 A wrapper for the combination of "normalize()" and
160 "splitOnLastStarter()". Note that $unprocessed will be modified as
161 a side-effect.
162 If you have an array of @string that should be concatenated and
164 then normalized, you can do like this:
165 my $result = "";
167 my $unproc = "";
168 foreach my $str (@string) {
169 $unproc .= $str;
170 $result .= normalize_partial($form, $unproc);
171 }
172 $result .= $unproc;
173 # instead of normalize($form, join('', @string))
174 "$processed = NFD_partial($unprocessed)"
176 It does like "normalize_partial('NFD', $unprocessed)". Note that
177 $unprocessed will be modified as a side-effect.
178 "$processed = NFC_partial($unprocessed)"
180 It does like "normalize_partial('NFC', $unprocessed)". Note that
181 $unprocessed will be modified as a side-effect.
182 "$processed = NFKD_partial($unprocessed)"
184 It does like "normalize_partial('NFKD', $unprocessed)". Note that
185 $unprocessed will be modified as a side-effect.
186 "$processed = NFKC_partial($unprocessed)"
188 It does like "normalize_partial('NFKC', $unprocessed)". Note that
189 $unprocessed will be modified as a side-effect.
190 Quick Check
192 (see Annex 8, UAX #15; and DerivedNormalizationProps.txt)
193 The following functions check whether the string is in that
195 normalization form.
196 The result returned will be one of the following:
198 YES The string is in that normalization form.
200 NO The string is not in that normalization form.
201 MAYBE Dubious. Maybe yes, maybe no.
202 "$result = checkNFD($string)"
204 It returns true (1) if "YES"; false ("empty string") if "NO".
205 "$result = checkNFC($string)"
207 It returns true (1) if "YES"; false ("empty string") if "NO";
208 "undef" if "MAYBE".
209 "$result = checkNFKD($string)"
211 It returns true (1) if "YES"; false ("empty string") if "NO".
212 "$result = checkNFKC($string)"
214 It returns true (1) if "YES"; false ("empty string") if "NO";
215 "undef" if "MAYBE".
216 "$result = checkFCD($string)"
218 It returns true (1) if "YES"; false ("empty string") if "NO".
219 "$result = checkFCC($string)"
221 It returns true (1) if "YES"; false ("empty string") if "NO";
222 "undef" if "MAYBE".
223 Note: If a string is not in FCD, it must not be in FCC. So
225 "checkFCC($not_FCD_string)" should return "NO".
226 "$result = check($form_name, $string)"
228 It returns true (1) if "YES"; false ("empty string") if "NO";
229 "undef" if "MAYBE".
230 As $form_name, one of the following names must be given.
232 'C' or 'NFC' for Normalization Form C (UAX #15)
234 'D' or 'NFD' for Normalization Form D (UAX #15)
235 'KC' or 'NFKC' for Normalization Form KC (UAX #15)
236 'KD' or 'NFKD' for Normalization Form KD (UAX #15)
237 'FCD' for "Fast C or D" Form (UTN #5)
239 'FCC' for "Fast C Contiguous" (UTN #5)
240 Note
242 In the cases of NFD, NFKD, and FCD, the answer must be either "YES" or
244 "NO". The answer "MAYBE" may be returned in the cases of NFC, NFKC, and
245 FCC.
246 A "MAYBE" string should contain at least one combining character or the
248 like. For example, "COMBINING ACUTE ACCENT" has the
249 MAYBE_NFC/MAYBE_NFKC property.
250 Both "checkNFC("A\N{COMBINING ACUTE ACCENT}")" and
252 "checkNFC("B\N{COMBINING ACUTE ACCENT}")" will return "MAYBE".
253 "A\N{COMBINING ACUTE ACCENT}" is not in NFC (its NFC is "\N{LATIN
254 CAPITAL LETTER A WITH ACUTE}"), while "B\N{COMBINING ACUTE ACCENT}" is
255 in NFC.
256 If you want to check exactly, compare the string with its NFC/NFKC/FCC.
258 if ($string eq NFC($string)) {
260 # $string is exactly normalized in NFC;
261 } else {
262 # $string is not normalized in NFC;
263 }
264 if ($string eq NFKC($string)) {
266 # $string is exactly normalized in NFKC;
267 } else {
268 # $string is not normalized in NFKC;
269 }
270 Character Data
272 These functions are interface of character data used internally. If
273 you want only to get Unicode normalization forms, you don't need call
274 them yourself.
275 "$canonical_decomposition = getCanon($code_point)"
277 If the character is canonically decomposable (including Hangul
278 Syllables), it returns the (full) canonical decomposition as a
279 string. Otherwise it returns "undef".
280 Note: According to the Unicode standard, the canonical
282 decomposition of the character that is not canonically decomposable
283 is same as the character itself.
284 "$compatibility_decomposition = getCompat($code_point)"
286 If the character is compatibility decomposable (including Hangul
287 Syllables), it returns the (full) compatibility decomposition as a
288 string. Otherwise it returns "undef".
289 Note: According to the Unicode standard, the compatibility
291 decomposition of the character that is not compatibility
292 decomposable is same as the character itself.
293 "$code_point_composite = getComposite($code_point_here,
295 $code_point_next)"
296 If two characters here and next (as code points) are composable
297 (including Hangul Jamo/Syllables and Composition Exclusions), it
298 returns the code point of the composite.
299 If they are not composable, it returns "undef".
301 "$combining_class = getCombinClass($code_point)"
303 It returns the combining class (as an integer) of the character.
304 "$may_be_composed_with_prev_char = isComp2nd($code_point)"
306 It returns a boolean whether the character of the specified
307 codepoint may be composed with the previous one in a certain
308 composition (including Hangul Compositions, but excluding
309 Composition Exclusions and Non-Starter Decompositions).
310 "$is_exclusion = isExclusion($code_point)"
312 It returns a boolean whether the code point is a composition
313 exclusion.
314 "$is_singleton = isSingleton($code_point)"
316 It returns a boolean whether the code point is a singleton
317 "$is_non_starter_decomposition = isNonStDecomp($code_point)"
319 It returns a boolean whether the code point has Non-Starter
320 Decomposition.
321 "$is_Full_Composition_Exclusion = isComp_Ex($code_point)"
323 It returns a boolean of the derived property Comp_Ex
324 (Full_Composition_Exclusion). This property is generated from
325 Composition Exclusions + Singletons + Non-Starter Decompositions.
326 "$NFD_is_NO = isNFD_NO($code_point)"
328 It returns a boolean of the derived property NFD_NO
329 (NFD_Quick_Check=No).
330 "$NFC_is_NO = isNFC_NO($code_point)"
332 It returns a boolean of the derived property NFC_NO
333 (NFC_Quick_Check=No).
334 "$NFC_is_MAYBE = isNFC_MAYBE($code_point)"
336 It returns a boolean of the derived property NFC_MAYBE
337 (NFC_Quick_Check=Maybe).
338 "$NFKD_is_NO = isNFKD_NO($code_point)"
340 It returns a boolean of the derived property NFKD_NO
341 (NFKD_Quick_Check=No).
342 "$NFKC_is_NO = isNFKC_NO($code_point)"
344 It returns a boolean of the derived property NFKC_NO
345 (NFKC_Quick_Check=No).
346 "$NFKC_is_MAYBE = isNFKC_MAYBE($code_point)"
348 It returns a boolean of the derived property NFKC_MAYBE
349 (NFKC_Quick_Check=Maybe).
350
352 "NFC", "NFD", "NFKC", "NFKD": by default.
353 "normalize" and other some functions: on request.
355
357 Perl's version vs. Unicode version
358 Since this module refers to perl core's Unicode database in the
359 directory /lib/unicore (or formerly /lib/unicode), the Unicode
360 version of normalization implemented by this module depends on what
361 has been compiled into your perl. The following table lists the
362 default Unicode version that comes with various perl versions. (It
363 is possible to change the Unicode version in any perl version to be
364 any earlier Unicode version, so one could cause Unicode 3.2 to be
365 used in any perl version starting with 5.8.0. See
366 $Config{privlib}/unicore/README.perl.
367 perl's version implemented Unicode version
369 5.6.1 3.0.1
370 5.7.2 3.1.0
371 5.7.3 3.1.1 (normalization is same as 3.1.0)
372 5.8.0 3.2.0
373 5.8.1-5.8.3 4.0.0
374 5.8.4-5.8.6 4.0.1 (normalization is same as 4.0.0)
375 5.8.7-5.8.8 4.1.0
376 5.10.0 5.0.0
377 5.8.9, 5.10.1 5.1.0
378 5.12.x 5.2.0
379 5.14.x 6.0.0
380 5.16.x 6.1.0
381 5.18.x 6.2.0
382 5.20.x 6.3.0
383 5.22.x 7.0.0
384 Correction of decomposition mapping
386 In older Unicode versions, a small number of characters (all of
387 which are CJK compatibility ideographs as far as they have been
388 found) may have an erroneous decomposition mapping (see
389 NormalizationCorrections.txt). Anyhow, this module will neither
390 refer to NormalizationCorrections.txt nor provide any specific
391 version of normalization. Therefore this module running on an older
392 perl with an older Unicode database may use the erroneous
393 decomposition mapping blindly conforming to the Unicode database.
394 Revised definition of canonical composition
396 In Unicode 4.1.0, the definition D2 of canonical composition (which
397 affects NFC and NFKC) has been changed (see Public Review Issue #29
398 and recent UAX #15). This module has used the newer definition
399 since the version 0.07 (Oct 31, 2001). This module will not
400 support the normalization according to the older definition, even
401 if the Unicode version implemented by perl is lower than 4.1.0.
402
404 SADAHIRO Tomoyuki <SADAHIRO@cpan.org>
405 Currently maintained by <perl5-porters@perl.org>
407 Copyright(C) 2001-2012, SADAHIRO Tomoyuki. Japan. All rights reserved.
409
411 This module is free software; you can redistribute it and/or modify it
412 under the same terms as Perl itself.
413
415 http://www.unicode.org/reports/tr15/
416 Unicode Normalization Forms - UAX #15
417 http://www.unicode.org/Public/UNIDATA/CompositionExclusions.txt
419 Composition Exclusion Table
420 http://www.unicode.org/Public/UNIDATA/DerivedNormalizationProps.txt
422 Derived Normalization Properties
423 http://www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt
425 Normalization Corrections
426 http://www.unicode.org/review/pr-29.html
428 Public Review Issue #29: Normalization Issue
429 http://www.unicode.org/notes/tn5/
431 Canonical Equivalence in Applications - UTN #5
432perl v5.26.3 2015-12-16 Normalize(3)