1Collate(3) User Contributed Perl Documentation Collate(3)
2
6 Unicode::Collate - Unicode Collation Algorithm
7
9 use Unicode::Collate;
10 #construct
12 $Collator = Unicode::Collate->new(%tailoring);
13 #sort
15 @sorted = $Collator->sort(@not_sorted);
16 #compare
18 $result = $Collator->cmp($a, $b); # returns 1, 0, or -1.
19 Note: Strings in @not_sorted, $a and $b are interpreted according to
21 Perl's Unicode support. See perlunicode, perluniintro, perlunitut,
22 perlunifaq, utf8. Otherwise you can use "preprocess" or should decode
23 them before.
24
26 This module is an implementation of Unicode Technical Standard #10
27 (a.k.a. UTS #10) - Unicode Collation Algorithm (a.k.a. UCA).
28 Constructor and Tailoring
30 The "new" method returns a collator object. If new() is called with no
31 parameters, the collator should do the default collation.
32 $Collator = Unicode::Collate->new(
34 UCA_Version => $UCA_Version,
35 alternate => $alternate, # alias for 'variable'
36 backwards => $levelNumber, # or \@levelNumbers
37 entry => $element,
38 hangul_terminator => $term_primary_weight,
39 highestFFFF => $bool,
40 identical => $bool,
41 ignoreName => qr/$ignoreName/,
42 ignoreChar => qr/$ignoreChar/,
43 ignore_level2 => $bool,
44 katakana_before_hiragana => $bool,
45 level => $collationLevel,
46 long_contraction => $bool,
47 minimalFFFE => $bool,
48 normalization => $normalization_form,
49 overrideCJK => \&overrideCJK,
50 overrideHangul => \&overrideHangul,
51 preprocess => \&preprocess,
52 rearrange => \@charList,
53 rewrite => \&rewrite,
54 suppress => \@charList,
55 table => $filename,
56 undefName => qr/$undefName/,
57 undefChar => qr/$undefChar/,
58 upper_before_lower => $bool,
59 variable => $variable,
60 );
61 UCA_Version
63 If the revision (previously "tracking version") number of UCA is
64 given, behavior of that revision is emulated on collating. If
65 omitted, the return value of "UCA_Version()" is used.
66 The following revisions are supported. The default is 36.
68 UCA Unicode Standard DUCET (@version)
70 -------------------------------------------------------
71 8 3.1 3.0.1 (3.0.1d9)
72 9 3.1 with Corrigendum 3 3.1.1 (3.1.1)
73 11 4.0 4.0.0 (4.0.0)
74 14 4.1.0 4.1.0 (4.1.0)
75 16 5.0 5.0.0 (5.0.0)
76 18 5.1.0 5.1.0 (5.1.0)
77 20 5.2.0 5.2.0 (5.2.0)
78 22 6.0.0 6.0.0 (6.0.0)
79 24 6.1.0 6.1.0 (6.1.0)
80 26 6.2.0 6.2.0 (6.2.0)
81 28 6.3.0 6.3.0 (6.3.0)
82 30 7.0.0 7.0.0 (7.0.0)
83 32 8.0.0 8.0.0 (8.0.0)
84 34 9.0.0 9.0.0 (9.0.0)
85 36 10.0.0 10.0.0(10.0.0)
86 * See below for "long_contraction" with "UCA_Version" 22 and 24.
88 * Noncharacters (e.g. U+FFFF) are not ignored, and can be
90 overridden since "UCA_Version" 22.
91 * Out-of-range codepoints (greater than U+10FFFF) are not ignored,
93 and can be overridden since "UCA_Version" 22.
94 * Fully ignorable characters were ignored, and would not interrupt
96 contractions with "UCA_Version" 9 and 11.
97 * Treatment of ignorables after variables and some behaviors were
99 changed at "UCA_Version" 9.
100 * Characters regarded as CJK unified ideographs (cf. "overrideCJK")
102 depend on "UCA_Version".
103 * Many hangul jamo are assigned at "UCA_Version" 20, that will
105 affect "hangul_terminator".
106 alternate
108 -- see 3.2.2 Alternate Weighting, version 8 of UTS #10
109 For backward compatibility, "alternate" (old name) can be used as
111 an alias for "variable".
112 backwards
114 -- see 3.4 Backward Accents, UTS #10.
115 backwards => $levelNumber or \@levelNumbers
117 Weights in reverse order; ex. level 2 (diacritic ordering) in
119 French. If omitted (or $levelNumber is "undef" or "\@levelNumbers"
120 is "[]"), forwards at all the levels.
121 entry
123 -- see 5 Tailoring; 9.1 Allkeys File Format, UTS #10.
124 If the same character (or a sequence of characters) exists in the
126 collation element table through "table", mapping to collation
127 elements is overridden. If it does not exist, the mapping is
128 defined additionally.
129 entry => <<'ENTRY', # for DUCET v4.0.0 (allkeys-4.0.0.txt)
131 0063 0068 ; [.0E6A.0020.0002.0063] # ch
132 0043 0068 ; [.0E6A.0020.0007.0043] # Ch
133 0043 0048 ; [.0E6A.0020.0008.0043] # CH
134 006C 006C ; [.0F4C.0020.0002.006C] # ll
135 004C 006C ; [.0F4C.0020.0007.004C] # Ll
136 004C 004C ; [.0F4C.0020.0008.004C] # LL
137 00F1 ; [.0F7B.0020.0002.00F1] # n-tilde
138 006E 0303 ; [.0F7B.0020.0002.00F1] # n-tilde
139 00D1 ; [.0F7B.0020.0008.00D1] # N-tilde
140 004E 0303 ; [.0F7B.0020.0008.00D1] # N-tilde
141 ENTRY
142 entry => <<'ENTRY', # for DUCET v4.0.0 (allkeys-4.0.0.txt)
144 00E6 ; [.0E33.0020.0002.00E6][.0E8B.0020.0002.00E6] # ae ligature as <a><e>
145 00C6 ; [.0E33.0020.0008.00C6][.0E8B.0020.0008.00C6] # AE ligature as <A><E>
146 ENTRY
147 NOTE: The code point in the UCA file format (before ';') must be a
149 Unicode code point (defined as hexadecimal), but not a native code
150 point. So 0063 must always denote "U+0063", but not a character of
151 "\x63".
152 Weighting may vary depending on collation element table. So ensure
154 the weights defined in "entry" will be consistent with those in the
155 collation element table loaded via "table".
156 In DUCET v4.0.0, primary weight of "C" is 0E60 and that of "D" is
158 "0E6D". So setting primary weight of "CH" to "0E6A" (as a value
159 between 0E60 and "0E6D") makes ordering as "C < CH < D". Exactly
160 speaking DUCET already has some characters between "C" and "D":
161 "small capital C" ("U+1D04") with primary weight 0E64,
162 "c-hook/C-hook" ("U+0188/U+0187") with 0E65, and "c-curl"
163 ("U+0255") with 0E69. Then primary weight "0E6A" for "CH" makes
164 "CH" ordered between "c-curl" and "D".
165 hangul_terminator
167 -- see 7.1.4 Trailing Weights, UTS #10.
168 If a true value is given (non-zero but should be positive), it will
170 be added as a terminator primary weight to the end of every
171 standard Hangul syllable. Secondary and any higher weights for
172 terminator are set to zero. If the value is false or
173 "hangul_terminator" key does not exist, insertion of terminator
174 weights will not be performed.
175 Boundaries of Hangul syllables are determined according to
177 conjoining Jamo behavior in the Unicode Standard and
178 HangulSyllableType.txt.
179 Implementation Note: [22m(1) For expansion mapping (Unicode character
181 mapped to a sequence of collation elements), a terminator will not
182 be added between collation elements, even if Hangul syllable
183 boundary exists there. Addition of terminator is restricted to the
184 next position to the last collation element.
185 (2) Non-conjoining Hangul letters (Compatibility Jamo, halfwidth
187 Jamo, and enclosed letters) are not automatically terminated with a
188 terminator primary weight. These characters may need terminator
189 included in a collation element table beforehand.
190 highestFFFF
192 -- see 2.4 Tailored noncharacter weights, UTS #35 (LDML) Part 5:
193 Collation.
194 If the parameter is made true, "U+FFFF" has a highest primary
196 weight. When a boolean of "$coll->ge($str, "abc")" and
197 "$coll->le($str, "abc\x{FFFF}")" is true, it is expected that $str
198 begins with "abc", or another primary equivalent. $str may be
199 "abcd", "abc012", but should not include "U+FFFF" such as
200 "abc\x{FFFF}xyz".
201 "$coll->le($str, "abc\x{FFFF}")" works like "$coll->lt($str,
203 "abd")" almost, but the latter has a problem that you should know
204 which letter is next to "c". For a certain language where "ch" as
205 the next letter, "abch" is greater than "abc\x{FFFF}", but less
206 than "abd".
207 Note: This is equivalent to "(entry => 'FFFF ;
209 [.FFFE.0020.0005.FFFF]')". Any other character than "U+FFFF" can
210 be tailored by "entry".
211 identical
213 -- see A.3 Deterministic Comparison, UTS #10.
214 By default, strings whose weights are equal should be equal, even
216 though their code points are not equal. Completely ignorable
217 characters are ignored.
218 If the parameter is made true, a final, tie-breaking level is used.
220 If no difference of weights is found after the comparison through
221 all the level specified by "level", the comparison with code points
222 will be performed. For the tie-breaking comparison, the sort key
223 has code points of the original string appended. Completely
224 ignorable characters are not ignored.
225 If "preprocess" and/or "normalization" is applied, the code points
227 of the string after them (in NFD by default) are used.
228 ignoreChar
230 ignoreName
231 -- see 3.6 Variable Weighting, UTS #10.
232 Makes the entry in the table completely ignorable; i.e. as if the
234 weights were zero at all level.
235 Through "ignoreChar", any character matching "qr/$ignoreChar/" will
237 be ignored. Through "ignoreName", any character whose name (given
238 in the "table" file as a comment) matches "qr/$ignoreName/" will be
239 ignored.
240 E.g. when 'a' and 'e' are ignorable, 'element' is equal to 'lament'
242 (or 'lmnt').
243 ignore_level2
245 -- see 5.1 Parametric Tailoring, UTS #10.
246 By default, case-sensitive comparison (that is level 3 difference)
248 won't ignore accents (that is level 2 difference).
249 If the parameter is made true, accents (and other primary ignorable
251 characters) are ignored, even though cases are taken into account.
252 NOTE: "level" should be 3 or greater.
254 katakana_before_hiragana
256 -- see 7.2 Tertiary Weight Table, UTS #10.
257 By default, hiragana is before katakana. If the parameter is made
259 true, this is reversed.
260 NOTE: This parameter simplemindedly assumes that any
262 hiragana/katakana distinctions must occur in level 3, and their
263 weights at level 3 must be same as those mentioned in 7.3.1, UTS
264 #10. If you define your collation elements which violate this
265 requirement, this parameter does not work validly.
266 level
268 -- see 4.3 Form Sort Key, UTS #10.
269 Set the maximum level. Any higher levels than the specified one
271 are ignored.
272 Level 1: alphabetic ordering
274 Level 2: diacritic ordering
275 Level 3: case ordering
276 Level 4: tie-breaking (e.g. in the case when variable is 'shifted')
277 ex.level => 2,
279 If omitted, the maximum is the 4th.
281 NOTE: The DUCET includes weights over 0xFFFF at the 4th level. But
283 this module only uses weights within 0xFFFF. When "variable" is
284 'blanked' or 'non-ignorable' (other than 'shifted' and
285 'shift-trimmed'), the level 4 may be unreliable.
286 See also "identical".
288 long_contraction
290 -- see 3.8.2 Well-Formedness of the DUCET, 4.2 Produce Array, UTS
291 #10.
292 If the parameter is made true, for a contraction with three or more
294 characters (here nicknamed "long contraction"), initial substrings
295 will be handled. For example, a contraction ABC, where A is a
296 starter, and B and C are non-starters (character with non-zero
297 combining character class), will be detected even if there is not
298 AB as a contraction.
299 Default: Usually false. If "UCA_Version" is 22 or 24, and the
301 value of "long_contraction" is not specified in "new()", a true
302 value is set implicitly. This is a workaround to pass Conformance
303 Tests for Unicode 6.0.0 and 6.1.0.
304 "change()" handles "long_contraction" explicitly only. If
306 "long_contraction" is not specified in "change()", even though
307 "UCA_Version" is changed, "long_contraction" will not be changed.
308 Limitation: Scanning non-starters is one-way (no back tracking).
310 If AB is found but not ABC is not found, other long contraction
311 where the first character is A and the second is not B may not be
312 found.
313 Under "(normalization => undef)", detection step of discontiguous
315 contractions will be skipped.
316 Note: The following contractions in DUCET are not considered in
318 steps S2.1.1 to S2.1.3, where they are discontiguous.
319 0FB2 0F71 0F80 (TIBETAN VOWEL SIGN VOCALIC RR)
321 0FB3 0F71 0F80 (TIBETAN VOWEL SIGN VOCALIC LL)
322 For example "TIBETAN VOWEL SIGN VOCALIC RR" with "COMBINING TILDE
324 OVERLAY" ("U+0344") is "0FB2 0344 0F71 0F80" in NFD. In this case
325 "0FB2 0F80" ("TIBETAN VOWEL SIGN VOCALIC R") is detected, instead
326 of "0FB2 0F71 0F80". Inserted 0344 makes "0FB2 0F71 0F80"
327 discontiguous and lack of contraction "0FB2 0F71" prohibits "0FB2
328 0F71 0F80" from being detected.
329 minimalFFFE
331 -- see 1.1.1 U+FFFE, UTS #35 (LDML) Part 5: Collation.
332 If the parameter is made true, "U+FFFE" has a minimal primary
334 weight. The comparison between "$a1\x{FFFE}$a2" and
335 "$b1\x{FFFE}$b2" first compares $a1 and $b1 at level 1, and then
336 $a2 and $b2 at level 1, as followed.
337 "ab\x{FFFE}a"
339 "Ab\x{FFFE}a"
340 "ab\x{FFFE}c"
341 "Ab\x{FFFE}c"
342 "ab\x{FFFE}xyz"
343 "abc\x{FFFE}def"
344 "abc\x{FFFE}xYz"
345 "aBc\x{FFFE}xyz"
346 "abcX\x{FFFE}def"
347 "abcx\x{FFFE}xyz"
348 "b\x{FFFE}aaa"
349 "bbb\x{FFFE}a"
350 Note: This is equivalent to "(entry => 'FFFE ;
352 [.0001.0020.0005.FFFE]')". Any other character than "U+FFFE" can
353 be tailored by "entry".
354 normalization
356 -- see 4.1 Normalize, UTS #10.
357 If specified, strings are normalized before preparation of sort
359 keys (the normalization is executed after preprocess).
360 A form name "Unicode::Normalize::normalize()" accepts will be
362 applied as $normalization_form. Acceptable names include 'NFD',
363 'NFC', 'NFKD', and 'NFKC'. See "Unicode::Normalize::normalize()"
364 for detail. If omitted, 'NFD' is used.
365 "normalization" is performed after "preprocess" (if defined).
367 Furthermore, special values, "undef" and "prenormalized", can be
369 used, though they are not concerned with
370 "Unicode::Normalize::normalize()".
371 If "undef" (not a string "undef") is passed explicitly as the value
373 for this key, any normalization is not carried out (this may make
374 tailoring easier if any normalization is not desired). Under
375 "(normalization => undef)", only contiguous contractions are
376 resolved; e.g. even if "A-ring" (and "A-ring-cedilla") is ordered
377 after "Z", "A-cedilla-ring" would be primary equal to "A". In this
378 point, "(normalization => undef, preprocess => sub { NFD(shift) })"
379 is not equivalent to "(normalization => 'NFD')".
380 In the case of "(normalization => "prenormalized")", any
382 normalization is not performed, but discontiguous contractions with
383 combining characters are performed. Therefore "(normalization =>
384 'prenormalized', preprocess => sub { NFD(shift) })" is equivalent
385 to "(normalization => 'NFD')". If source strings are finely
386 prenormalized, "(normalization => 'prenormalized')" may save time
387 for normalization.
388 Except "(normalization => undef)", Unicode::Normalize is required
390 (see also CAVEAT).
391 overrideCJK
393 -- see 7.1 Derived Collation Elements, UTS #10.
394 By default, CJK unified ideographs are ordered in Unicode codepoint
396 order, but those in the CJK Unified Ideographs block are less than
397 those in the CJK Unified Ideographs Extension A etc.
398 In the CJK Unified Ideographs block:
400 U+4E00..U+9FA5 if UCA_Version is 8, 9 or 11.
401 U+4E00..U+9FBB if UCA_Version is 14 or 16.
402 U+4E00..U+9FC3 if UCA_Version is 18.
403 U+4E00..U+9FCB if UCA_Version is 20 or 22.
404 U+4E00..U+9FCC if UCA_Version is 24 to 30.
405 U+4E00..U+9FD5 if UCA_Version is 32 or 34.
406 U+4E00..U+9FEA if UCA_Version is 36.
407 In the CJK Unified Ideographs Extension blocks:
409 Ext.A (U+3400..U+4DB5) and Ext.B (U+20000..U+2A6D6) in any UCA_Version.
410 Ext.C (U+2A700..U+2B734) if UCA_Version is 20 or later.
411 Ext.D (U+2B740..U+2B81D) if UCA_Version is 22 or later.
412 Ext.E (U+2B820..U+2CEA1) if UCA_Version is 32 or later.
413 Ext.F (U+2CEB0..U+2EBE0) if UCA_Version is 36.
414 Through "overrideCJK", ordering of CJK unified ideographs
416 (including extensions) can be overridden.
417 ex. CJK unified ideographs in the JIS code point order.
419 overrideCJK => sub {
421 my $u = shift; # get a Unicode codepoint
422 my $b = pack('n', $u); # to UTF-16BE
423 my $s = your_unicode_to_sjis_converter($b); # convert
424 my $n = unpack('n', $s); # convert sjis to short
425 [ $n, 0x20, 0x2, $u ]; # return the collation element
426 },
427 The return value may be an arrayref of 1st to 4th weights as shown
429 above. The return value may be an integer as the primary weight as
430 shown below. If "undef" is returned, the default derived collation
431 element will be used.
432 overrideCJK => sub {
434 my $u = shift; # get a Unicode codepoint
435 my $b = pack('n', $u); # to UTF-16BE
436 my $s = your_unicode_to_sjis_converter($b); # convert
437 my $n = unpack('n', $s); # convert sjis to short
438 return $n; # return the primary weight
439 },
440 The return value may be a list containing zero or more of an
442 arrayref, an integer, or "undef".
443 ex. ignores all CJK unified ideographs.
445 overrideCJK => sub {()}, # CODEREF returning empty list
447 # where ->eq("Pe\x{4E00}rl", "Perl") is true
449 # as U+4E00 is a CJK unified ideograph and to be ignorable.
450 If a false value (including "undef") is passed, "overrideCJK" has
452 no effect. "$Collator->change(overrideCJK => 0)" resets the old
453 one.
454 But assignment of weight for CJK unified ideographs in "table" or
456 "entry" is still valid. If "undef" is passed explicitly as the
457 value for this key, weights for CJK unified ideographs are treated
458 as undefined. However when "UCA_Version" > 8, "(overrideCJK =>
459 undef)" has no special meaning.
460 Note: In addition to them, 12 CJK compatibility ideographs
462 ("U+FA0E", "U+FA0F", "U+FA11", "U+FA13", "U+FA14", "U+FA1F",
463 "U+FA21", "U+FA23", "U+FA24", "U+FA27", "U+FA28", "U+FA29") are
464 also treated as CJK unified ideographs. But they can't be
465 overridden via "overrideCJK" when you use DUCET, as the table
466 includes weights for them. "table" or "entry" has priority over
467 "overrideCJK".
468 overrideHangul
470 -- see 7.1 Derived Collation Elements, UTS #10.
471 By default, Hangul syllables are decomposed into Hangul Jamo, even
473 if "(normalization => undef)". But the mapping of Hangul syllables
474 may be overridden.
475 This parameter works like "overrideCJK", so see there for examples.
477 If you want to override the mapping of Hangul syllables, NFD and
479 NFKD are not appropriate, since NFD and NFKD will decompose Hangul
480 syllables before overriding. FCD may decompose Hangul syllables as
481 the case may be.
482 If a false value (but not "undef") is passed, "overrideHangul" has
484 no effect. "$Collator->change(overrideHangul => 0)" resets the old
485 one.
486 If "undef" is passed explicitly as the value for this key, weight
488 for Hangul syllables is treated as undefined without decomposition
489 into Hangul Jamo. But definition of weight for Hangul syllables in
490 "table" or "entry" is still valid.
491 overrideOut
493 -- see 7.1.1 Handling Ill-Formed Code Unit Sequences, UTS #10.
494 Perl seems to allow out-of-range values (greater than 0x10FFFF).
496 By default, out-of-range values are replaced with "U+FFFD"
497 (REPLACEMENT CHARACTER) when "UCA_Version" >= 22, or ignored when
498 "UCA_Version" <= 20.
499 When "UCA_Version" >= 22, the weights of out-of-range values can be
501 overridden. Though "table" or "entry" are available for them, out-
502 of-range values are too many.
503 "overrideOut" can perform it algorithmically. This parameter works
505 like "overrideCJK", so see there for examples.
506 ex. ignores all out-of-range values.
508 overrideOut => sub {()}, # CODEREF returning empty list
510 If a false value (including "undef") is passed, "overrideOut" has
512 no effect. "$Collator->change(overrideOut => 0)" resets the old
513 one.
514 NOTE ABOUT U+FFFD:
516 UCA recommends that out-of-range values should not be ignored for
518 security reasons. Say, "pe\x{110000}rl" should not be equal to
519 "perl". However, "U+FFFD" is wrongly mapped to a variable
520 collation element in DUCET for Unicode 6.0.0 to 6.2.0, that means
521 out-of-range values will be ignored when "variable" isn't
522 "Non-ignorable".
523 The mapping of "U+FFFD" is corrected in Unicode 6.3.0. see
525 <http://www.unicode.org/reports/tr10/tr10-28.html#Trailing_Weights>
526 (7.1.4 Trailing Weights). Such a correction is reproduced by this.
527 overrideOut => sub { 0xFFFD }, # CODEREF returning a very large integer
529 This workaround is unnecessary since Unicode 6.3.0.
531 preprocess
533 -- see 5.4 Preprocessing, UTS #10.
534 If specified, the coderef is used to preprocess each string before
536 the formation of sort keys.
537 ex. dropping English articles, such as "a" or "the". Then, "the
539 pen" is before "a pencil".
540 preprocess => sub {
542 my $str = shift;
543 $str =~ s/\b(?:an?|the)\s+//gi;
544 return $str;
545 },
546 "preprocess" is performed before "normalization" (if defined).
548 ex. decoding strings in a legacy encoding such as shift-jis:
550 $sjis_collator = Unicode::Collate->new(
552 preprocess => \&your_shiftjis_to_unicode_decoder,
553 );
554 @result = $sjis_collator->sort(@shiftjis_strings);
555 Note: Strings returned from the coderef will be interpreted
557 according to Perl's Unicode support. See perlunicode, perluniintro,
558 perlunitut, perlunifaq, utf8.
559 rearrange
561 -- see 3.5 Rearrangement, UTS #10.
562 Characters that are not coded in logical order and to be
564 rearranged. If "UCA_Version" is equal to or less than 11, default
565 is:
566 rearrange => [ 0x0E40..0x0E44, 0x0EC0..0x0EC4 ],
568 If you want to disallow any rearrangement, pass "undef" or "[]" (a
570 reference to empty list) as the value for this key.
571 If "UCA_Version" is equal to or greater than 14, default is "[]"
573 (i.e. no rearrangement).
574 According to the version 9 of UCA, this parameter shall not be
576 used; but it is not warned at present.
577 rewrite
579 If specified, the coderef is used to rewrite lines in "table" or
580 "entry". The coderef will get each line, and then should return a
581 rewritten line according to the UCA file format. If the coderef
582 returns an empty line, the line will be skipped.
583 e.g. any primary ignorable characters into tertiary ignorable:
585 rewrite => sub {
587 my $line = shift;
588 $line =~ s/\[\.0000\..{4}\..{4}\./[.0000.0000.0000./g;
589 return $line;
590 },
591 This example shows rewriting weights. "rewrite" is allowed to
593 affect code points, weights, and the name.
594 NOTE: "table" is available to use another table file; preparing a
596 modified table once would be more efficient than rewriting lines on
597 reading an unmodified table every time.
598 suppress
600 -- see 3.12 Special-Purpose Commands, UTS #35 (LDML) Part 5:
601 Collation.
602 Contractions beginning with the specified characters are
604 suppressed, even if those contractions are defined in "table".
605 An example for Russian and some languages using the Cyrillic
607 script:
608 suppress => [0x0400..0x0417, 0x041A..0x0437, 0x043A..0x045F],
610 where 0x0400 stands for "U+0400", CYRILLIC CAPITAL LETTER IE WITH
612 GRAVE.
613 NOTE: Contractions via "entry" will not be suppressed.
615 table
617 -- see 3.8 Default Unicode Collation Element Table, UTS #10.
618 You can use another collation element table if desired.
620 The table file should locate in the Unicode/Collate directory on
622 @INC. Say, if the filename is Foo.txt, the table file is searched
623 as Unicode/Collate/Foo.txt in @INC.
624 By default, allkeys.txt (as the filename of DUCET) is used. If you
626 will prepare your own table file, any name other than allkeys.txt
627 may be better to avoid namespace conflict.
628 NOTE: When XSUB is used, the DUCET is compiled on building this
630 module, and it may save time at the run time. Explicit saying
631 "(table => 'allkeys.txt')", or using another table, or using
632 "ignoreChar", "ignoreName", "undefChar", "undefName" or "rewrite"
633 will prevent this module from using the compiled DUCET.
634 If "undef" is passed explicitly as the value for this key, no file
636 is read (but you can define collation elements via "entry").
637 A typical way to define a collation element table without any file
639 of table:
640 $onlyABC = Unicode::Collate->new(
642 table => undef,
643 entry => << 'ENTRIES',
644 0061 ; [.0101.0020.0002.0061] # LATIN SMALL LETTER A
645 0041 ; [.0101.0020.0008.0041] # LATIN CAPITAL LETTER A
646 0062 ; [.0102.0020.0002.0062] # LATIN SMALL LETTER B
647 0042 ; [.0102.0020.0008.0042] # LATIN CAPITAL LETTER B
648 0063 ; [.0103.0020.0002.0063] # LATIN SMALL LETTER C
649 0043 ; [.0103.0020.0008.0043] # LATIN CAPITAL LETTER C
650 ENTRIES
651 );
652 If "ignoreName" or "undefName" is used, character names should be
654 specified as a comment (following "#") on each line.
655 undefChar
657 undefName
658 -- see 6.3.3 Reducing the Repertoire, UTS #10.
659 Undefines the collation element as if it were unassigned in the
661 "table". This reduces the size of the table. If an unassigned
662 character appears in the string to be collated, the sort key is
663 made from its codepoint as a single-character collation element, as
664 it is greater than any other assigned collation elements (in the
665 codepoint order among the unassigned characters). But, it'd be
666 better to ignore characters unfamiliar to you and maybe never used.
667 Through "undefChar", any character matching "qr/$undefChar/" will
669 be undefined. Through "undefName", any character whose name (given
670 in the "table" file as a comment) matches "qr/$undefName/" will be
671 undefined.
672 ex. Collation weights for beyond-BMP characters are not stored in
674 object:
675 undefChar => qr/[^\0-\x{fffd}]/,
677 upper_before_lower
679 -- see 6.6 Case Comparisons, UTS #10.
680 By default, lowercase is before uppercase. If the parameter is
682 made true, this is reversed.
683 NOTE: This parameter simplemindedly assumes that any
685 lowercase/uppercase distinctions must occur in level 3, and their
686 weights at level 3 must be same as those mentioned in 7.3.1, UTS
687 #10. If you define your collation elements which differs from this
688 requirement, this parameter doesn't work validly.
689 variable
691 -- see 3.6 Variable Weighting, UTS #10.
692 This key allows for variable weighting of variable collation
694 elements, which are marked with an ASTERISK in the table (NOTE:
695 Many punctuation marks and symbols are variable in allkeys.txt).
696 variable => 'blanked', 'non-ignorable', 'shifted', or 'shift-trimmed'.
698 These names are case-insensitive. By default (if specification is
700 omitted), 'shifted' is adopted.
701 'Blanked' Variable elements are made ignorable at levels 1 through 3;
703 considered at the 4th level.
704 'Non-Ignorable' Variable elements are not reset to ignorable.
706 'Shifted' Variable elements are made ignorable at levels 1 through 3
708 their level 4 weight is replaced by the old level 1 weight.
709 Level 4 weight for Non-Variable elements is 0xFFFF.
710 'Shift-Trimmed' Same as 'shifted', but all FFFF's at the 4th level
712 are trimmed.
713 Methods for Collation
715 "@sorted = $Collator->sort(@not_sorted)"
716 Sorts a list of strings.
717 "$result = $Collator->cmp($a, $b)"
719 Returns 1 (when $a is greater than $b) or 0 (when $a is equal to
720 $b) or -1 (when $a is less than $b).
721 "$result = $Collator->eq($a, $b)"
723 "$result = $Collator->ne($a, $b)"
724 "$result = $Collator->lt($a, $b)"
725 "$result = $Collator->le($a, $b)"
726 "$result = $Collator->gt($a, $b)"
727 "$result = $Collator->ge($a, $b)"
728 They works like the same name operators as theirs.
729 eq : whether $a is equal to $b.
731 ne : whether $a is not equal to $b.
732 lt : whether $a is less than $b.
733 le : whether $a is less than $b or equal to $b.
734 gt : whether $a is greater than $b.
735 ge : whether $a is greater than $b or equal to $b.
736 "$sortKey = $Collator->getSortKey($string)"
738 -- see 4.3 Form Sort Key, UTS #10.
739 Returns a sort key.
741 You compare the sort keys using a binary comparison and get the
743 result of the comparison of the strings using UCA.
744 $Collator->getSortKey($a) cmp $Collator->getSortKey($b)
746 is equivalent to
748 $Collator->cmp($a, $b)
750 "$sortKeyForm = $Collator->viewSortKey($string)"
752 Converts a sorting key into its representation form. If
753 "UCA_Version" is 8, the output is slightly different.
754 use Unicode::Collate;
756 my $c = Unicode::Collate->new();
757 print $c->viewSortKey("Perl"),"\n";
758 # output:
760 # [0B67 0A65 0B7F 0B03 | 0020 0020 0020 0020 | 0008 0002 0002 0002 | FFFF FFFF FFFF FFFF]
761 # Level 1 Level 2 Level 3 Level 4
762 Methods for Searching
764 The "match", "gmatch", "subst", "gsubst" methods work like "m//",
765 "m//g", "s///", "s///g", respectively, but they are not aware of any
766 pattern, but only a literal substring.
767 DISCLAIMER: If "preprocess" or "normalization" parameter is true for
769 $Collator, calling these methods ("index", "match", "gmatch", "subst",
770 "gsubst") is croaked, as the position and the length might differ from
771 those on the specified string.
772 "rearrange" and "hangul_terminator" parameters are neglected.
774 "katakana_before_hiragana" and "upper_before_lower" don't affect
775 matching and searching, as it doesn't matter whether greater or less.
776 "$position = $Collator->index($string, $substring[, $position])"
778 "($position, $length) = $Collator->index($string, $substring[,
779 $position])"
780 If $substring matches a part of $string, returns the position of
781 the first occurrence of the matching part in scalar context; in
782 list context, returns a two-element list of the position and the
783 length of the matching part.
784 If $substring does not match any part of $string, returns "-1" in
786 scalar context and an empty list in list context.
787 e.g. when the content of $str is ""Ich mu"ß" studieren Perl."", you
789 say the following where $sub is ""M"ü"SS"",
790 my $Collator = Unicode::Collate->new( normalization => undef, level => 1 );
792 # (normalization => undef) is REQUIRED.
793 my $match;
794 if (my($pos,$len) = $Collator->index($str, $sub)) {
795 $match = substr($str, $pos, $len);
796 }
797 and get ""mu"ß""" in $match, since ""mu"ß""" is primary equal to
799 ""M"ü"SS"".
800 "$match_ref = $Collator->match($string, $substring)"
802 "($match) = $Collator->match($string, $substring)"
803 If $substring matches a part of $string, in scalar context, returns
804 a reference to the first occurrence of the matching part
805 ($match_ref is always true if matches, since every reference is
806 true); in list context, returns the first occurrence of the
807 matching part.
808 If $substring does not match any part of $string, returns "undef"
810 in scalar context and an empty list in list context.
811 e.g.
813 if ($match_ref = $Collator->match($str, $sub)) { # scalar context
815 print "matches [$$match_ref].\n";
816 } else {
817 print "doesn't match.\n";
818 }
819 or
821 if (($match) = $Collator->match($str, $sub)) { # list context
823 print "matches [$match].\n";
824 } else {
825 print "doesn't match.\n";
826 }
827 "@match = $Collator->gmatch($string, $substring)"
829 If $substring matches a part of $string, returns all the matching
830 parts (or matching count in scalar context).
831 If $substring does not match any part of $string, returns an empty
833 list.
834 "$count = $Collator->subst($string, $substring, $replacement)"
836 If $substring matches a part of $string, the first occurrence of
837 the matching part is replaced by $replacement ($string is modified)
838 and $count (always equals to 1) is returned.
839 $replacement can be a "CODEREF", taking the matching part as an
841 argument, and returning a string to replace the matching part (a
842 bit similar to "s/(..)/$coderef->($1)/e").
843 "$count = $Collator->gsubst($string, $substring, $replacement)"
845 If $substring matches a part of $string, all the occurrences of the
846 matching part are replaced by $replacement ($string is modified)
847 and $count is returned.
848 $replacement can be a "CODEREF", taking the matching part as an
850 argument, and returning a string to replace the matching part (a
851 bit similar to "s/(..)/$coderef->($1)/eg").
852 e.g.
854 my $Collator = Unicode::Collate->new( normalization => undef, level => 1 );
856 # (normalization => undef) is REQUIRED.
857 my $str = "Camel donkey zebra came\x{301}l CAMEL horse cam\0e\0l...";
858 $Collator->gsubst($str, "camel", sub { "<b>$_[0]</b>" });
859 # now $str is "<b>Camel</b> donkey zebra <b>came\x{301}l</b> <b>CAMEL</b> horse <b>cam\0e\0l</b>...";
861 # i.e., all the camels are made bold-faced.
862 Examples: levels and ignore_level2 - what does camel match?
864 ---------------------------------------------------------------------------
865 level ignore_level2 | camel Camel came\x{301}l c-a-m-e-l cam\0e\0l
866 -----------------------|---------------------------------------------------
867 1 false | yes yes yes yes yes
868 2 false | yes yes no yes yes
869 3 false | yes no no yes yes
870 4 false | yes no no no yes
871 -----------------------|---------------------------------------------------
872 1 true | yes yes yes yes yes
873 2 true | yes yes yes yes yes
874 3 true | yes no yes yes yes
875 4 true | yes no yes no yes
876 ---------------------------------------------------------------------------
877 note: if variable => non-ignorable, camel doesn't match c-a-m-e-l
878 at any level.
879 Other Methods
881 "%old_tailoring = $Collator->change(%new_tailoring)"
882 "$modified_collator = $Collator->change(%new_tailoring)"
883 Changes the value of specified keys and returns the changed part.
884 $Collator = Unicode::Collate->new(level => 4);
886 $Collator->eq("perl", "PERL"); # false
888 %old = $Collator->change(level => 2); # returns (level => 4).
890 $Collator->eq("perl", "PERL"); # true
892 $Collator->change(%old); # returns (level => 2).
894 $Collator->eq("perl", "PERL"); # false
896 Not all "(key,value)"s are allowed to be changed. See also
898 @Unicode::Collate::ChangeOK and @Unicode::Collate::ChangeNG.
899 In the scalar context, returns the modified collator (but it is not
901 a clone from the original).
902 $Collator->change(level => 2)->eq("perl", "PERL"); # true
904 $Collator->eq("perl", "PERL"); # true; now max level is 2nd.
906 $Collator->change(level => 4)->eq("perl", "PERL"); # false
908 "$version = $Collator->version()"
910 Returns the version number (a string) of the Unicode Standard which
911 the "table" file used by the collator object is based on. If the
912 table does not include a version line (starting with @version),
913 returns "unknown".
914 "UCA_Version()"
916 Returns the revision number of UTS #10 this module consults, that
917 should correspond with the DUCET incorporated.
918 "Base_Unicode_Version()"
920 Returns the version number of UTS #10 this module consults, that
921 should correspond with the DUCET incorporated.
922
924 No method will be exported.
925
927 Though this module can be used without any "table" file, to use this
928 module easily, it is recommended to install a table file in the UCA
929 format, by copying it under the directory <a place in
930 @INC>/Unicode/Collate.
931 The most preferable one is "The Default Unicode Collation Element
933 Table" (aka DUCET), available from the Unicode Consortium's website:
934 http://www.unicode.org/Public/UCA/
936 http://www.unicode.org/Public/UCA/latest/allkeys.txt
938 (latest version)
939 If DUCET is not installed, it is recommended to copy the file from
941 http://www.unicode.org/Public/UCA/latest/allkeys.txt to <a place in
942 @INC>/Unicode/Collate/allkeys.txt manually.
943
945 Normalization
946 Use of the "normalization" parameter requires the
947 Unicode::Normalize module (see Unicode::Normalize).
948 If you need not it (say, in the case when you need not handle any
950 combining characters), assign "(normalization => undef)"
951 explicitly.
952 -- see 6.5 Avoiding Normalization, UTS #10.
954 Conformance Test
956 The Conformance Test for the UCA is available under
957 <http://www.unicode.org/Public/UCA/>.
958 For CollationTest_SHIFTED.txt, a collator via
960 "Unicode::Collate->new( )" should be used; for
961 CollationTest_NON_IGNORABLE.txt, a collator via
962 "Unicode::Collate->new(variable => "non-ignorable", level => 3)".
963 If "UCA_Version" is 26 or later, the "identical" level is
965 preferred; "Unicode::Collate->new(identical => 1)" and
966 "Unicode::Collate->new(identical => 1," "variable =>
967 "non-ignorable", level => 3)" should be used.
968 Unicode::Normalize is required to try The Conformance Test.
970
972 The Unicode::Collate module for perl was written by SADAHIRO Tomoyuki,
973 <SADAHIRO@cpan.org>. This module is Copyright(C) 2001-2018, SADAHIRO
974 Tomoyuki. Japan. All rights reserved.
975 This module is free software; you can redistribute it and/or modify it
977 under the same terms as Perl itself.
978 The file Unicode/Collate/allkeys.txt was copied verbatim from
980 <http://www.unicode.org/Public/UCA/9.0.0/allkeys.txt>. For this file,
981 Copyright (c) 2016 Unicode, Inc.; distributed under the Terms of Use in
982 <http://www.unicode.org/terms_of_use.html>
983
985 Unicode Collation Algorithm - UTS #10
986 <http://www.unicode.org/reports/tr10/>
987 The Default Unicode Collation Element Table (DUCET)
989 <http://www.unicode.org/Public/UCA/latest/allkeys.txt>
990 The conformance test for the UCA
992 <http://www.unicode.org/Public/UCA/latest/CollationTest.html>
993 <http://www.unicode.org/Public/UCA/latest/CollationTest.zip>
995 Hangul Syllable Type
997 <http://www.unicode.org/Public/UNIDATA/HangulSyllableType.txt>
998 Unicode Normalization Forms - UAX #15
1000 <http://www.unicode.org/reports/tr15/>
1001 Unicode Locale Data Markup Language (LDML) - UTS #35
1003 <http://www.unicode.org/reports/tr35/>
1004perl v5.30.1 2020-02-06 Collate(3)