1UNICODE_CANONICAL(3) Courier Unicode Library UNICODE_CANONICAL(3)
2
6 unicode_canonical, unicode_ccc, unicode_decomposition_init,
7 unicode_decomposition_deinit, unicode_decompose,
8 unicode_decompose_reallocate_size, unicode_compose,
9 unicode_composition_init, unicode_composition_deinit,
10 unicode_composition_apply - unicode canonical normalization and
11 denormalization
12
14 #include <courier-unicode.h>
15 unicode_canonical_t unicode_canonical(char32_t c);
17 uint8_t unicode_ccc(char32_t c);
19 void unicode_decomposition_init(unicode_decomposition_t *info,
21 char32_t *string, size_t *string_size,
22 void *arg);
23 int unicode_decompose(unicode_decomposition_t *info);
25 void unicode_decomposition_deinit(unicode_decomposition_t *info);
27 size_t unicode_decompose_reallocate_size(unicode_decomposition_t *info,
29 const size_t *sizes,
30 size_t n);
31 int unicode_compose(char32_t *string, size_t string_size, int flags,
33 size_t *new_size);
34 int unicode_composition_init(const char32_t *string,
36 size_t string_size, int flags,
37 unicode_composition_t *compositions);
38 void unicode_composition_deinit(unicode_composition_t *compositions);
40 size_t unicode_composition_apply(char32_t *string, size_t string_size,
42 unicode_composition_t *compositions);
43
45 These functions compose or decompose a Unicode string into a canonical
46 or a compatible normalized form.
47 unicode_canonical() looks up the character's canonical and
49 compatibility mapping[1]. unicode_canonical() returns a structure with
50 the following fields:
51 canonical_chars
53 A pointer to the canonical or equivalent representation of the
54 character.
55 n_canonical_chars
57 Number of characters in the canonical_chars.
58 format
60 A value of UNICODE_CANONICAL_FMT_NONE indicates a canonical
61 mapping, other values indicate a compatibility equivalent mapping.
62 A NULL canonical_chars (with a 0 n_canonical_chars) indicates that the
64 character has no canonical or compatibility equivalence.
65 unicode_ccc() returns the character's canonical combining class value.
67 unicode_decomposition_init(), unicode_decompose() and
69 unicode_decomposition_deinit() implement a complete interface for
70 decomposing a Unicode string:
71 unicode_decomposition_t info;
73 unicode_decomposition_init(&info, before, (size_t)-1, NULL);
75 info.decompose_flags=UNICODE_DECOMPOSE_FLAG_QC;
76 unicode_decompose(&info);
77 unicode_decomposition_deinit(&info);
78 unicode_decomposition_init() initializes a new unicode_decomposition_t
80 structure, that gets passed in as its first parameter. The second
81 parameter is a pointer to a Unicode string, with the number of
82 characters in the string in the third parameter. A string size of -1
83 indicates a \0-terminated string and calculates its string_size (which
84 does not include the trailing \0. The last parameter is a void *, an
85 opaque pointer that gets stored in the initialized
86 unicode_decomposition_t object:
87 typedef struct unicode_decomposition {
89 char32_t *string;
90 size_t string_size;
91 int decompose_flags;
92 int (*reallocate)(
93 struct unicode_decomposition *info,
94 const size_t *offsets,
95 const size_t *sizes,
96 size_t n
97 );
98 void *arg;
99 } unicode_decomposition_t;
100 unicode_decompose() proceeds and decomposes the string and replaces it
103 with its decomposed string version.
104 unicode_decomposition_t's string, string_size and arg are copies of
106 unicode_decomposition_init's parameters. unicode_decomposition_init
107 initializes all other fields to their default values.
108 The decompose_flags bitmask gets initialized to 0, and is a bit mask:
110 UNICODE_DECOMPOSE_FLAG_QC
112 Check each character's appropriate “quick check” property and skip
113 decomposing Unicode characters that would get re-composed by
114 unicode_composition_apply().
115 UNICODE_DECOMPOSE_FLAG_COMPAT
117 Perform a compatibility decomposition instead of a canonical
118 decomposition.
119 reallocate is a pointer to a function that gets called to reallocate a
121 larger string. unicode_decompose() determines which characters in the
122 string need decomposing and calls the reallocate function pointer zero
123 or more times. Each call to reallocate passes information about where
124 new characters will get inserted into the string.
125 reallocate only needs to grow the size of the buffer where string
127 points so that it's big enough to hold a larger, decomposed string;
128 then update string accordingly. reallocate should not update
129 string_size or make any changes to the existing string, that's
130 unicode_decompose()'s job (after reallocate returns).
131 The reallocate callback function receives the following parameters.
133 • A pointer to the unicode_decomposition_t and, notably, its arg.
135 • A pointer to the array of offset indexes in the string where new
137 characters will get inserted in order to hold the decomposed
138 string.
139 • A pointer to the array that holds the number of characters that get
141 inserted each corresponding offset.
142 • The size of the two arrays.
144 reallocate must update the string if necessary to hold at least the
146 number of characters that's the sum total of the initial string_size
147 and the sum total of al sizes.
148 unicode_decomposition_init() initializes the reallocate pointer to a
150 default implementation that uses realloc(3) and updates string with its
151 return value. The application can use its own reallocate to handle this
152 task on its own, and use unicode_decompose_reallocate_size to compute
153 the minimum string size:
154 size_t unicode_decompose_reallocate_size(unicode_decomposition_t *info,
156 const size_t *sizes,
157 size_t n)
158 {
159 size_t i;
160 size_t new_size=info->string_size;
161 for (i=0; i<n; ++i)
163 new_size += sizes[i];
164 return new_size;
166 }
167 The reallocate function returns 0 on success and a non-0 error code to
170 report a failure; and unicode_decompose() does the same. The only error
171 condition from unicode_decompose() is a non-0 error code from the
172 reallocate function. Otherwise: a successful decomposition results in
173 unicode_decompose() returning 0 and unicode_decomposition_init()'s
174 string pointing to the decomposed string and string_size giving the
175 number of characters in the decomposed string.
176 Note
178 string_size does not include the trailing \0 character. The input
179 string also has its string_size specified without counting its \0
180 character. The default implementation of reallocate allocates an
181 extra char32_t ands sets it to a \0. Therefore:
182 • If the Unicode string before decomposition has a trailing \0
184 and no decomposition occurs, and no calls to reallocate takes
185 place: the string in the unicode_decomposition_t is unchanged
186 and it's still \0-terminated.
187 • The default reallocate allocates an extra char32_t ands sets it
189 to a \0; and it takes care of that for the decomposed string.
190 • An application that provides its own replacement reallocate is
192 responsible for doing the same, if it wants the decomposed
193 string to be \0 terminated.
194 Note
196 Multiple calls to the reallocate callback are possible. Each call
197 to reallocate reflect the prior calls' decompositions. Example: the
198 original string has five characters and the first call to
199 reallocate had two offsets, at position 1 and 3, with a value of 1
200 for their both sizes. This effects transforming an original Unicode
201 string "AAAAA" into "AXAAXAA" (with “A” representing unspecified
202 characters in the original string, and “X” showing the two
203 characters added in the first call to reallocate.
204 A second call to varname with am offset at position 4, and a size
206 of 1, results in the updated string of "AXAAYXAA" (with “Y”)
207 marking an unspecified character inserted by the second call.
208 Note
210 Unicode string decomposition involves replacing a given Unicode
211 character with one or more other characters. The sizes given to
212 reallocate reflect the net addition to the Unicode string. For
213 example: decomposing one Unicode character into three decomposed
214 characters results in a call to reallocate reporting an insert of
215 two more characters.
216 Note
218 offsets actually report the indices of each Unicode character
219 that's getting decomposed. A 1:1 decomposition of a Unicode
220 Character gets reported as an additional sizes entry of 0.
221 unicode_decomposition_deinit() releases all resources and destroys the
223 unicode_decomposition_t; it is no longer valid.
224 Note
226 unicode_decomposition_deinit() does not free(3) the string. The
227 original string gets passed in to unicode_decomposition_init() and
228 the decomposed string is left in the string.
229 The default implementation of the reallocate function assumes the
231 string is a malloc(3)-ed string, and reallocs it.
232 Note
234 At this time unicode_decomposition_deinit() does nothing. All code
235 should explicitly call it in order to remain forward-compatible (at
236 the source level).
237 unicode_compose() performs a canonical composition of a decomposed
239 string. Its parameters are:
240 • A pointer to the decomposed Unicode string.
242 • The number of characters in the Unicode string. The Unicode string
244 does not need to be \0-terminated; if it is this number does not
245 include it.
246 • A flags bitmask, which can have the following values:
248 UNICODE_COMPOSE_FLAG_REMOVEUNUSED
250 Remove all combining marks after doing all canonical
251 compositions. Normally any unused combining marks are left in
252 place, in the combined text. This option removes them.
253 UNICODE_COMPOSE_FLAG_ONESHOT
255 Perform canonical composition once per character, and do not
256 attempt to combine any resulting combined characters again.
257 • A non-NULL pointer to a size_t.
259 A successful composition sets this size_t to the number of
261 characters in the combined string, and returns 0. The combined
262 string gets placed back into the string parameter, this string gets
263 combined in place and this gives the size of the combined string.
264 unicode_compose() returns a non-zero value to indicate an error.
266 unicode_composition_init(), unicode_composition_apply() and
268 unicode_composition_deinit() implement a detailed interface for
269 canonical composition of a decomposed Unicode string:
270 unicode_compositions_t compositions;
272 if (unicode_composition_init(str, strsize, flags, &compositions) == 0)
274 {
275 size_t new_size=unicode_composition_apply(str, strsize, &compositions);
276 unicode_composition_deinit(&compositions);
278 }
279 The first two parameters to both unicode_composition_init() and
281 unicode_composition_apply() are the same: the Unicode string and the
282 number of characters (not including any trailing \0 character) in the
283 Unicode string.
284 unicode_composition_init()'s additional parameters are: any optional
286 flags (see unicode_compose() for a list of available flags), and the
287 address of a unicode_composition_t object. A non-0 return from
288 unicode_composition_init() indicates an error.
289 unicode_composition_init() indicates success by returning 0 and
290 initializing the unicode_composition_t's object which contains a
291 pointer to an array of pointers to of unicode_compose_info objects, and
292 the number of pointers. unicode_composition_init() does not change the
293 string; the only thing it does is initialize the unicode_composition_t
294 object.
295 unicode_composition_apply() applies the compositions to the string, in
297 place, and returns the new size of the string (also not including the
298 \0 byte, however it does append one if the composed string is smaller,
299 so the composed string is \0-terminated if the decomposed string was).
300 It is necessary to call unicode_composition_deinit() to free all memory
302 that was allocated for the unicode_composition_t object:
303 struct unicode_compose_info {
305 size_t index;
306 size_t n_composed;
307 char32_t *composition;
308 size_t n_composition;
309 };
310 typedef struct {
312 struct unicode_compose_info **compositions;
313 size_t n_compositions;
314 } unicode_composition_t;
315 index gives the character index in the string where each composition
318 occurs. n_composed gives the number of characters in the original
319 string that get composed. The composed characters are the composition;
320 and n_composition gives the number of composed characters.
321 Effectively: at the index position in the original string, #n_composed
323 characters get removed and there are #n_composition characters that
324 replace them (always n_composed or less).
325 Note
327 The UNICODE_COMPOSE_FLAG_REMOVEUNUSED flag has the effect of
328 including the combining marks that did not get combined in the
329 n_composed count. It's possible that, in this case, n_composition
330 is 0. This indicates complete removal of the combining marks,
331 without anything getting combined in their place.
332 unicode_composition_init() sets unicode_composition_t's compositions
334 pointer to an array of pointers to unicode_compose_infos that are
335 sorted according to their index. n_compositions gives the number of
336 pointers in the array, and is 0 if there are no compositions, the array
337 is empty. The empty array gets interpreted accordingly when it gets
338 passed to unicode_composition_apply() and unicode_composition_deinit():
339 nothing happens. unicode_composition_apply() simply returns the size
340 of the unchanged string, and unicode_composition_deinit() does a
341 pro-forma cleanup.
342
344 TR-15[2], courier-unicode(7), unicode::canonical(3).
345
347 Sam Varshavchik
348 Author
349
351 1. canonical and compatibility mapping
352 https://www.unicode.org/reports/tr15/tr15-50.html
353 2. TR-15
355 https://www.unicode.org/reports/tr15/tr15-50.html
356Courier Unicode Library 05/31/2022 UNICODE_CANONICAL(3)