1Map(3) User Contributed Perl Documentation Map(3)
2
6 Unicode::Map V0.112 - maps charsets from and to utf16 unicode
7
9 use Unicode::Map();
10 $Map = new Unicode::Map("ISO-8859-1");
12 $utf16 = $Map -> to_unicode ("Hello world!");
14 => $utf16 == "\0H\0e\0l\0l\0o\0 \0w\0o\0r\0l\0d\0!"
15 $locale = $Map -> from_unicode ($utf16);
17 => $locale == "Hello world!"
18 A more detailed description below.
20 2do: short note about perl's Unicode perspectives.
22
24 This module converts strings from and to 2-byte Unicode UCS2 format.
25 All mappings happen via 2 byte UTF16 encodings, not via 1 byte UTF8
26 encoding. To transform these use Unicode::String.
27 For historical reasons this module coexists with Unicode::Map8. Please
29 use Unicode::Map8 unless you need to care for two byte character sets,
30 e.g. chinese GB2312. Anyway, if you stick to the basic functionality
31 (see documentation) you can use both modules equivalently.
32 Practically this module will disappear from earth sooner or later as
34 Unicode mapping support needs somehow to get into perl's core. If you
35 like to work on this field please don't hesitate contacting Gisle Aas!
36 This module can't deal directly with utf8. Use Unicode::String to
38 convert utf8 to utf16 and vice versa.
39 Character mapping is according to the data of binary mapfiles in
41 Unicode::Map hierarchy. Binary mapfiles can also be created with this
42 module, enabling you to install own specific character sets. Refer to
43 mkmapfile or file REGISTRY in the Unicode::Map hierarchy.
44
46 Probably these are the only methods you will need from this module.
47 Their usage is compatible with Unicode::Map8.
48 new $Map = new Unicode::Map("GB2312-80")
50 Returns a new Map object for GB2312-80 encoding.
52 from_unicode
54 $dest = $Map -> from_unicode ($src)
55 Creates a string in locale charset representation from utf16
57 encoded string $src.
58 to_unicode
60 $dest = $Map -> to_unicode ($src)
61 Creates a string in utf16 representation from $src.
63 to8 Alias for from_unicode. For compatibility with Unicode::Map8
65 to16
67 Alias for to_unicode. For compatibility with Unicode::Map8
68
70 You can demand Unicode::Map to issue warnings at deprecated or
71 incompatible usage with the constants WARN_DEFAULT,
72 WARN_DEPRECATION or WARN_COMPATIBILITY. The latter both can be
73 ored together.
74 No special warnings:
76 $Unicode::Map::WARNINGS = Unicode::Map::WARN_DEFAULT
78 Warnings for deprecated usage:
80 $Unicode::Map::WARNINGS = Unicode::Map::WARN_DEPRECATION
82 Warnings for incompatible usage:
84 $Unicode::Map::WARNINGS = Unicode::Map::WARN_COMPATIBILITY
86
88 Note: These methods are solely for the maintainance of Unicode::Map.
89 Using any of these methods will lead to programs incompatible with
90 Unicode::Map8.
91 alias
93 @list = $Map -> alias ($csid)
94 Returns a list of alias names of character set $csid.
96 mapping
98 $path = $Map -> mapping ($csid)
99 Returns the absolute path of binary character mapping for character
101 set $csid according to REGISTRY file of Unicode::Map.
102 id $real_id||"" = $Map -> id ($test_id)
104 Returns a valid character set identifier $real_id, if $test_id is a
106 valid character set name or alias name according to REGISTRY file
107 of Unicode::Map.
108 ids @ids = $Map -> ids()
110 Returns a list of all character set names defined in REGISTRY file.
112 read_text_mapping
114 1||0 = $Map -> read_text_mapping ($csid, $path, $style)
115 Read a text mapping of style $style named $csid from filename
117 $path. The mapping then can be saved to a file with method:
118 write_binary_mapping. <$style> can be:
119 style description
121 "unicode" A text mapping as of ftp://ftp.unicode.org/MAPPINGS/
123 "" Same as "unicode"
124 "reverse" Similar to unicode, but both columns are switched
125 "keld" A text mapping as of ftp://dkuug.dk/i18n/charmaps/
126 src $path = $Map -> src ($csid)
128 Returns the path of textual character mapping for character set
130 $csid according to REGISTRY file of Unicode::Map.
131 style
133 $path = $Map -> style ($csid)
134 Returns the style of textual character mapping for character set
136 $csid according to REGISTRY file of Unicode::Map.
137 write_binary_mapping
139 1||0 = $Map -> write_binary_mapping ($csid, $path)
140 Stores a mapping that has been loaded via method read_text_mapping
142 in file $path.
143
145 Some functionality is no longer promoted.
146 noise
148 Deprecated! Don't use any longer.
149 reverse_unicode
151 Deprecated! Use Unicode::String::byteswap instead.
152
154 Structure of binary Mapfiles
155 Unicode character mapping tables have sequences of sequential key and
157 sequential value codes. This property is used to crunch the maps
158 easily. n (0<n<256) sequential characters are represented as a
159 bytecount n and the first character code key_start. For these
160 subsequences the according value sequences are crunched together, also.
161 The value 0 is used to start an extended information block (that is
162 just partially implemented, though).
163 One could think of two ways to make a binary mapfile. First method
165 would be first to write a list of all key codes, and then to write a
166 list of all value codes. Second method, used here, appends to all
167 partial key code lists the according crunched value code lists. This
168 makes value codes a little bit closer to key codes.
169 Note: the file format is still in a very liquid state. Neither rely on
171 that it will stay as this, nor that the description is bugless, nor
172 that all features are implemented.
173 STRUCTURE:
175 <main>:
177 offset structure value
178 0x00 word 0x27b8 (magic)
180 0x02 @(<extended> || <submapping>)
181 The mapfile ends with extended mode <end> in main stream.
183 <submapping>:
185 0x00 byte != 0 charsize1 (bits)
186 0x01 byte n1 number of chars for one entry
187 0x02 byte charsize2 (bits)
188 0x03 byte n2 number of chars for one entry
189 0x04 @(<extended> || <key_seq> || <key_val_seq)
190 bs1=int((charsize1+7)/8), bs2=int((charsize2+7)/8)
192 One submapping ends when <mapend> entry occurs.
194 <key_val_seq>:
196 0x00 size=0|1|2|4 n, number of sequential characters
197 size bs1 key1
198 +bs1 bs2 value1
199 +bs2 bs1 key2
200 +bs1 bs2 value2
201 ...
202 key_val_seq ends, if either file ends (n = infinite mode) or n
204 pairs are read.
205 <key_seq>:
207 0x00 byte n, number of sequential characters
208 0x01 bs1 key_start, first character of sequence
209 1+bs1 @(<extended> || <val_seq>)
210 A key sequence starts with a byte count telling how long the
212 sequence is. It is followed by the key start code. After this comes
213 a list of value sequences. The list of value sequences ends, if
214 sum(m) equals n.
215 <val_seq>:
217 0x00 byte m, number of sequential characters
218 0x01 bs2 val_start, first character of sequence
219 <extended>:
221 0x00 byte 0
222 0x01 byte ftype
223 0x02 byte fsize, size of following structure
224 0x03 fsize bytes something
225 For future extensions or private use one can insert here 1..255
227 byte long streams. ftype can have values 30..255, values 0..29 are
228 reserved. Modi are not fully defined now and could change. They
229 will be explained later.
230
232 - Something clever, when a character has no translation.
233 - Direct charset -> charset mapping.
235 - Better performance.
237 - Support for mappings according to RFC 1345.
239
241 - File "REGISTRY" and binary mappings in directory "Unicode/Map" of
242 your perl library path
243 - recode(1), map(1), mkmapfile(1), Unicode::Map(3), Unicode::Map8(3),
245 Unicode::String(3), Unicode::CharName(3), mirrorMappings(1)
246 - RFC 1345
248 - Mappings at Unicode consortium ftp://ftp.unicode.org/MAPPINGS/
250 - Registrated Internet character sets ftp://dkuug.dk/i18n/charmaps/
252 - 2do: more references
254
256 Martin Schwartz <martin@nacho.de>
257
259 Hey! The above document had some coding errors, which are explained
260 below:
261 Around line 1112:
263 You can't have =items (as at line 1118) unless the first thing
264 after the =over is an =item
265perl v5.32.0 2020-07-28 Map(3)