1PRECONV(1) General Commands Manual PRECONV(1)
2
6 preconv - convert encoding of input files to something GNU troff under‐
7 stands
8
10 preconv [-dr] [-D default_encoding] [-e encoding] [file ...]
11 preconv -h
12 preconv --help
13 preconv -v
15 preconv --version
16
18 preconv reads files and converts its encoding(s) to a form GNU troff(1)
19 can process, sending the data to standard output. Currently, this
20 means ASCII characters and ‘\[uXXXX]’ entities, where ‘XXXX’ is a hexa‐
21 decimal number with four to six digits, representing a Unicode input
22 code. Normally, preconv should be invoked with the -k and -K options
23 of groff.
24
26 Whitespace is permitted between a command-line option and its argument.
27 -d Emit debugging messages to standard error (mainly the used en‐
29 coding).
30 -Dencoding
32 Specify default encoding if everything fails (see below).
33 -eencoding
35 Specify input encoding explicitly, overriding all other methods.
36 This corresponds to groff's -Kencoding option. Without this
37 switch, preconv uses the algorithm described below to select the
38 input encoding.
39 --help
41 -h Print a help message and exit.
42 -r Do not add .lf requests.
44 --version
46 -v Print the version number and exit.
47
49 preconv tries to find the input encoding with the following algorithm.
50 1. If the input encoding has been explicitly specified with option
52 -e, use it.
53 2. Otherwise, check whether the input starts with a Byte Order Mark
55 (BOM, see below). If found, use it.
56 3. Otherwise, check whether there is a known coding tag (see below)
58 in either the first or second input line. If found, use it.
59 4 Finally, if the uchardet library (an encoding detector library
61 available on most major distributions) is available on the sys‐
62 tem, use it to try to detect the encoding of the file.
63 5. If everything fails, use a default encoding as given with option
65 -D, by the current locale, or ‘latin1’ if the locale is set to
66 ‘C’, ‘POSIX’, or empty (in that order).
67 Note that the groff program supports a GROFF_ENCODING environment vari‐
69 able which is eventually expanded to option -k.
70 Byte Order Mark
72 The Unicode Standard defines character U+FEFF as the Byte Order Mark
73 (BOM). On the other hand, value U+FFFE is guaranteed not be a Unicode
74 character at all. This allows detection of the byte order within the
75 data stream (either big-endian or little-endian), and the MIME encod‐
76 ings ‘UTF-16’ and ‘UTF-32’ mandate that the data stream starts with
77 U+FEFF. Similarly, the data stream encoded as ‘UTF-8’ might start with
78 a BOM (to ease the conversion from and to UTF-16 and UTF-32). In all
79 cases, the byte order mark is not part of the data but part of the en‐
80 coding protocol; in other words, preconv's output doesn't contain it.
81 Note that U+FEFF not at the start of the input data actually is emit‐
83 ted; it has then the meaning of a ‘zero width no-break space’ character
84 – something not needed normally in groff.
85 Coding Tags
87 Editors which support more than a single character encoding need tags
88 within the input files to mark the file's encoding. While it is possi‐
89 ble to guess the right input encoding with the help of heuristic algo‐
90 rithms for data which represents a greater amount of a natural lan‐
91 guage, it is still just a guess. Additionally, all algorithms fail
92 easily for input which is either too short or doesn't represent a natu‐
93 ral language.
94 For these reasons, preconv supports the coding tag convention (with
96 some restrictions) as used by GNU Emacs and XEmacs (and probably other
97 programs too).
98 Coding tags in GNU Emacs and XEmacs are stored in so-called File Vari‐
100 ables. preconv recognizes the following syntax form which must be put
101 into a troff comment in the first or second line.
102 -*- tag1: value1; tag2: value2; ... -*-
104 The only relevant tag for preconv is ‘coding’ which can take the values
106 listed below. Here an example line which tells Emacs to edit a file in
107 troff mode, and to use latin2 as its encoding.
108 .\" -*- mode: troff; coding: latin-2 -*-
110 The following list gives all MIME coding tags (either lowercase or up‐
112 percase) supported by preconv; this list is hard-coded in the source.
113 big5, cp1047, euc-jp, euc-kr, gb2312, iso-8859-1, iso-8859-2,
115 iso-8859-5, iso-8859-7, iso-8859-9, iso-8859-13, iso-8859-15,
116 koi8-r, us-ascii, utf-8, utf-16, utf-16be, utf-16le
117 In addition, the following hard-coded list of other tags is recognized
119 which eventually map to values from the list above.
120 ascii, chinese-big5, chinese-euc, chinese-iso-8bit, cn-big5,
122 cn-gb, cn-gb-2312, cp878, csascii, csisolatin1,
123 cyrillic-iso-8bit, cyrillic-koi8, euc-china, euc-cn, euc-japan,
124 euc-japan-1990, euc-korea, greek-iso-8bit, iso-10646/utf8,
125 iso-10646/utf-8, iso-latin-1, iso-latin-2, iso-latin-5,
126 iso-latin-7, iso-latin-9, japanese-euc, japanese-iso-8bit, jis8,
127 koi8, korean-euc, korean-iso-8bit, latin-0, latin1, latin-1,
128 latin-2, latin-5, latin-7, latin-9, mule-utf-8, mule-utf-16,
129 mule-utf-16be, mule-utf-16-be, mule-utf-16be-with-signature,
130 mule-utf-16le, mule-utf-16-le, mule-utf-16le-with-signature,
131 utf8, utf-16-be, utf-16-be-with-signature,
132 utf-16be-with-signature, utf-16-le, utf-16-le-with-signature,
133 utf-16le-with-signature
134 Those tags are taken from GNU Emacs and XEmacs, together with some
136 aliases. Trailing ‘-dos’, ‘-unix’, and ‘-mac’ suffixes of coding tags
137 (which give the end-of-line convention used in the file) are stripped
138 off before the comparison with the above tags happens.
139 Iconv Issues
141 preconv by itself only supports three encodings: latin-1, cp1047, and
142 UTF-8; all other encodings are passed to the iconv library functions.
143 At compile time it is searched and checked for a valid iconv implemen‐
144 tation; a call to ‘preconv --version’ shows whether iconv is used.
145
147 preconv doesn't support local variable lists yet. This is a different
148 syntax form to specify local variables at the end of a file.
149
151 groff(1)
152 the GNU Emacs and XEmacs info pages
153groff 1.22.4 22 July 2021 PRECONV(1)