1UNIQ(1P) POSIX Programmer's Manual UNIQ(1P)
2
6 This manual page is part of the POSIX Programmer's Manual. The Linux
7 implementation of this interface may differ (consult the corresponding
8 Linux manual page for details of Linux behavior), or the interface may
9 not be implemented on Linux.
10
12 uniq — report or filter out repeated lines in a file
13
15 uniq [-c|-d|-u] [-f fields] [-s char] [input_file [output_file]]
16
18 The uniq utility shall read an input file comparing adjacent lines, and
19 write one copy of each input line on the output. The second and suc‐
20 ceeding copies of repeated adjacent input lines shall not be written.
21 The trailing <newline> of each line in the input shall be ignored when
22 doing comparisons.
23 Repeated lines in the input shall not be detected if they are not adja‐
25 cent.
26
28 The uniq utility shall conform to the Base Definitions volume of
29 POSIX.1‐2017, Section 12.2, Utility Syntax Guidelines, except that '+'
30 may be recognized as an option delimiter as well as '-'.
31 The following options shall be supported:
33 -c Precede each output line with a count of the number of times
35 the line occurred in the input.
36 -d Suppress the writing of lines that are not repeated in the
38 input.
39 -f fields Ignore the first fields fields on each input line when doing
41 comparisons, where fields is a positive decimal integer. A
42 field is the maximal string matched by the basic regular
43 expression:
44 [[:blank:]]*[^[:blank:]]*
47 If the fields option-argument specifies more fields than
49 appear on an input line, a null string shall be used for com‐
50 parison.
51 -s chars Ignore the first chars characters when doing comparisons,
53 where chars shall be a positive decimal integer. If specified
54 in conjunction with the -f option, the first chars characters
55 after the first fields fields shall be ignored. If the chars
56 option-argument specifies more characters than remain on an
57 input line, a null string shall be used for comparison.
58 -u Suppress the writing of lines that are repeated in the input.
60
62 The following operands shall be supported:
63 input_file
65 A pathname of the input file. If the input_file operand is
66 not specified, or if the input_file is '-', the standard
67 input shall be used.
68 output_file
70 A pathname of the output file. If the output_file operand is
71 not specified, the standard output shall be used. The results
72 are unspecified if the file named by output_file is the file
73 named by input_file.
74
76 The standard input shall be used only if no input_file operand is spec‐
77 ified or if input_file is '-'. See the INPUT FILES section.
78
80 The input file shall be a text file.
81
83 The following environment variables shall affect the execution of uniq:
84 LANG Provide a default value for the internationalization vari‐
86 ables that are unset or null. (See the Base Definitions vol‐
87 ume of POSIX.1‐2017, Section 8.2, Internationalization Vari‐
88 ables for the precedence of internationalization variables
89 used to determine the values of locale categories.)
90 LC_ALL If set to a non-empty string value, override the values of
92 all the other internationalization variables.
93 LC_CTYPE Determine the locale for the interpretation of sequences of
95 bytes of text data as characters (for example, single-byte as
96 opposed to multi-byte characters in arguments and input
97 files) and which characters constitute a <blank> in the cur‐
98 rent locale.
99 LC_MESSAGES
101 Determine the locale that should be used to affect the format
102 and contents of diagnostic messages written to standard
103 error.
104 NLSPATH Determine the location of message catalogs for the processing
106 of LC_MESSAGES.
107
109 Default.
110
112 The standard output shall be used if no output_file operand is speci‐
113 fied, and shall be used if the output_file operand is '-' and the
114 implementation treats the '-' as meaning standard output. Otherwise,
115 the standard output shall not be used. See the OUTPUT FILES section.
116
118 The standard error shall be used only for diagnostic messages.
119
121 If the -c option is specified, the output file shall be empty or each
122 line shall be of the form:
123 "%d %s", <number of duplicates>, <line>
126 otherwise, the output file shall be empty or each line shall be of the
128 form:
129 "%s", <line>
132
134 None.
135
137 The following exit values shall be returned:
138 0 The utility executed successfully.
140 >0 An error occurred.
142
144 Default.
145 The following sections are informative.
147
149 If the collating sequence of the current locale has a total ordering of
150 all characters, the sort utility can be used to cause repeated lines to
151 be adjacent in the input file. If the collating sequence does not have
152 a total ordering of all characters, the sort utility should still do
153 this but it might not. To ensure that all duplicate lines are elimi‐
154 nated, and have the output sorted according the collating sequence of
155 the current locale, applications should use:
156 LC_ALL=C sort -u | sort
159 instead of:
161 sort | uniq
164 To remove duplicate lines based on whether they collate equally instead
166 of whether they are identical, applications should use:
167 sort -u
170 instead of:
172 sort | uniq
175 When using uniq to process pathnames, it is recommended that LC_ALL, or
177 at least LC_CTYPE and LC_COLLATE, are set to POSIX or C in the environ‐
178 ment, since pathnames can contain byte sequences that do not form valid
179 characters in some locales, in which case the utility's behavior would
180 be undefined. In the POSIX locale each byte is a valid single-byte
181 character, and therefore this problem is avoided.
182
184 The following input file data (but flushed left) was used for a test
185 series on uniq:
186 #01 foo0 bar0 foo1 bar1
189 #02 bar0 foo1 bar1 foo1
190 #03 foo0 bar0 foo1 bar1
191 #04
192 #05 foo0 bar0 foo1 bar1
193 #06 foo0 bar0 foo1 bar1
194 #07 bar0 foo1 bar1 foo0
195 What follows is a series of test invocations of the uniq utility that
197 use a mixture of uniq options against the input file data. These tests
198 verify the meaning of adjacent. The uniq utility views the input data
199 as a sequence of strings delimited by '\n'. Accordingly, for the
200 fieldsth member of the sequence, uniq interprets unique or repeated
201 adjacent lines strictly relative to the fields+1th member.
202 1. This first example tests the line counting option, comparing each
204 line of the input file data starting from the second field:
205 uniq -c -f 1 uniq_0I.t
208 1 #01 foo0 bar0 foo1 bar1
209 1 #02 bar0 foo1 bar1 foo1
210 1 #03 foo0 bar0 foo1 bar1
211 1 #04
212 2 #05 foo0 bar0 foo1 bar1
213 1 #07 bar0 foo1 bar1 foo0
214 The number '2', prefixing the fifth line of output, signifies that
216 the uniq utility detected a pair of repeated lines. Given the input
217 data, this can only be true when uniq is run using the -f 1 option
218 (which shall cause uniq to ignore the first field on each input
219 line).
220 2. The second example tests the option to suppress unique lines, com‐
222 paring each line of the input file data starting from the second
223 field:
224 uniq -d -f 1 uniq_0I.t
227 #05 foo0 bar0 foo1 bar1
228 3. This test suppresses repeated lines, comparing each line of the
230 input file data starting from the second field:
231 uniq -u -f 1 uniq_0I.t
234 #01 foo0 bar0 foo1 bar1
235 #02 bar0 foo1 bar1 foo1
236 #03 foo0 bar0 foo1 bar1
237 #04
238 #07 bar0 foo1 bar1 foo0
239 4. This suppresses unique lines, comparing each line of the input file
241 data starting from the third character:
242 uniq -d -s 2 uniq_0I.t
245 In the last example, the uniq utility found no input matching the
247 above criteria.
248
250 Some historical implementations have limited lines to be 1080 bytes in
251 length, which does not meet the implied {LINE_MAX} limit.
252 Earlier versions of this standard allowed the -number and +number
254 options. These options are no longer specified by POSIX.1‐2008 but may
255 be present in some implementations.
256
258 None.
259
261 comm, sort
262 The Base Definitions volume of POSIX.1‐2017, Chapter 8, Environment
264 Variables, Section 12.2, Utility Syntax Guidelines
265
267 Portions of this text are reprinted and reproduced in electronic form
268 from IEEE Std 1003.1-2017, Standard for Information Technology -- Por‐
269 table Operating System Interface (POSIX), The Open Group Base Specifi‐
270 cations Issue 7, 2018 Edition, Copyright (C) 2018 by the Institute of
271 Electrical and Electronics Engineers, Inc and The Open Group. In the
272 event of any discrepancy between this version and the original IEEE and
273 The Open Group Standard, the original IEEE and The Open Group Standard
274 is the referee document. The original Standard can be obtained online
275 at http://www.opengroup.org/unix/online.html .
276 Any typographical or formatting errors that appear in this page are
278 most likely to have been introduced during the conversion of the source
279 files to man page format. To report such errors, see https://www.ker‐
280 nel.org/doc/man-pages/reporting_bugs.html .
281IEEE/The Open Group 2017 UNIQ(1P)