1UNIQ(1P)                   POSIX Programmer's Manual                  UNIQ(1P)
2
3
4

PROLOG

6       This  manual  page is part of the POSIX Programmer's Manual.  The Linux
7       implementation of this interface may differ (consult the  corresponding
8       Linux  manual page for details of Linux behavior), or the interface may
9       not be implemented on Linux.
10

NAME

12       uniq — report or filter out repeated lines in a file
13

SYNOPSIS

15       uniq [-c|-d|-u] [-f fields] [-s char] [input_file [output_file]]
16

DESCRIPTION

18       The uniq utility shall read an input file comparing adjacent lines, and
19       write  one  copy  of each input line on the output. The second and suc‐
20       ceeding copies of repeated adjacent input lines shall not  be  written.
21       The  trailing <newline> of each line in the input shall be ignored when
22       doing comparisons.
23
24       Repeated lines in the input shall not be detected if they are not adja‐
25       cent.
26

OPTIONS

28       The  uniq  utility  shall  conform  to  the  Base Definitions volume of
29       POSIX.1‐2017, Section 12.2, Utility Syntax Guidelines, except that  '+'
30       may be recognized as an option delimiter as well as '-'.
31
32       The following options shall be supported:
33
34       -c        Precede  each output line with a count of the number of times
35                 the line occurred in the input.
36
37       -d        Suppress the writing of lines that are not  repeated  in  the
38                 input.
39
40       -f fields Ignore  the first fields fields on each input line when doing
41                 comparisons, where fields is a positive  decimal  integer.  A
42                 field  is  the  maximal  string  matched by the basic regular
43                 expression:
44
45
46                     [[:blank:]]*[^[:blank:]]*
47
48                 If the fields  option-argument  specifies  more  fields  than
49                 appear on an input line, a null string shall be used for com‐
50                 parison.
51
52       -s chars  Ignore the first chars  characters  when  doing  comparisons,
53                 where chars shall be a positive decimal integer. If specified
54                 in conjunction with the -f option, the first chars characters
55                 after  the first fields fields shall be ignored. If the chars
56                 option-argument specifies more characters than remain  on  an
57                 input line, a null string shall be used for comparison.
58
59       -u        Suppress the writing of lines that are repeated in the input.
60

OPERANDS

62       The following operands shall be supported:
63
64       input_file
65                 A  pathname  of  the input file. If the input_file operand is
66                 not specified, or if the  input_file  is  '-',  the  standard
67                 input shall be used.
68
69       output_file
70                 A  pathname of the output file. If the output_file operand is
71                 not specified, the standard output shall be used. The results
72                 are  unspecified if the file named by output_file is the file
73                 named by input_file.
74

STDIN

76       The standard input shall be used only if no input_file operand is spec‐
77       ified or if input_file is '-'.  See the INPUT FILES section.
78

INPUT FILES

80       The input file shall be a text file.
81

ENVIRONMENT VARIABLES

83       The following environment variables shall affect the execution of uniq:
84
85       LANG      Provide  a  default  value for the internationalization vari‐
86                 ables that are unset or null. (See the Base Definitions  vol‐
87                 ume  of POSIX.1‐2017, Section 8.2, Internationalization Vari‐
88                 ables for the precedence  of  internationalization  variables
89                 used to determine the values of locale categories.)
90
91       LC_ALL    If  set  to  a non-empty string value, override the values of
92                 all the other internationalization variables.
93
94       LC_CTYPE  Determine the locale for the interpretation of  sequences  of
95                 bytes of text data as characters (for example, single-byte as
96                 opposed to  multi-byte  characters  in  arguments  and  input
97                 files)  and which characters constitute a <blank> in the cur‐
98                 rent locale.
99
100       LC_MESSAGES
101                 Determine the locale that should be used to affect the format
102                 and  contents  of  diagnostic  messages  written  to standard
103                 error.
104
105       NLSPATH   Determine the location of message catalogs for the processing
106                 of LC_MESSAGES.
107

ASYNCHRONOUS EVENTS

109       Default.
110

STDOUT

112       The  standard  output shall be used if no output_file operand is speci‐
113       fied, and shall be used if the  output_file  operand  is  '-'  and  the
114       implementation  treats  the  '-' as meaning standard output. Otherwise,
115       the standard output shall not be used.  See the OUTPUT FILES section.
116

STDERR

118       The standard error shall be used only for diagnostic messages.
119

OUTPUT FILES

121       If the -c option is specified, the output file shall be empty  or  each
122       line shall be of the form:
123
124
125           "%d %s", <number of duplicates>, <line>
126
127       otherwise,  the output file shall be empty or each line shall be of the
128       form:
129
130
131           "%s", <line>
132

EXTENDED DESCRIPTION

134       None.
135

EXIT STATUS

137       The following exit values shall be returned:
138
139        0    The utility executed successfully.
140
141       >0    An error occurred.
142

CONSEQUENCES OF ERRORS

144       Default.
145
146       The following sections are informative.
147

APPLICATION USAGE

149       If the collating sequence of the current locale has a total ordering of
150       all characters, the sort utility can be used to cause repeated lines to
151       be adjacent in the input file. If the collating sequence does not  have
152       a  total  ordering  of all characters, the sort utility should still do
153       this but it might not. To ensure that all duplicate  lines  are  elimi‐
154       nated,  and  have the output sorted according the collating sequence of
155       the current locale, applications should use:
156
157
158           LC_ALL=C sort -u | sort
159
160       instead of:
161
162
163           sort | uniq
164
165       To remove duplicate lines based on whether they collate equally instead
166       of whether they are identical, applications should use:
167
168
169           sort -u
170
171       instead of:
172
173
174           sort | uniq
175
176       When using uniq to process pathnames, it is recommended that LC_ALL, or
177       at least LC_CTYPE and LC_COLLATE, are set to POSIX or C in the environ‐
178       ment, since pathnames can contain byte sequences that do not form valid
179       characters in some locales, in which case the utility's behavior  would
180       be  undefined.  In  the  POSIX  locale each byte is a valid single-byte
181       character, and therefore this problem is avoided.
182

EXAMPLES

184       The following input file data (but flushed left) was used  for  a  test
185       series on uniq:
186
187
188           #01 foo0 bar0 foo1 bar1
189           #02 bar0 foo1 bar1 foo1
190           #03 foo0 bar0 foo1 bar1
191           #04
192           #05 foo0 bar0 foo1 bar1
193           #06 foo0 bar0 foo1 bar1
194           #07 bar0 foo1 bar1 foo0
195
196       What  follows  is a series of test invocations of the uniq utility that
197       use a mixture of uniq options against the input file data. These  tests
198       verify  the meaning of adjacent.  The uniq utility views the input data
199       as a sequence of strings  delimited  by  '\n'.   Accordingly,  for  the
200       fieldsth  member  of  the  sequence, uniq interprets unique or repeated
201       adjacent lines strictly relative to the fields+1th member.
202
203        1. This first example tests the line counting option,  comparing  each
204           line of the input file data starting from the second field:
205
206
207               uniq -c -f 1 uniq_0I.t
208                   1 #01 foo0 bar0 foo1 bar1
209                   1 #02 bar0 foo1 bar1 foo1
210                   1 #03 foo0 bar0 foo1 bar1
211                   1 #04
212                   2 #05 foo0 bar0 foo1 bar1
213                   1 #07 bar0 foo1 bar1 foo0
214
215           The  number '2', prefixing the fifth line of output, signifies that
216           the uniq utility detected a pair of repeated lines. Given the input
217           data,  this can only be true when uniq is run using the -f 1 option
218           (which shall cause uniq to ignore the first  field  on  each  input
219           line).
220
221        2. The  second example tests the option to suppress unique lines, com‐
222           paring each line of the input file data starting  from  the  second
223           field:
224
225
226               uniq -d -f 1 uniq_0I.t
227               #05 foo0 bar0 foo1 bar1
228
229        3. This  test  suppresses  repeated  lines, comparing each line of the
230           input file data starting from the second field:
231
232
233               uniq -u -f 1 uniq_0I.t
234               #01 foo0 bar0 foo1 bar1
235               #02 bar0 foo1 bar1 foo1
236               #03 foo0 bar0 foo1 bar1
237               #04
238               #07 bar0 foo1 bar1 foo0
239
240        4. This suppresses unique lines, comparing each line of the input file
241           data starting from the third character:
242
243
244               uniq -d -s 2 uniq_0I.t
245
246           In  the  last example, the uniq utility found no input matching the
247           above criteria.
248

RATIONALE

250       Some historical implementations have limited lines to be 1080 bytes  in
251       length, which does not meet the implied {LINE_MAX} limit.
252
253       Earlier  versions  of  this  standard  allowed  the -number and +number
254       options. These options are no longer specified by POSIX.1‐2008 but  may
255       be present in some implementations.
256

FUTURE DIRECTIONS

258       None.
259

SEE ALSO

261       comm, sort
262
263       The  Base  Definitions  volume  of POSIX.1‐2017, Chapter 8, Environment
264       Variables, Section 12.2, Utility Syntax Guidelines
265
267       Portions of this text are reprinted and reproduced in  electronic  form
268       from  IEEE Std 1003.1-2017, Standard for Information Technology -- Por‐
269       table Operating System Interface (POSIX), The Open Group Base  Specifi‐
270       cations  Issue  7, 2018 Edition, Copyright (C) 2018 by the Institute of
271       Electrical and Electronics Engineers, Inc and The Open Group.   In  the
272       event of any discrepancy between this version and the original IEEE and
273       The Open Group Standard, the original IEEE and The Open Group  Standard
274       is  the  referee document. The original Standard can be obtained online
275       at http://www.opengroup.org/unix/online.html .
276
277       Any typographical or formatting errors that appear  in  this  page  are
278       most likely to have been introduced during the conversion of the source
279       files to man page format. To report such errors,  see  https://www.ker
280       nel.org/doc/man-pages/reporting_bugs.html .
281
282
283
284IEEE/The Open Group                  2017                             UNIQ(1P)
Impressum