1DUFF(1)                   BSD General Commands Manual                  DUFF(1)
2

NAME

4     duff — duplicate file finder
5

SYNOPSIS

7     duff [-0HLPaeqprtz] [-d function] [-f format] [-l limit] [file ...]
8     duff [-h]
9     duff [-v]
10

DESCRIPTION

12     The duff utility reports clusters of duplicates in the specified files
13     and/or directories.  In the default mode, duff prints a customizable
14     header, followed by the names of all the files in the cluster.  In excess
15     mode, duff does not print a header, but instead for each cluster prints
16     the names of all but the first of the files it includes.
17
18     If no files are specified as arguments, duff reads file names from stdin.
19
20     Note that as of version 0.4, duff ignores symbolic links to files, as
21     that behavior was conceptually broken.  Therefore, the -H, -L and -P
22     options now apply only to directories.
23
24     The following options are available:
25
26     -0      If reading file names from stdin, assume they are null-termi‐
27             nated, instead of separated by newlines.  Also, when printing
28             file names and cluster headers, terminate them with null charac‐
29             ters instead of newlines.
30
31             This is useful for file names containing whitespace or other non-
32             standard characters.
33
34     -H      Follow symbolic links listed on the command line.  This overrides
35             any previous -L or -P option.  Note that this only applies to
36             directories, as symbolic links to files are never followed.
37
38     -L      Follow all symbolic links.  This overrides any previous -H or -P
39             option.  Note that this only applies to directories, as symbolic
40             links to files are never followed.
41
42     -P      Don't follow any symbolic links.  This overrides any previous -H
43             or -L option.  This is the default.  Note that this only applies
44             to directories, as symbolic links to files are never followed.
45
46     -a      Include hidden files and directories when searching recursively.
47
48     -d function
49             The message digest function to use.  The supported functions are
50             sha1, sha256, sha384 and sha512.  The default is sha1.
51
52     -e      Excess mode.  List all but one file from each cluster of dupli‐
53             cates.  Also suppresses output of the cluster header.  This is
54             useful when you want to automate removal of duplicate files and
55             don't care which duplicates are removed.
56
57     -f format
58             Set the format of the cluster header.  If the header is set to
59             the empty string, no header line is printed.
60
61             The following escape sequences are available:
62
63                 %n      The number of files in the cluster.
64
65                 %c      A legacy synonym for %d, for compatibility reasons.
66
67                 %d      The message digest of files in the cluster.  This may
68                         not be combined with -t as no digest is calculated.
69
70                 %i      The one-based index of the file cluster.
71
72                 %s      The size, in bytes, of a file in the cluster.
73
74                 %%      A ‘%’ character.
75
76             The default format string when using -t is:
77
78                   %n files in cluster %i (%s bytes)
79
80             The default format string for other modes is:
81
82                   %n files in cluster %i (%s bytes, digest %d)
83
84     -h      Display help information and exit.
85
86     -l limit
87             The minimum size of files to be sampled.  If the size of files in
88             a cluster is equal or greater than the specified limit, duff will
89             sample and compare a few bytes from the start of each file before
90             calculating a full digest.  This is stricly an optimization and
91             does not affect which files are considered by duff.  The default
92             limit is zero bytes, i.e. to use sampling on all files.
93
94     -q      Quiet mode.  Suppress warnings and error messages.
95
96     -p      Physical mode.  Make duff consider physical files instead of hard
97             links.  If specified, multiple hard links to the same physical
98             file will not be reported as duplicates.
99
100     -r      Recursively search into all specified directories.
101
102     -t      Thorough mode.  Distrust digests as a guarantee for equality.  In
103             thorough mode, duff compares files byte by byte when their sizes
104             match.
105
106     -v      Display version information and exit.
107
108     -z      Do not consider empty files to be equal.  This option prevents
109             empty files from being reported as duplicates.
110

EXAMPLES

112     The command:
113           duff -r foo/
114
115     lists all duplicate files in the directory foo and its subdirectories.
116
117     The command:
118           duff -e0 * | xargs -0 rm
119
120     removes all duplicate files in the current directory.  Note that you have
121     no control over which files in each cluster that are selected by -e
122     (excess mode).  Use with care.
123
124     The command:
125           find . -name '*.h' -type f | duff
126
127     lists all duplicate header files in the current directory and its subdi‐
128     rectories.
129
130     The command:
131           find . -name '*.h' -type f -print0 | duff -0 | xargs -0 -n1 echo
132
133     lists all duplicate header files in the current directory and its subdi‐
134     rectories, correctly handling file names containing whitespace.  Note the
135     use of xargs and echo to remove the null separators again before listing.
136

DIAGNOSTICS

138     The duff utility exits 0 on success, and >0 if an error occurs.
139

SEE ALSO

141     find(1), xargs(1)
142

AUTHORS

144     Camilla Berglund <elmindreda@elmindreda.org>
145

BUGS

147     duff doesn't check whether the same file has been specified twice on the
148     command line.  This will lead it to report files listed multiple times as
149     duplicates when not using -p (physical mode).  Note that this problem
150     only affects files, not directories.
151
152     duff no longer (as of version 0.4) reports symbolic links to files as
153     duplicates, as they're by definition always duplicates.  This may break
154     scripts relying on the previous behavior.
155
156     If the underlying files are modified while duff is running, all bets are
157     off.  This is not really a bug, but it can still bite you.
158
159BSD                            January 18, 2012                            BSD
Impressum