1FileList(3) User Contributed Perl Documentation FileList(3)
2
6 File::RsyncP::FileList - Perl interface to rsync file list encoding and
7 decoding.
8
10 use File::RsyncP::FileList;
11 $fileList = File::RsyncP::FileList->new({
13 preserve_uid => 1,
14 preserve_gid => 1,
15 preserve_links => 1,
16 preserve_devices => 1,
17 preserve_hard_links => 0,
18 always_checksum => 0,
19 remote_version => 26,
20 });
21 # decoding an incoming file list
23 while ( !$fileList->decodeDone && !$fileList->fatalError ) {
24 $data .= readMoreDataFromRemoteRsync();
25 $bytesDone = $fileList->decode($data);
26 $data = substr($data, $bytesDone) if ( $bytesDone > 0 );
27 }
28 $fileList->clean;
29 # create (encode) a file list
31 $fileList->encode({
32 name => $filePath,
33 dev => $stat[0],
34 inode => $stat[1],
35 mode => $stat[2],
36 uid => $stat[4],
37 gid => $stat[5],
38 rdev => $stat[6],
39 mtime => $stat[9],
40 });
41 $data = $fileList->encodeData;
42 # get file information, for file number 5:
44 $fileInfo = $fileList->get(5);
45 # utility functions
47 $numberOfFiles = $fileList->count;
48 $gotFatalError = $fileList->fatalError;
49
51 The File::RsyncP::FileList module is used to encode and decode file
52 lists in using the same format at Rsync.
53 The sender side of Rsync creates a list of all the files the are going
55 to be sent. This list is sent in a compact format to the receiver
56 side. Each side then sorts the list and removes duplicate entries.
57 From this point on, all files are referred to by their integer index
58 into the sorted file list.
59 A new file list object is created by calling
61 File::RsyncP::FileList->new. An object can be used to decode or encode
62 a file list. There is no mechanism to reset the state of a file list:
63 you should create a new object each time you need to do a new decode or
64 encode.
65 The new() function takes a hashref of options, which correspond to
67 various rsync command-line switches. These must exactly match the
68 arguments to the remote rsync, otherwise the file list format will not
69 be compatible and decoding will fail.
70 $fileList = File::RsyncP::FileList->new({
72 preserve_uid => 1, # --owner
73 preserve_gid => 1, # --group
74 preserve_links => 1, # --links
75 preserve_devices => 1, # --devices
76 preserve_hard_links => 0, # --hard-links
77 always_checksum => 0, # --checksum
78 remote_version => 26, # remote protocol version
79 });
80 Decoding
82 The decoding functions take a stream of bytes from the remote rsync and
83 convert them into an internal data structure. Rather than store the
84 file list as a native perl list of hashes (which occupies too much
85 memory for large file lists), the same internal data structure as rsync
86 is used. Individual file list entries can be returned with the get()
87 function.
88 File list data read from the remote rsync should be passed to the
90 decode() function. The data may be read and processed in arbitrary
91 sized chunks. The decode() function returns how many bytes were
92 actually processed. It is the caller's responsbility to remove that
93 number of bytes from the input argument, preserving the remaining bytes
94 for the next call to decode(). The decodeDone() function returns true
95 when the file list is complete. The fatalError() function returns true
96 if there was a non-recoverable error while decoding.
97 The clean() function needs to be called after the file list decode is
99 complete. The clean() function sorts the file list and removes
100 repeated entries. Skipping this step will produce unexpected results:
101 since files are referred to using integers, each side will refer to
102 different files is the file lists are not sorted and purged in exactly
103 the same manner.
104 A typical decode loop looks like:
106 while ( !$fileList->decodeDone && !$fileList->fatalError ) {
108 $data .= readMoreDataFromRemoteRsync();
109 $bytesDone = $fileList->decode($data);
110 $data = substr($data, $bytesDone) if ( $bytesDone > 0 );
111 }
112 $fileList->clean;
113 After clean() is called, the number of files in the file list can be
115 found by calling count(). Files can be fetched by calling the get()
116 function, with an index from 0 to count()-1:
117 $fileInfo = $fileList->get(5);
119 The get() function returns a hashref with various entries:
121 name path name of the file (relative to rsync dir):
123 equal to dirname/basename
124 basename file name, without directory
125 dirname directory where file resides
126 sum file MD4 checksum (only present if --checksum specified)
127 uid file user id
128 gid file group id
129 mode file mode
130 mtime file modification time
131 size file length
132 dev device number on which file resides
133 inode file inode
134 link link contents if the file is a sym link
135 rdev major/minor device number if file is char/block special
136 Various fields will only have valid values if the corresponding options
138 are set (eg: uid if preserve_uid is set, dev and inode if
139 preserve_hard_links is set etc).
140 For example, to dump out each of hash you could do this:
142 use Data::Dumper;
144 my $count = $fileList->count;
145 for ( my $i = 0 ; $i < $count ; $i++ ) {
146 print("File $i is:\n");
147 print Dumper($fileList->get($i));
148 }
149 Encoding
151 The encode() function is used to build a file list in preparation for
152 encoding and sending a file list to a remote rsync. The encode()
153 function takes a hashref argument with the parameters for one file. It
154 should be called once for each file. The parameter names are the same
155 as those returned by get().
156 In this example the matching stat() values are shown:
158 $fileList->encode({
160 name => $filePath,
161 dev => $stat[0],
162 inode => $stat[1],
163 mode => $stat[2],
164 uid => $stat[4],
165 gid => $stat[5],
166 rdev => $stat[6],
167 size => $stat[7],
168 mtime => $stat[9],
169 });
170 It is not necessary to specify basename and dirname; these are
172 extracted from name. You only need to specify the parameters that
173 match the options given to new(). You can also specify sum and link as
174 necessary.
175 To compute the encoded file list data the encodeData() function should
177 be called. It can be called every time encode() is called, or once at
178 the end of all the encode() calls. It returns the encoded data that
179 should be sent to the remote rsync:
180 $data = $fileList->encodeData;
182 It is recommended that encodeData() be called frequently to avoid the
184 need to allocate large internal buffers to hold the entire encoded file
185 list. Since encodeData() does not know when the last file has been
186 encoded, it is the caller's responsbility to add the final null byte
187 (eg: pack("C", 0)) to the data to indicate the end of the file list
188 data.
189 After all the file list entries are processed you should call clean():
191 $fileList->clean;
193 This ensures that each side (sender/receiver) has identical sorted file
195 lists.
196 Utility functions
198 The count() function returns the total number of files in the internal
199 file list (either decoded or encoded).
200 The fatalError() function returns true if a fatal error has occured
202 during file decoding. It should be called in the decode loop to make
203 sure no error has occured.
204
206 File::RsyncP::FileList was written by Craig Barratt
207 <cbarratt@users.sourceforge.net> based on rsync 2.5.5.
208 Rsync was written by Andrew Tridgell <tridge@samba.org> and Paul
210 Mackerras. It is available under a GPL license. See
211 http://rsync.samba.org
212
214 This program is free software; you can redistribute it and/or modify it
215 under the terms of the GNU General Public License as published by the
216 Free Software Foundation; either version 2 of the License, or (at your
217 option) any later version.
218 This program is distributed in the hope that it will be useful, but
220 WITHOUT ANY WARRANTY; without even the implied warranty of
221 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
222 General Public License for more details.
223 You should have received a copy of the GNU General Public License in
225 the LICENSE file along with this program; if not, write to the Free
226 Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
227 02111-1307 USA.
228
230 See <http://perlrsync.sourceforge.net> for File::RsyncP's SourceForge
231 home page.
232 See File::RsyncP and File::RsyncP::FileIO for more precise examples of
234 using File::RsyncP::FileList.
235 Also see BackupPC's lib/BackupPC/Xfer/RsyncFileIO.pm for other
237 examples.
238perl v5.32.0 2020-09-29 FileList(3)