1FileList(3) User Contributed Perl Documentation FileList(3)
2
6 File::RsyncP::FileList - Perl interface to rsync file list encoding and
7 decoding.
8
10 use File::RsyncP::FileList;
11 $fileList = File::RsyncP::FileList->new({
13 preserve_uid => 1,
14 preserve_gid => 1,
15 preserve_links => 1,
16 preserve_devices => 1,
17 preserve_hard_links => 0,
18 always_checksum => 0,
19 remote_version => 26,
20 });
21 # decoding an incoming file list
23 while ( !$fileList->decodeDone && !$fileList->fatalError ) {
24 $data .= readMoreDataFromRemoteRsync();
25 $bytesDone = $fileList->decode($data);
26 $data = substr($data, $bytesDone) if ( $bytesDone > 0 );
27 }
28 $fileList->clean;
29 # create (encode) a file list
31 $fileList->encode({
32 name => $filePath,
33 dev => $stat[0],
34 inode => $stat[1],
35 mode => $stat[2],
36 uid => $stat[4],
37 gid => $stat[5],
38 rdev => $stat[6],
39 mtime => $stat[9],
40 });
41 $data = $fileList->encodeData;
42 # get file information, for file number 5:
44 $fileInfo = $fileList->get(5);
45 # utility functions
47 $numberOfFiles = $fileList->count;
48 $gotFatalError = $fileList->fatalError;
49
51 The File::RsyncP::FileList module is used to encode and decode file
52 lists in using the same format at Rsync.
53 The sender side of Rsync creates a list of all the files the are going
55 to be sent. This list is sent in a compact format to the receiver
56 side. Each side then sorts the list and removes duplicate entries.
57 From this point on, all files are referred to by their integer index
58 into the sorted file list.
59 A new file list object is created by calling
61 File::RsyncP::FileList->new. An object can be used to decode or encode
62 a file list. There is no mechanism to reset the state of a file list:
63 you should create a new object each time you need to do a new decode or
64 encode.
65 The new() function takes a hashref of options, which correspond to var‐
67 ious rsync command-line switches. These must exactly match the argu‐
68 ments to the remote rsync, otherwise the file list format will not be
69 compatible and decoding will fail.
70 $fileList = File::RsyncP::FileList->new({
72 preserve_uid => 1, # --owner
73 preserve_gid => 1, # --group
74 preserve_links => 1, # --links
75 preserve_devices => 1, # --devices
76 preserve_hard_links => 0, # --hard-links
77 always_checksum => 0, # --checksum
78 remote_version => 26, # remote protocol version
79 });
80 Decoding
82 The decoding functions take a stream of bytes from the remote rsync and
84 convert them into an internal data structure. Rather than store the
85 file list as a native perl list of hashes (which occupies too much mem‐
86 ory for large file lists), the same internal data structure as rsync is
87 used. Individual file list entries can be returned with the get()
88 function.
89 File list data read from the remote rsync should be passed to the
91 decode() function. The data may be read and processed in arbitrary
92 sized chunks. The decode() function returns how many bytes were actu‐
93 ally processed. It is the caller's responsbility to remove that number
94 of bytes from the input argument, preserving the remaining bytes for
95 the next call to decode(). The decodeDone() function returns true when
96 the file list is complete. The fatalError() function returns true if
97 there was a non-recoverable error while decoding.
98 The clean() function needs to be called after the file list decode is
100 complete. The clean() function sorts the file list and removes
101 repeated entries. Skipping this step will produce unexpected results:
102 since files are referred to using integers, each side will refer to
103 different files is the file lists are not sorted and purged in exactly
104 the same manner.
105 A typical decode loop looks like:
107 while ( !$fileList->decodeDone && !$fileList->fatalError ) {
109 $data .= readMoreDataFromRemoteRsync();
110 $bytesDone = $fileList->decode($data);
111 $data = substr($data, $bytesDone) if ( $bytesDone > 0 );
112 }
113 $fileList->clean;
114 After clean() is called, the number of files in the file list can be
116 found by calling count(). Files can be fetched by calling the get()
117 function, with an index from 0 to count()-1:
118 $fileInfo = $fileList->get(5);
120 The get() function returns a hashref with various entries:
122 name path name of the file (relative to rsync dir):
124 equal to dirname/basename
125 basename file name, without directory
126 dirname directory where file resides
127 sum file MD4 checksum (only present if --checksum specified)
128 uid file user id
129 gid file group id
130 mode file mode
131 mtime file modification time
132 size file length
133 dev device number on which file resides
134 inode file inode
135 link link contents if the file is a sym link
136 rdev major/minor device number if file is char/block special
137 Various fields will only have valid values if the corresponding options
139 are set (eg: uid if preserve_uid is set, dev and inode if pre‐
140 serve_hard_links is set etc).
141 For example, to dump out each of hash you could do this:
143 use Data::Dumper;
145 my $count = $fileList->count;
146 for ( my $i = 0 ; $i < $count ; $i++ ) {
147 print("File $i is:\n");
148 print Dumper($fileList->get($i));
149 }
150 Encoding
152 The encode() function is used to build a file list in preparation for
154 encoding and sending a file list to a remote rsync. The encode() func‐
155 tion takes a hashref argument with the parameters for one file. It
156 should be called once for each file. The parameter names are the same
157 as those returned by get().
158 In this example the matching stat() values are shown:
160 $fileList->encode({
162 name => $filePath,
163 dev => $stat[0],
164 inode => $stat[1],
165 mode => $stat[2],
166 uid => $stat[4],
167 gid => $stat[5],
168 rdev => $stat[6],
169 size => $stat[7],
170 mtime => $stat[9],
171 });
172 It is not necessary to specify basename and dirname; these are
174 extracted from name. You only need to specify the parameters that
175 match the options given to new(). You can also specify sum and link as
176 necessary.
177 To compute the encoded file list data the encodeData() function should
179 be called. It can be called every time encode() is called, or once at
180 the end of all the encode() calls. It returns the encoded data that
181 should be sent to the remote rsync:
182 $data = $fileList->encodeData;
184 It is recommended that encodeData() be called frequently to avoid the
186 need to allocate large internal buffers to hold the entire encoded file
187 list. Since encodeData() does not know when the last file has been
188 encoded, it is the caller's responsbility to add the final null byte
189 (eg: pack("C", 0)) to the data to indicate the end of the file list
190 data.
191 After all the file list entries are processed you should call clean():
193 $fileList->clean;
195 This ensures that each side (sender/receiver) has identical sorted file
197 lists.
198 Utility functions
200 The count() function returns the total number of files in the internal
202 file list (either decoded or encoded).
203 The fatalError() function returns true if a fatal error has occured
205 during file decoding. It should be called in the decode loop to make
206 sure no error has occured.
207
209 File::RsyncP::FileList was written by Craig Barratt <cbar‐
210 ratt@users.sourceforge.net> based on rsync 2.5.5.
211 Rsync was written by Andrew Tridgell <tridge@samba.org> and Paul Mack‐
213 erras. It is available under a GPL license. See
214 http://rsync.samba.org
215
217 This program is free software; you can redistribute it and/or modify it
218 under the terms of the GNU General Public License as published by the
219 Free Software Foundation; either version 2 of the License, or (at your
220 option) any later version.
221 This program is distributed in the hope that it will be useful, but
223 WITHOUT ANY WARRANTY; without even the implied warranty of MER‐
224 CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
225 Public License for more details.
226 You should have received a copy of the GNU General Public License in
228 the LICENSE file along with this program; if not, write to the Free
229 Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
230 02111-1307 USA.
231
233 See <http://perlrsync.sourceforge.net> for File::RsyncP's SourceForge
234 home page.
235 See File::RsyncP and File::RsyncP::FileIO for more precise examples of
237 using File::RsyncP::FileList.
238 Also see BackupPC's lib/BackupPC/Xfer/RsyncFileIO.pm for other exam‐
240 ples.
241perl v5.8.8 2006-11-19 FileList(3)