1Tie::File(3pm) Perl Programmers Reference Guide Tie::File(3pm)
2
6 Tie::File - Access the lines of a disk file via a Perl array
7
9 # This file documents Tie::File version 0.97
10 use Tie::File;
11 tie @array, 'Tie::File', filename or die ...;
13 $array[13] = 'blah'; # line 13 of the file is now 'blah'
15 print $array[42]; # display line 42 of the file
16 $n_recs = @array; # how many records are in the file?
18 $#array -= 2; # chop two records off the end
19 for (@array) {
21 s/PERL/Perl/g; # Replace PERL with Perl everywhere in the file
22 }
23 # These are just like regular push, pop, unshift, shift, and splice
25 # Except that they modify the file in the way you would expect
26 push @array, new recs...;
28 my $r1 = pop @array;
29 unshift @array, new recs...;
30 my $r2 = shift @array;
31 @old_recs = splice @array, 3, 7, new recs...;
32 untie @array; # all finished
34
36 "Tie::File" represents a regular text file as a Perl array. Each ele‐
37 ment in the array corresponds to a record in the file. The first line
38 of the file is element 0 of the array; the second line is element 1,
39 and so on.
40 The file is not loaded into memory, so this will work even for gigantic
42 files.
43 Changes to the array are reflected in the file immediately.
45 Lazy people and beginners may now stop reading the manual.
47 "recsep"
49 What is a 'record'? By default, the meaning is the same as for the
51 "<...>" operator: It's a string terminated by $/, which is probably
52 "\n". (Minor exception: on DOS and Win32 systems, a 'record' is a
53 string terminated by "\r\n".) You may change the definition of
54 "record" by supplying the "recsep" option in the "tie" call:
55 tie @array, 'Tie::File', $file, recsep => 'es';
57 This says that records are delimited by the string "es". If the file
59 contained the following data:
60 Curse these pesky flies!\n
62 then the @array would appear to have four elements:
64 "Curse th"
66 "e p"
67 "ky fli"
68 "!\n"
69 An undefined value is not permitted as a record separator. Perl's spe‐
71 cial "paragraph mode" semantics (A la "$/ = """) are not emulated.
72 Records read from the tied array do not have the record separator
74 string on the end; this is to allow
75 $array[17] .= "extra";
77 to work as expected.
79 (See "autochomp", below.) Records stored into the array will have the
81 record separator string appended before they are written to the file,
82 if they don't have one already. For example, if the record separator
83 string is "\n", then the following two lines do exactly the same thing:
84 $array[17] = "Cherry pie";
86 $array[17] = "Cherry pie\n";
87 The result is that the contents of line 17 of the file will be replaced
89 with "Cherry pie"; a newline character will separate line 17 from line
90 18. This means that this code will do nothing:
91 chomp $array[17];
93 Because the "chomp"ed value will have the separator reattached when it
95 is written back to the file. There is no way to create a file whose
96 trailing record separator string is missing.
97 Inserting records that contain the record separator string is not sup‐
99 ported by this module. It will probably produce a reasonable result,
100 but what this result will be may change in a future version. Use
101 'splice' to insert records or to replace one record with several.
102 "autochomp"
104 Normally, array elements have the record separator removed, so that if
106 the file contains the text
107 Gold
109 Frankincense
110 Myrrh
111 the tied array will appear to contain "("Gold", "Frankincense",
113 "Myrrh")". If you set "autochomp" to a false value, the record separa‐
114 tor will not be removed. If the file above was tied with
115 tie @gifts, "Tie::File", $gifts, autochomp => 0;
117 then the array @gifts would appear to contain "("Gold\n", "Frankin‐
119 cense\n", "Myrrh\n")", or (on Win32 systems) "("Gold\r\n", "Frankin‐
120 cense\r\n", "Myrrh\r\n")".
121 "mode"
123 Normally, the specified file will be opened for read and write access,
125 and will be created if it does not exist. (That is, the flags "O_RDWR
126 ⎪ O_CREAT" are supplied in the "open" call.) If you want to change
127 this, you may supply alternative flags in the "mode" option. See Fcntl
128 for a listing of available flags. For example:
129 # open the file if it exists, but fail if it does not exist
131 use Fcntl 'O_RDWR';
132 tie @array, 'Tie::File', $file, mode => O_RDWR;
133 # create the file if it does not exist
135 use Fcntl 'O_RDWR', 'O_CREAT';
136 tie @array, 'Tie::File', $file, mode => O_RDWR ⎪ O_CREAT;
137 # open an existing file in read-only mode
139 use Fcntl 'O_RDONLY';
140 tie @array, 'Tie::File', $file, mode => O_RDONLY;
141 Opening the data file in write-only or append mode is not supported.
143 "memory"
145 This is an upper limit on the amount of memory that "Tie::File" will
147 consume at any time while managing the file. This is used for two
148 things: managing the read cache and managing the deferred write buffer.
149 Records read in from the file are cached, to avoid having to re-read
151 them repeatedly. If you read the same record twice, the first time it
152 will be stored in memory, and the second time it will be fetched from
153 the read cache. The amount of data in the read cache will not exceed
154 the value you specified for "memory". If "Tie::File" wants to cache a
155 new record, but the read cache is full, it will make room by expiring
156 the least-recently visited records from the read cache.
157 The default memory limit is 2Mib. You can adjust the maximum read
159 cache size by supplying the "memory" option. The argument is the
160 desired cache size, in bytes.
161 # I have a lot of memory, so use a large cache to speed up access
163 tie @array, 'Tie::File', $file, memory => 20_000_000;
164 Setting the memory limit to 0 will inhibit caching; records will be
166 fetched from disk every time you examine them.
167 The "memory" value is not an absolute or exact limit on the memory
169 used. "Tie::File" objects contains some structures besides the read
170 cache and the deferred write buffer, whose sizes are not charged
171 against "memory".
172 The cache itself consumes about 310 bytes per cached record, so if your
174 file has many short records, you may want to decrease the cache memory
175 limit, or else the cache overhead may exceed the size of the cached
176 data.
177 "dw_size"
179 (This is an advanced feature. Skip this section on first reading.)
181 If you use deferred writing (See "Deferred Writing", below) then data
183 you write into the array will not be written directly to the file;
184 instead, it will be saved in the deferred write buffer to be written
185 out later. Data in the deferred write buffer is also charged against
186 the memory limit you set with the "memory" option.
187 You may set the "dw_size" option to limit the amount of data that can
189 be saved in the deferred write buffer. This limit may not exceed the
190 total memory limit. For example, if you set "dw_size" to 1000 and
191 "memory" to 2500, that means that no more than 1000 bytes of deferred
192 writes will be saved up. The space available for the read cache will
193 vary, but it will always be at least 1500 bytes (if the deferred write
194 buffer is full) and it could grow as large as 2500 bytes (if the
195 deferred write buffer is empty.)
196 If you don't specify a "dw_size", it defaults to the entire memory
198 limit.
199 Option Format
201 "-mode" is a synonym for "mode". "-recsep" is a synonym for "recsep".
203 "-memory" is a synonym for "memory". You get the idea.
204
206 The "tie" call returns an object, say $o. You may call
207 $rec = $o->FETCH($n);
209 $o->STORE($n, $rec);
210 to fetch or store the record at line $n, respectively; similarly the
212 other tied array methods. (See perltie for details.) You may also
213 call the following methods on this object:
214 "flock"
216 $o->flock(MODE)
218 will lock the tied file. "MODE" has the same meaning as the second
220 argument to the Perl built-in "flock" function; for example "LOCK_SH"
221 or "LOCK_EX ⎪ LOCK_NB". (These constants are provided by the "use
222 Fcntl ':flock'" declaration.)
223 "MODE" is optional; the default is "LOCK_EX".
225 "Tie::File" maintains an internal table of the byte offset of each
227 record it has seen in the file.
228 When you use "flock" to lock the file, "Tie::File" assumes that the
230 read cache is no longer trustworthy, because another process might have
231 modified the file since the last time it was read. Therefore, a suc‐
232 cessful call to "flock" discards the contents of the read cache and the
233 internal record offset table.
234 "Tie::File" promises that the following sequence of operations will be
236 safe:
237 my $o = tie @array, "Tie::File", $filename;
239 $o->flock;
240 In particular, "Tie::File" will not read or write the file during the
242 "tie" call. (Exception: Using "mode => O_TRUNC" will, of course, erase
243 the file during the "tie" call. If you want to do this safely, then
244 open the file without "O_TRUNC", lock the file, and use "@array = ()".)
245 The best way to unlock a file is to discard the object and untie the
247 array. It is probably unsafe to unlock the file without also untying
248 it, because if you do, changes may remain unwritten inside the object.
249 That is why there is no shortcut for unlocking. If you really want to
250 unlock the file prematurely, you know what to do; if you don't know
251 what to do, then don't do it.
252 All the usual warnings about file locking apply here. In particular,
254 note that file locking in Perl is advisory, which means that holding a
255 lock will not prevent anyone else from reading, writing, or erasing the
256 file; it only prevents them from getting another lock at the same time.
257 Locks are analogous to green traffic lights: If you have a green light,
258 that does not prevent the idiot coming the other way from plowing into
259 you sideways; it merely guarantees to you that the idiot does not also
260 have a green light at the same time.
261 "autochomp"
263 my $old_value = $o->autochomp(0); # disable autochomp option
265 my $old_value = $o->autochomp(1); # enable autochomp option
266 my $ac = $o->autochomp(); # recover current value
268 See "autochomp", above.
270 "defer", "flush", "discard", and "autodefer"
272 See "Deferred Writing", below.
274 "offset"
276 $off = $o->offset($n);
278 This method returns the byte offset of the start of the $nth record in
280 the file. If there is no such record, it returns an undefined value.
281
283 If $fh is a filehandle, such as is returned by "IO::File" or one of the
284 other "IO" modules, you may use:
285 tie @array, 'Tie::File', $fh, ...;
287 Similarly if you opened that handle "FH" with regular "open" or
289 "sysopen", you may use:
290 tie @array, 'Tie::File', \*FH, ...;
292 Handles that were opened write-only won't work. Handles that were
294 opened read-only will work as long as you don't try to modify the
295 array. Handles must be attached to seekable sources of data---that
296 means no pipes or sockets. If "Tie::File" can detect that you supplied
297 a non-seekable handle, the "tie" call will throw an exception. (On
298 Unix systems, it can detect this.)
299 Note that Tie::File will only close any filehandles that it opened
301 internally. If you passed it a filehandle as above, you "own" the
302 filehandle, and are responsible for closing it after you have untied
303 the @array.
304
306 (This is an advanced feature. Skip this section on first reading.)
307 Normally, modifying a "Tie::File" array writes to the underlying file
309 immediately. Every assignment like "$a[3] = ..." rewrites as much of
310 the file as is necessary; typically, everything from line 3 through the
311 end will need to be rewritten. This is the simplest and most transpar‐
312 ent behavior. Performance even for large files is reasonably good.
313 However, under some circumstances, this behavior may be excessively
315 slow. For example, suppose you have a million-record file, and you
316 want to do:
317 for (@FILE) {
319 $_ = "> $_";
320 }
321 The first time through the loop, you will rewrite the entire file, from
323 line 0 through the end. The second time through the loop, you will re‐
324 write the entire file from line 1 through the end. The third time
325 through the loop, you will rewrite the entire file from line 2 to the
326 end. And so on.
327 If the performance in such cases is unacceptable, you may defer the
329 actual writing, and then have it done all at once. The following loop
330 will perform much better for large files:
331 (tied @a)->defer;
333 for (@a) {
334 $_ = "> $_";
335 }
336 (tied @a)->flush;
337 If "Tie::File"'s memory limit is large enough, all the writing will
339 done in memory. Then, when you call "->flush", the entire file will be
340 rewritten in a single pass.
341 (Actually, the preceding discussion is something of a fib. You don't
343 need to enable deferred writing to get good performance for this common
344 case, because "Tie::File" will do it for you automatically unless you
345 specifically tell it not to. See "autodeferring", below.)
346 Calling "->flush" returns the array to immediate-write mode. If you
348 wish to discard the deferred writes, you may call "->discard" instead
349 of "->flush". Note that in some cases, some of the data will have been
350 written already, and it will be too late for "->discard" to discard all
351 the changes. Support for "->discard" may be withdrawn in a future ver‐
352 sion of "Tie::File".
353 Deferred writes are cached in memory up to the limit specified by the
355 "dw_size" option (see above). If the deferred-write buffer is full and
356 you try to write still more deferred data, the buffer will be flushed.
357 All buffered data will be written immediately, the buffer will be emp‐
358 tied, and the now-empty space will be used for future deferred writes.
359 If the deferred-write buffer isn't yet full, but the total size of the
361 buffer and the read cache would exceed the "memory" limit, the oldest
362 records will be expired from the read cache until the total size is
363 under the limit.
364 "push", "pop", "shift", "unshift", and "splice" cannot be deferred.
366 When you perform one of these operations, any deferred data is written
367 to the file and the operation is performed immediately. This may
368 change in a future version.
369 If you resize the array with deferred writing enabled, the file will be
371 resized immediately, but deferred records will not be written. This
372 has a surprising consequence: "@a = (...)" erases the file immediately,
373 but the writing of the actual data is deferred. This might be a bug.
374 If it is a bug, it will be fixed in a future version.
375 Autodeferring
377 "Tie::File" tries to guess when deferred writing might be helpful, and
379 to turn it on and off automatically.
380 for (@a) {
382 $_ = "> $_";
383 }
384 In this example, only the first two assignments will be done immedi‐
386 ately; after this, all the changes to the file will be deferred up to
387 the user-specified memory limit.
388 You should usually be able to ignore this and just use the module with‐
390 out thinking about deferring. However, special applications may
391 require fine control over which writes are deferred, or may require
392 that all writes be immediate. To disable the autodeferment feature,
393 use
394 (tied @o)->autodefer(0);
396 or
398 tie @array, 'Tie::File', $file, autodefer => 0;
400 Similarly, "->autodefer(1)" re-enables autodeferment, and "->autode‐
402 fer()" recovers the current value of the autodefer setting.
403
405 Caching and deferred writing are inappropriate if you want the same
406 file to be accessed simultaneously from more than one process. Other
407 optimizations performed internally by this module are also incompatible
408 with concurrent access. A future version of this module will support a
409 "concurrent => 1" option that enables safe concurrent access.
410 Previous versions of this documentation suggested using "memory => 0"
412 for safe concurrent access. This was mistaken. Tie::File will not
413 support safe concurrent access before version 0.98.
414
416 (That's Latin for 'warnings'.)
417 · Reasonable effort was made to make this module efficient. Never‐
419 theless, changing the size of a record in the middle of a large
420 file will always be fairly slow, because everything after the new
421 record must be moved.
422 · The behavior of tied arrays is not precisely the same as for regu‐
424 lar arrays. For example:
425 # This DOES print "How unusual!"
427 undef $a[10]; print "How unusual!\n" if defined $a[10];
428 "undef"-ing a "Tie::File" array element just blanks out the corre‐
430 sponding record in the file. When you read it back again, you'll
431 get the empty string, so the supposedly-"undef"'ed value will be
432 defined. Similarly, if you have "autochomp" disabled, then
433 # This DOES print "How unusual!" if 'autochomp' is disabled
435 undef $a[10];
436 print "How unusual!\n" if $a[10];
437 Because when "autochomp" is disabled, $a[10] will read back as "\n"
439 (or whatever the record separator string is.)
440 There are other minor differences, particularly regarding "exists"
442 and "delete", but in general, the correspondence is extremely
443 close.
444 · I have supposed that since this module is concerned with file I/O,
446 almost all normal use of it will be heavily I/O bound. This means
447 that the time to maintain complicated data structures inside the
448 module will be dominated by the time to actually perform the I/O.
449 When there was an opportunity to spend CPU time to avoid doing I/O,
450 I usually tried to take it.
451 · You might be tempted to think that deferred writing is like trans‐
453 actions, with "flush" as "commit" and "discard" as "rollback", but
454 it isn't, so don't.
455 · There is a large memory overhead for each record offset and for
457 each cache entry: about 310 bytes per cached data record, and about
458 21 bytes per offset table entry.
459 The per-record overhead will limit the maximum number of records
461 you can access per file. Note that accessing the length of the
462 array via "$x = scalar @tied_file" accesses all records and stores
463 their offsets. The same for "foreach (@tied_file)", even if you
464 exit the loop early.
465
467 This version promises absolutely nothing about the internals, which may
468 change without notice. A future version of the module will have a
469 well-defined and stable subclassing API.
470
472 People sometimes point out that DB_File will do something similar, and
473 ask why "Tie::File" module is necessary.
474 There are a number of reasons that you might prefer "Tie::File". A
476 list is available at "http://perl.plover.com/TieFile/why-not-DB_File".
477
479 Mark Jason Dominus
480 To contact the author, send email to: "mjd-perl-tiefile+@plover.com"
482 To receive an announcement whenever a new version of this module is
484 released, send a blank email message to "mjd-perl-tiefile-sub‐
485 scribe@plover.com".
486 The most recent version of this module, including documentation and any
488 news of importance, will be available at
489 http://perl.plover.com/TieFile/
491
493 "Tie::File" version 0.97 is copyright (C) 2003 Mark Jason Dominus.
494 This library is free software; you may redistribute it and/or modify it
496 under the same terms as Perl itself.
497 These terms are your choice of any of (1) the Perl Artistic Licence, or
499 (2) version 2 of the GNU General Public License as published by the
500 Free Software Foundation, or (3) any later version of the GNU General
501 Public License.
502 This library is distributed in the hope that it will be useful, but
504 WITHOUT ANY WARRANTY; without even the implied warranty of MER‐
505 CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
506 Public License for more details.
507 You should have received a copy of the GNU General Public License along
509 with this library program; it should be in the file "COPYING". If not,
510 write to the Free Software Foundation, Inc., 59 Temple Place, Suite
511 330, Boston, MA 02111 USA
512 For licensing inquiries, contact the author at:
514 Mark Jason Dominus
516 255 S. Warnock St.
517 Philadelphia, PA 19107
518
520 "Tie::File" version 0.97 comes with ABSOLUTELY NO WARRANTY. For
521 details, see the license.
522
524 Gigantic thanks to Jarkko Hietaniemi, for agreeing to put this in the
525 core when I hadn't written it yet, and for generally being helpful,
526 supportive, and competent. (Usually the rule is "choose any one.")
527 Also big thanks to Abhijit Menon-Sen for all of the same things.
528 Special thanks to Craig Berry and Peter Prymmer (for VMS portability
530 help), Randy Kobes (for Win32 portability help), Clinton Pierce and
531 Autrijus Tang (for heroic eleventh-hour Win32 testing above and beyond
532 the call of duty), Michael G Schwern (for testing advice), and the rest
533 of the CPAN testers (for testing generally).
534 Special thanks to Tels for suggesting several speed and memory opti‐
536 mizations.
537 Additional thanks to: Edward Avis / Mattia Barbon / Tom Christiansen /
539 Gerrit Haase / Gurusamy Sarathy / Jarkko Hietaniemi (again) / Nikola
540 Knezevic / John Kominetz / Nick Ing-Simmons / Tassilo von Parseval / H.
541 Dieter Pearcey / Slaven Rezic / Eric Roode / Peter Scott / Peter Somu /
542 Autrijus Tang (again) / Tels (again) / Juerd Waalboer
543
545 More tests. (Stuff I didn't think of yet.)
546 Paragraph mode?
548 Fixed-length mode. Leave-blanks mode.
550 Maybe an autolocking mode?
552 For many common uses of the module, the read cache is a liability. For
554 example, a program that inserts a single record, or that scans the file
555 once, will have a cache hit rate of zero. This suggests a major opti‐
556 mization: The cache should be initially disabled. Here's a hybrid
557 approach: Initially, the cache is disabled, but the cache code main‐
558 tains statistics about how high the hit rate would be *if* it were
559 enabled. When it sees the hit rate get high enough, it enables itself.
560 The STAT comments in this code are the beginning of an implementation
561 of this.
562 Record locking with fcntl()? Then the module might support an undo log
564 and get real transactions. What a tour de force that would be.
565 Keeping track of the highest cached record. This would allow reads-in-
567 a-row to skip the cache lookup faster (if reading from 1..N with empty
568 cache at start, the last cached value will be always N-1).
569 More tests.
571perl v5.8.8 2001-09-21 Tie::File(3pm)