1FlexRaw(3) User Contributed Perl Documentation FlexRaw(3)
2
6 PDL::IO::FlexRaw -- A flexible binary i/o format for PerlDL.
7
9 use PDL;
10 use PDL::IO::FlexRaw;
11 ($x,$y,...) = readflex("filename" [, $hdr])
13 ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
14 $hdr = writeflex($file, $pdl1, $pdl2,...)
16 writeflexhdr($file, $hdr)
17
19 FlexRaw is a generic method for the input and output of `raw' data
20 arrays. In particular, it is designed to read output from FORTRAN 77
21 UNFORMATTED files and the low-level C write function, even if the files
22 are compressed or gzipped. As in FastRaw, the data file is supple‐
23 mented by a header file (although this can be replaced by the optional
24 $hdr argument). More information can be included in the header file
25 than for FastRaw -- the description can be extended to several data
26 objects within a single input file.
27 For example, to read the output of a FORTRAN program
29 real*4 a(4,600,600)
31 open (8,file='banana',status='new',form='unformatted')
32 write (8) a
33 close (8)
34 the header file (`banana.hdr') could look like
36 # FlexRaw file header
38 # Header word for F77 form=unformatted
39 Byte 1 4
40 # Data
41 Float 3 # this is ignored
42 4 600 600
43 Byte 1 4 As is this, as we've got all dims
44 The data can then be input using
46 $a = (readflex('banana'))[1];
48 The format of the hdr file is an extension of that used by FastRaw.
50 Comment lines (starting with #) are allowed, as are descriptive names
51 (as elsewhere: byte, short, ushort, long, float, double) for the data
52 types -- note that case is ignored by FlexRaw. After the type, one
53 integer specifies the number of dimensions of the data `chunk', and
54 subsequent integers the size of each dimension. So the specifier above
55 (`Float 3 4 600 600') describes our FORTRAN array. A scalar can be
56 described as `float 0' (or `float 1 1', or `float 2 1 1', etc.). When
57 all the dimensions are read -- or a # appears after whitespace -- the
58 rest of the current input line is ignored.
59 What about the extra 4 bytes at the head and tail, which we just threw
61 away? These are added by FORTRAN (at least on Suns, Alphas and Linux),
62 and specify the number of bytes written by each WRITE -- the same num‐
63 ber is put at the start and the end of each chunk of data. You may
64 need to know all this in some cases. In general, FlexRaw tries to han‐
65 dle it itself, if you simply add a line saying `f77' to the header
66 file, before any data specifiers:
67 # FlexRaw file header for F77 form=unformatted
69 F77
70 # Data
71 Float 3
72 4 600 600
73 -- the redundancy in FORTRAN data files even allows FlexRaw to automat‐
75 ically deal with files written on other machines which use back-to-
76 front byte ordering. This won't always work -- it's a 1 in 4 billion
77 chance it won't, even if you regularly read 4Gb files! Also, it cur‐
78 rently doesn't work for compressed files, so you can say `swap' (again
79 before any data specifiers) to make certain the byte order is swapped.
80 The optional $hdr argument allows the use of an anonymous array to give
82 header information, rather than using a .hdr file. For example,
83 $header = [
85 {Type => 'f77'},
86 {Type => 'float', NDims => 3, Dims => [ 4,600,600 ] }
87 ];
88 @a = readflex('banana',$header);
89 reads our example file again. As a special case, when NDims is 1, Dims
91 may be given as a scalar.
92 Within PDL, readflex and writeflex can be used to write several pdls to
94 a single file -- e.g.
95 use PDL;
97 use PDL::IO::FastRaw;
98 @pdls = ($pdl1, $pdl2, ...);
100 $hdr = writeflex("fname",@pdls);
101 @pdl2 = readflex("fname",$hdr);
102 writeflexhdr("fname",$hdr);
104 @pdl3 = readflex("fname");
105 -- writeflex produces the data file and returns the file header as an
107 anonymous hash, which can be written to a .hdr file using writeflexhdr.
108 The reading of compressed data is switched on automatically if the
110 filename requested ends in .gz or .Z, or if the originally specified
111 filename does not exist, but one of these compressed forms does.
112 If writeflex and readflex are given a reference to a file handle as a
114 first parameter instead of a filename, then the data is read or written
115 to the open filehandle. This gives an easy way to read an arbitrary
116 slice in a big data volume, as in the following example:
117 use PDL;
119 use PDL::IO::FastRaw;
120 open(DATA, "raw3d.dat");
122 binmode(DATA);
123 # assume we know the data size from an external source
125 ($width, $height, $data_size) = (256,256, 4);
126 my $slice_num = 64; # slice to look at
128 # Seek to slice
129 seek(DATA, $width*$height*$data_size * $slice_num, 0);
130 $pdl = readflex \*DATA, [{Dims=>[$width, $height], Type=>'long'}];
131 WARNING: In later versions of perl (5.8 and up) you must be sure that
133 your file is in "raw" mode (see the perlfunc man page entry for "bin‐
134 mode", for details). Both readflex and writeflex automagically switch
135 the file to raw mode for you -- but in code like the snipped above, you
136 could end up seeking the wrong byte if you forget to make the binmode()
137 call.
138 Mapflex memory maps, rather than reads, the data files. Its interface
140 is similar to `readflex'. Extra options specify if the data is to be
141 loaded `ReadOnly', if the data file is to be `Creat'-ed anew on the
142 basis of the header information or `Trunc'-ated to the length of the
143 data read. The extra speed of access brings with it some limitations:
144 mapflex won't read compressed data, auto-detect f77 files or read f77
145 files written by more than a single unformatted write statement. More
146 seriously, data alignment constraints mean that mapflex cannot read
147 some files, depending on the requirements of the host OS (it may also
148 vary depending on the setting of the `uac' flag on any given machine).
149 You may have run into similar problems with common blocks in FORTRAN.
150 For instance, floating point numbers may have to align on 4 byte bound‐
152 aries -- if the data file consists of 3 bytes then a float, it cannot
153 be read. Mapflex will warn about this problem when it occurs, and
154 return the PDLs mapped before the problem arose. This can be dealt
155 with either by reorganizing the data file (large types first helps, as
156 a rule-of-thumb), or more simply by using `readflex'.
157
159 The test on two dimensional byte arrays fail using g77 2.7.2, but not
160 Sun f77. I hope this isn't my problem!
161 Assumes gzip is on the PATH.
163 Can't auto-swap compressed files, because it can't seek on them.
165 The header format may not agree with that used elsewhere.
167 Should it handle handles?
169 Mapflex should warn and fallback to reading on SEGV? Would have to
171 make sure that the data was written back after it was `destroyed'.
172
174 readflex
175 Read a binary file with flexible format specification
177 ($x,$y,...) = readflex("filename" [, $hdr])
179 ($x,$y,...) = readflex(FILEHANDLE [, $hdr])
180 writeflex
182 Write a binary file with flexible format specification
184 $hdr = writeflex($file, $pdl1, $pdl2,...)
186 $hdr = writeflex(FILEHANDLE, $pdl1, $pdl2,...)
187 mapflex
189 Memory map a binary file with flexible format specification
191 ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
193
195 Copyright (C) Robin Williams <rjrw@ast.leeds.ac.uk> 1997. All rights
196 reserved. There is no warranty. You are allowed to redistribute this
197 software / documentation under certain conditions. For details, see the
198 file COPYING in the PDL distribution. If this file is separated from
199 the PDL distribution, the copyright notice should be included in the
200 file.
201perl v5.8.8 2003-05-19 FlexRaw(3)