1FlexRaw(3) User Contributed Perl Documentation FlexRaw(3)
2
6 PDL::IO::FlexRaw -- A flexible binary i/o format for PerlDL.
7
9 use PDL;
10 use PDL::IO::FlexRaw;
11 ($x,$y,...) = readflex("filename" [, $hdr])
13 ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
14 $hdr = writeflex($file, $pdl1, $pdl2,...)
16 writeflexhdr($file, $hdr)
17
19 FlexRaw is a generic method for the input and output of `raw' data
20 arrays. In particular, it is designed to read output from FORTRAN 77
21 UNFORMATTED files and the low-level C write function, even if the files
22 are compressed or gzipped. As in FastRaw, the data file is
23 supplemented by a header file (although this can be replaced by the
24 optional $hdr argument). More information can be included in the
25 header file than for FastRaw -- the description can be extended to
26 several data objects within a single input file.
27 For example, to read the output of a FORTRAN program
29 real*4 a(4,600,600)
31 open (8,file='banana',status='new',form='unformatted')
32 write (8) a
33 close (8)
34 the header file (`banana.hdr') could look like
36 # FlexRaw file header
38 # Header word for F77 form=unformatted
39 Byte 1 4
40 # Data
41 Float 3 # this is ignored
42 4 600 600
43 Byte 1 4 As is this, as we've got all dims
44 The data can then be input using
46 $a = (readflex('banana'))[1];
48 The format of the hdr file is an extension of that used by FastRaw.
50 Comment lines (starting with #) are allowed, as are descriptive names
51 (as elsewhere: byte, short, ushort, long, float, double) for the data
52 types -- note that case is ignored by FlexRaw. After the type, one
53 integer specifies the number of dimensions of the data `chunk', and
54 subsequent integers the size of each dimension. So the specifier above
55 (`Float 3 4 600 600') describes our FORTRAN array. A scalar can be
56 described as `float 0' (or `float 1 1', or `float 2 1 1', etc.). When
57 all the dimensions are read -- or a # appears after whitespace -- the
58 rest of the current input line is ignored.
59 What about the extra 4 bytes at the head and tail, which we just threw
61 away? These are added by FORTRAN (at least on Suns, Alphas and Linux),
62 and specify the number of bytes written by each WRITE -- the same
63 number is put at the start and the end of each chunk of data. You may
64 need to know all this in some cases. In general, FlexRaw tries to
65 handle it itself, if you simply add a line saying `f77' to the header
66 file, before any data specifiers:
67 # FlexRaw file header for F77 form=unformatted
69 F77
70 # Data
71 Float 3
72 4 600 600
73 -- the redundancy in FORTRAN data files even allows FlexRaw to
75 automatically deal with files written on other machines which use back-
76 to-front byte ordering. This won't always work -- it's a 1 in 4
77 billion chance it won't, even if you regularly read 4Gb files! Also,
78 it currently doesn't work for compressed files, so you can say `swap'
79 (again before any data specifiers) to make certain the byte order is
80 swapped.
81 The optional $hdr argument allows the use of an anonymous array to give
83 header information, rather than using a .hdr file. For example,
84 $header = [
86 {Type => 'f77'},
87 {Type => 'float', NDims => 3, Dims => [ 4,600,600 ] }
88 ];
89 @a = readflex('banana',$header);
90 reads our example file again. As a special case, when NDims is 1, Dims
92 may be given as a scalar.
93 Within PDL, readflex and writeflex can be used to write several pdls to
95 a single file -- e.g.
96 use PDL;
98 use PDL::IO::FastRaw;
99 @pdls = ($pdl1, $pdl2, ...);
101 $hdr = writeflex("fname",@pdls);
102 @pdl2 = readflex("fname",$hdr);
103 writeflexhdr("fname",$hdr);
105 @pdl3 = readflex("fname");
106 -- writeflex produces the data file and returns the file header as an
108 anonymous hash, which can be written to a .hdr file using writeflexhdr.
109 The reading of compressed data is switched on automatically if the
111 filename requested ends in .gz or .Z, or if the originally specified
112 filename does not exist, but one of these compressed forms does.
113 If writeflex and readflex are given a reference to a file handle as a
115 first parameter instead of a filename, then the data is read or written
116 to the open filehandle. This gives an easy way to read an arbitrary
117 slice in a big data volume, as in the following example:
118 use PDL;
120 use PDL::IO::FastRaw;
121 open(DATA, "raw3d.dat");
123 binmode(DATA);
124 # assume we know the data size from an external source
126 ($width, $height, $data_size) = (256,256, 4);
127 my $slice_num = 64; # slice to look at
129 # Seek to slice
130 seek(DATA, $width*$height*$data_size * $slice_num, 0);
131 $pdl = readflex \*DATA, [{Dims=>[$width, $height], Type=>'long'}];
132 WARNING: In later versions of perl (5.8 and up) you must be sure that
134 your file is in "raw" mode (see the perlfunc man page entry for
135 "binmode", for details). Both readflex and writeflex automagically
136 switch the file to raw mode for you -- but in code like the snipped
137 above, you could end up seeking the wrong byte if you forget to make
138 the binmode() call.
139 Mapflex memory maps, rather than reads, the data files. Its interface
141 is similar to `readflex'. Extra options specify if the data is to be
142 loaded `ReadOnly', if the data file is to be `Creat'-ed anew on the
143 basis of the header information or `Trunc'-ated to the length of the
144 data read. The extra speed of access brings with it some limitations:
145 mapflex won't read compressed data, auto-detect f77 files or read f77
146 files written by more than a single unformatted write statement. More
147 seriously, data alignment constraints mean that mapflex cannot read
148 some files, depending on the requirements of the host OS (it may also
149 vary depending on the setting of the `uac' flag on any given machine).
150 You may have run into similar problems with common blocks in FORTRAN.
151 For instance, floating point numbers may have to align on 4 byte
153 boundaries -- if the data file consists of 3 bytes then a float, it
154 cannot be read. Mapflex will warn about this problem when it occurs,
155 and return the PDLs mapped before the problem arose. This can be dealt
156 with either by reorganizing the data file (large types first helps, as
157 a rule-of-thumb), or more simply by using `readflex'.
158
160 The test on two dimensional byte arrays fail using g77 2.7.2, but not
161 Sun f77. I hope this isn't my problem!
162 Assumes gzip is on the PATH.
164 Can't auto-swap compressed files, because it can't seek on them.
166 The header format may not agree with that used elsewhere.
168 Should it handle handles?
170 Mapflex should warn and fallback to reading on SEGV? Would have to
172 make sure that the data was written back after it was `destroyed'.
173
175 readflex
176 Read a binary file with flexible format specification
177 ($x,$y,...) = readflex("filename" [, $hdr])
179 ($x,$y,...) = readflex(FILEHANDLE [, $hdr])
180 writeflex
182 Write a binary file with flexible format specification
183 $hdr = writeflex($file, $pdl1, $pdl2,...)
185 $hdr = writeflex(FILEHANDLE, $pdl1, $pdl2,...)
186 mapflex
188 Memory map a binary file with flexible format specification
189 ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
191
193 Copyright (C) Robin Williams <rjrw@ast.leeds.ac.uk> 1997. All rights
194 reserved. There is no warranty. You are allowed to redistribute this
195 software / documentation under certain conditions. For details, see the
196 file COPYING in the PDL distribution. If this file is separated from
197 the PDL distribution, the copyright notice should be included in the
198 file.
199perl v5.12.3 2009-10-24 FlexRaw(3)