1FlexRaw(3) User Contributed Perl Documentation FlexRaw(3)
2
6 PDL::IO::FlexRaw -- A flexible binary I/O format for PerlDL
7
9 use PDL;
10 use PDL::IO::FlexRaw;
11 # To obtain the header for reading (if multiple files use the
13 # same header, for example):
14 #
15 $hdr = PDL::IO::FlexRaw::_read_flexhdr("filename.hdr")
16 ($x,$y,...) = readflex("filename" [, $hdr])
18 ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
19 $hdr = writeflex($file, $pdl1, $pdl2,...)
21 writeflexhdr($file, $hdr)
22 # if $PDL::IO::FlexRaw::writeflexhdr is true and
24 # $file is a filename, writeflexhdr() is called automatically
25 #
26 $hdr = writeflex($file, $pdl1, $pdl2,...) # need $hdr for something
27 writeflex($file, $pdl1, $pdl2,...) # ..if $hdr not needed
28
30 FlexRaw is a generic method for the input and output of `raw' data
31 arrays. In particular, it is designed to read output from FORTRAN 77
32 UNFORMATTED files and the low-level C write function, even if the files
33 are compressed or gzipped. As in FastRaw, the data file is
34 supplemented by a header file (although this can be replaced by the
35 optional $hdr argument). More information can be included in the
36 header file than for FastRaw -- the description can be extended to
37 several data objects within a single input file.
38 For example, to read the output of a FORTRAN program
40 real*4 a(4,600,600)
42 open (8,file='banana',status='new',form='unformatted')
43 write (8) a
44 close (8)
45 the header file (`banana.hdr') could look like
47 # FlexRaw file header
49 # Header word for F77 form=unformatted
50 Byte 1 4
51 # Data
52 Float 3 # this is ignored
53 4 600 600
54 Byte 1 4 As is this, as we've got all dims
55 The data can then be input using
57 $a = (readflex('banana'))[1];
59 The format of the hdr file is an extension of that used by FastRaw.
61 Comment lines (starting with #) are allowed, as are descriptive names
62 (as elsewhere: byte, short, ushort, long, float, double) for the data
63 types -- note that case is ignored by FlexRaw. After the type, one
64 integer specifies the number of dimensions of the data `chunk', and
65 subsequent integers the size of each dimension. So the specifier above
66 (`Float 3 4 600 600') describes our FORTRAN array. A scalar can be
67 described as `float 0' (or `float 1 1', or `float 2 1 1', etc.).
68 When all the dimensions are read -- or a # appears after whitespace --
70 the rest of the current input line is ignored, unless badvalues are
71 being read or written. In that case, the next token will be the string
72 "badvalue" followed by the bad value used, if needed.
73 What about the extra 4 bytes at the head and tail, which we just threw
75 away? These are added by FORTRAN (at least on Suns, Alphas and Linux),
76 and specify the number of bytes written by each WRITE -- the same
77 number is put at the start and the end of each chunk of data. You may
78 need to know all this in some cases. In general, FlexRaw tries to
79 handle it itself, if you simply add a line saying `f77' to the header
80 file, before any data specifiers:
81 # FlexRaw file header for F77 form=unformatted
83 F77
84 # Data
85 Float 3
86 4 600 600
87 -- the redundancy in FORTRAN data files even allows FlexRaw to
89 automatically deal with files written on other machines which use back-
90 to-front byte ordering. This won't always work -- it's a 1 in 4
91 billion chance it won't, even if you regularly read 4Gb files! Also,
92 it currently doesn't work for compressed files, so you can say `swap'
93 (again before any data specifiers) to make certain the byte order is
94 swapped.
95 The optional $hdr argument allows the use of an anonymous array to give
97 header information, rather than using a .hdr file. For example,
98 $header = [
100 {Type => 'f77'},
101 {Type => 'float', NDims => 3, Dims => [ 4,600,600 ] }
102 ];
103 @a = readflex('banana',$header);
104 reads our example file again. As a special case, when NDims is 1, Dims
106 may be given as a scalar.
107 Within PDL, readflex and writeflex can be used to write several pdls to
109 a single file -- e.g.
110 use PDL;
112 use PDL::IO::FlexRaw;
113 @pdls = ($pdl1, $pdl2, ...);
115 $hdr = writeflex("fname",@pdls);
116 @pdl2 = readflex("fname",$hdr);
117 writeflexhdr("fname",$hdr); # not needed if $PDL::IO::FlexRaw::writeflexhdr is set
119 @pdl3 = readflex("fname");
120 -- "writeflex" produces the data file and returns the file header as an
122 anonymous hash, which can be written to a .hdr file using
123 "writeflexhdr".
124 If the package variable $PDL::IO::FlexRaw::writeflexhdr is true, and
126 the "writeflex" call was with a filename and not a handle,
127 "writeflexhdr" will be called automatically (as done by "writefraw".
128 The reading of compressed data is switched on automatically if the
130 filename requested ends in .gz or .Z, or if the originally specified
131 filename does not exist, but one of these compressed forms does.
132 If "writeflex" and "readflex" are given a reference to a file handle as
134 a first parameter instead of a filename, then the data is read or
135 written to the open filehandle. This gives an easy way to read an
136 arbitrary slice in a big data volume, as in the following example:
137 use PDL;
139 use PDL::IO::FastRaw;
140 open(DATA, "raw3d.dat");
142 binmode(DATA);
143 # assume we know the data size from an external source
145 ($width, $height, $data_size) = (256,256, 4);
146 my $slice_num = 64; # slice to look at
148 # Seek to slice
149 seek(DATA, $width*$height*$data_size * $slice_num, 0);
150 $pdl = readflex \*DATA, [{Dims=>[$width, $height], Type=>'long'}];
151 WARNING: In later versions of perl (5.8 and up) you must be sure that
153 your file is in "raw" mode (see the perlfunc man page entry for
154 "binmode", for details). Both readflex and writeflex automagically
155 switch the file to raw mode for you -- but in code like the snipped
156 above, you could end up seeking the wrong byte if you forget to make
157 the binmode() call.
158 "mapflex" memory maps, rather than reads, the data files. Its
160 interface is similar to "readflex". Extra options specify if the data
161 is to be loaded `ReadOnly', if the data file is to be `Creat'-ed anew
162 on the basis of the header information or `Trunc'-ated to the length of
163 the data read. The extra speed of access brings with it some
164 limitations: "mapflex" won't read compressed data, auto-detect f77
165 files, or read f77 files written by more than a single unformatted
166 write statement. More seriously, data alignment constraints mean that
167 "mapflex" cannot read some files, depending on the requirements of the
168 host OS (it may also vary depending on the setting of the `uac' flag on
169 any given machine). You may have run into similar problems with common
170 blocks in FORTRAN.
171 For instance, floating point numbers may have to align on 4 byte
173 boundaries -- if the data file consists of 3 bytes then a float, it
174 cannot be read. "mapflex" will warn about this problem when it occurs,
175 and return the PDLs mapped before the problem arose. This can be dealt
176 with either by reorganizing the data file (large types first helps, as
177 a rule-of-thumb), or more simply by using "readflex".
178
180 The test on two dimensional byte arrays fail using g77 2.7.2, but not
181 Sun f77. I hope this isn't my problem!
182 Assumes gzip is on the PATH.
184 Can't auto-swap compressed files, because it can't seek on them.
186 The header format may not agree with that used elsewhere.
188 Should it handle handles?
190 Mapflex should warn and fallback to reading on SEGV? Would have to
192 make sure that the data was written back after it was `destroyed'.
193
195 readflex
196 Read a binary file with flexible format specification
197 Usage:
199 ($x,$y,...) = readflex("filename" [, $hdr])
201 ($x,$y,...) = readflex(FILEHANDLE [, $hdr])
202 writeflex
204 Write a binary file with flexible format specification
205 Usage:
207 $hdr = writeflex($file, $pdl1, $pdl2,...) # or
209 $hdr = writeflex(FILEHANDLE, $pdl1, $pdl2,...)
210 # now you must call writeflexhdr()
211 writeflexhdr($file, $hdr)
212 or
214 $PDL::IO::FlexRaw::writeflexhdr = 1; # set so we don't have to call writeflexhdr
216 $hdr = writeflex($file, $pdl1, $pdl2,...) # remember, $file must be filename
218 writeflex($file, $pdl1, $pdl2,...) # remember, $file must be filename
219 writeflexhdr
221 Write the header file corresponding to a previous writeflex call
222 Usage:
224 writeflexhdr($file, $hdr)
226 $file or "filename" is the filename used in a previous writeflex
228 If $file is actually a "filename" then writeflexhdr() will be
229 called automatically if $PDL::IO::FlexRaw::writeflexhdr is true.
230 If writeflex() was to a FILEHANDLE, you will need to call
231 writeflexhdr() yourself since the filename cannot be determined
232 (at least easily).
233 mapflex
235 Memory map a binary file with flexible format specification
236 Usage:
238 ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
240 All of these options default to false unless set true:
242 ReadOnly - Data should be readonly
244 Creat - Create file if it doesn't exist
245 Trunc - File should be truncated to a length that conforms
246 with the header
247 _read_flexhdr
249 Read a FlexRaw header file and return a header structure.
250 Usage:
252 $hdr = PDL::IO::FlexRaw::_read_flexhdr($file)
254 Note that "_read_flexhdr" is supposed to be an internal function. It
256 was not originally documented and it is not tested. However, there
257 appeared to be no other method for obtaining a header structure from a
258 file, so I figured I would write a small bit of documentation on it.
259
261 As of PDL-2.4.8, PDL::IO::FlexRaw has support for reading and writing
262 pdls with bad values in them.
263 On "writeflex", a piddle argument with "$pdl->badflag == 1" will have
265 the keyword/token "badvalue" added to the header file after the
266 dimension list and an additional token with the bad value for that pdl
267 if "$pdl->badvalue != $pdl->orig_badvalue".
268 On "readflex", a pdl with the "badvalue" token in the header will
270 automatically have its badflag set and its badvalue as well if it is
271 not the standard default for that type.
272 The new badvalue support required some additions to the header
274 structure. However, the interface is still being finalized. For
275 reference the current $hdr looks like this:
276 $hdr = {
278 Type => 'byte', # data type
279 NDims => 2, # number of dimensions
280 Dims => [640,480], # dims
281 BadFlag => 1, # is set/set badflag
282 BadValue => undef, # undef==default
283 };
284 $badpdl = readflex('badpdl', [$hdr]);
286 If you use bad values and try the new PDL::IO::FlexRaw bad value
288 support, please let us know via the perldl mailing list. Suggestions
289 and feedback are also welcome.
290
292 Copyright (C) Robin Williams <rjrw@ast.leeds.ac.uk> 1997. All rights
293 reserved. There is no warranty. You are allowed to redistribute this
294 software / documentation under certain conditions. For details, see the
295 file COPYING in the PDL distribution. If this file is separated from
296 the PDL distribution, the copyright notice should be included in the
297 file.
298 Documentation contributions copyright (C) David Mertens, 2010.
300perl v5.28.0 2018-05-05 FlexRaw(3)