1FlexRaw(3)            User Contributed Perl Documentation           FlexRaw(3)
2
3
4

NAME

6       PDL::IO::FlexRaw -- A flexible binary I/O format for PerlDL
7

SYNOPSIS

9           use PDL;
10           use PDL::IO::FlexRaw;
11
12           # To obtain the header for reading (if multiple files use the
13           # same header, for example):
14           #
15           $hdr = PDL::IO::FlexRaw::_read_flexhdr("filename.hdr")
16
17           ($x,$y,...) = readflex("filename" [, $hdr])
18           ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
19
20           $hdr = writeflex($file, $pdl1, $pdl2,...)
21           writeflexhdr($file, $hdr)
22
23           # if $PDL::IO::FlexRaw::writeflexhdr is true and
24           #    $file is a filename, writeflexhdr() is called automatically
25           #
26           $hdr = writeflex($file, $pdl1, $pdl2,...)  # need $hdr for something
27           writeflex($file, $pdl1, $pdl2,...)         # ..if $hdr not needed
28

DESCRIPTION

30       FlexRaw is a generic method for the input and output of `raw' data
31       arrays.  In particular, it is designed to read output from FORTRAN 77
32       UNFORMATTED files and the low-level C write function, even if the files
33       are compressed or gzipped.  As in FastRaw, the data file is
34       supplemented by a header file (although this can be replaced by the
35       optional $hdr argument).  More information can be included in the
36       header file than for FastRaw -- the description can be extended to
37       several data objects within a single input file.
38
39       For example, to read the output of a FORTRAN program
40
41           real*4 a(4,600,600)
42           open (8,file='banana',status='new',form='unformatted')
43           write (8) a
44           close (8)
45
46       the header file (`banana.hdr') could look like
47
48           # FlexRaw file header
49           # Header word for F77 form=unformatted
50           Byte 1 4
51           # Data
52           Float 3            # this is ignored
53                    4 600 600
54           Byte 1 4           As is this, as we've got all dims
55
56       The data can then be input using
57
58           $x = (readflex('banana'))[1];
59
60       The format of the hdr file is an extension of that used by FastRaw.
61       Comment lines (starting with #) are allowed, as are descriptive names
62       (as elsewhere: byte, short, ushort, long, float, double) for the data
63       types -- note that case is ignored by FlexRaw.  After the type, one
64       integer specifies the number of dimensions of the data `chunk', and
65       subsequent integers the size of each dimension.  So the specifier above
66       (`Float 3 4 600 600') describes our FORTRAN array.  A scalar can be
67       described as `float 0' (or `float 1 1', or `float 2 1 1', etc.).
68
69       When all the dimensions are read -- or a # appears after whitespace --
70       the rest of the current input line is ignored, unless badvalues are
71       being read or written.  In that case, the next token will be the string
72       "badvalue" followed by the bad value used, if needed.
73
74       What about the extra 4 bytes at the head and tail, which we just threw
75       away?  These are added by FORTRAN (at least on Suns, Alphas and Linux),
76       and specify the number of bytes written by each WRITE -- the same
77       number is put at the start and the end of each chunk of data.  You may
78       need to know all this in some cases.  In general, FlexRaw tries to
79       handle it itself, if you simply add a line saying `f77' to the header
80       file, before any data specifiers:
81
82           # FlexRaw file header for F77 form=unformatted
83           F77
84           # Data
85           Float 3
86           4 600 600
87
88       -- the redundancy in FORTRAN data files even allows FlexRaw to
89       automatically deal with files written on other machines which use back-
90       to-front byte ordering.  This won't always work -- it's a 1 in 4
91       billion chance it won't, even if you regularly read 4Gb files!  Also,
92       it currently doesn't work for compressed files, so you can say `swap'
93       (again before any data specifiers) to make certain the byte order is
94       swapped.
95
96       The optional $hdr argument allows the use of an anonymous array to give
97       header information, rather than using a .hdr file.  For example,
98
99           $header = [
100               {Type => 'f77'},
101               {Type => 'float', NDims => 3, Dims => [ 4,600,600 ] }
102           ];
103           @a = readflex('banana',$header);
104
105       reads our example file again.  As a special case, when NDims is 1, Dims
106       may be given as a scalar.
107
108       The highest dimension can be given as "undef", which will read as many
109       frames as possible of the given size (but only if only one hash-ref is
110       given):
111
112         $video = readflex('frames.raw', [
113           { Type=>'byte', NDims=>4, Dims=>[4,640,480,undef] },
114         ]);
115
116       Within PDL, readflex and writeflex can be used to write several pdls to
117       a single file -- e.g.
118
119           use PDL;
120           use PDL::IO::FlexRaw;
121
122           @pdls = ($pdl1, $pdl2, ...);
123           $hdr = writeflex("fname",@pdls);
124           @pdl2 = readflex("fname",$hdr);
125
126           writeflexhdr("fname",$hdr);  # not needed if $PDL::IO::FlexRaw::writeflexhdr is set
127           @pdl3 = readflex("fname");
128
129       -- "writeflex" produces the data file and returns the file header as an
130       anonymous hash, which can be written to a .hdr file using
131       "writeflexhdr".
132
133       If the package variable $PDL::IO::FlexRaw::writeflexhdr is true, and
134       the "writeflex" call was with a filename and not a handle,
135       "writeflexhdr" will be called automatically (as done by "writefraw".
136
137       The reading of compressed data is switched on automatically if the
138       filename requested ends in .gz or .Z, or if the originally specified
139       filename does not exist, but one of these compressed forms does.
140
141       If "writeflex" and "readflex" are given a reference to a file handle as
142       a first parameter instead of a filename, then the data is read or
143       written to the open filehandle.  This gives an easy way to read an
144       arbitrary slice in a big data volume, as in the following example:
145
146           use PDL;
147           use PDL::IO::FastRaw;
148
149           open(DATA, "raw3d.dat");
150           binmode(DATA);
151
152           # assume we know the data size from an external source
153           ($width, $height, $data_size) = (256,256, 4);
154
155           my $slice_num = 64;   # slice to look at
156           # Seek to slice
157           seek(DATA, $width*$height*$data_size * $slice_num, 0);
158           $pdl = readflex \*DATA, [{Dims=>[$width, $height], Type=>'long'}];
159
160       WARNING: In later versions of perl (5.8 and up) you must be sure that
161       your file is in "raw" mode (see the perlfunc man page entry for
162       "binmode", for details).  Both readflex and writeflex automagically
163       switch the file to raw mode for you -- but in code like the snipped
164       above, you could end up seeking the wrong byte if you forget to make
165       the binmode() call.
166
167       "mapflex" memory maps, rather than reads, the data files.  Its
168       interface is similar to "readflex".  Extra options specify if the data
169       is to be loaded `ReadOnly', if the data file is to be `Creat'-ed anew
170       on the basis of the header information or `Trunc'-ated to the length of
171       the data read.  The extra speed of access brings with it some
172       limitations: "mapflex" won't read compressed data, auto-detect f77
173       files, or read f77 files written by more than a single unformatted
174       write statement.  More seriously, data alignment constraints mean that
175       "mapflex" cannot read some files, depending on the requirements of the
176       host OS (it may also vary depending on the setting of the `uac' flag on
177       any given machine).  You may have run into similar problems with common
178       blocks in FORTRAN.
179
180       For instance, floating point numbers may have to align on 4 byte
181       boundaries -- if the data file consists of 3 bytes then a float, it
182       cannot be read.  "mapflex" will warn about this problem when it occurs,
183       and return the PDLs mapped before the problem arose.  This can be dealt
184       with either by reorganizing the data file (large types first helps, as
185       a rule-of-thumb), or more simply by using "readflex".
186

FUNCTIONS

188   glueflex
189       Append a single data item to an existing binary file written by
190       "writeflex".  Must be to the last data item in that file. Error if dims
191       not compatible with existing data.
192
193           $hdr = glueflex($file, $pdl[, $hdr]); # or
194           $hdr = glueflex(FILEHANDLE, $pdl[, $hdr]);
195           # now you must call writeflexhdr()
196           writeflexhdr($file, $hdr);
197
198       or
199
200           $PDL::IO::FlexRaw::writeflexhdr = 1; # set so we don't have to call writeflexhdr
201           $hdr = glueflex($file, $pdl[, $hdr])  # remember, $file must be filename
202           glueflex($file, $pdl[, $hdr])         # remember, $file must be filename
203
204   readflex
205       Read a binary file with flexible format specification
206
207           Usage:
208
209           ($x,$y,...) = readflex("filename" [, $hdr])
210           ($x,$y,...) = readflex(FILEHANDLE [, $hdr])
211
212   mapflex
213       Memory map a binary file with flexible format specification
214
215           Usage:
216
217           ($x,$y,...) = mapflex("filename" [, $hdr] [, $opts])
218
219           All of these options default to false unless set true:
220
221           ReadOnly - Data should be readonly
222           Creat    - Create file if it doesn't exist
223           Trunc    - File should be truncated to a length that conforms
224                      with the header
225
226   writeflex
227       Write a binary file with flexible format specification
228
229           Usage:
230
231           $hdr = writeflex($file, $pdl1, $pdl2,...) # or
232           $hdr = writeflex(FILEHANDLE, $pdl1, $pdl2,...)
233           # now you must call writeflexhdr()
234           writeflexhdr($file, $hdr)
235
236       or
237
238           $PDL::IO::FlexRaw::writeflexhdr = 1;  # set so we don't have to call writeflexhdr
239
240           $hdr = writeflex($file, $pdl1, $pdl2,...)  # remember, $file must be filename
241           writeflex($file, $pdl1, $pdl2,...)         # remember, $file must be filename
242
243   writeflexhdr
244       Write the header file corresponding to a previous writeflex call
245
246           Usage:
247
248           writeflexhdr($file, $hdr)
249
250           $file or "filename" is the filename used in a previous writeflex
251           If $file is actually a "filename" then writeflexhdr() will be
252           called automatically if $PDL::IO::FlexRaw::writeflexhdr is true.
253           If writeflex() was to a FILEHANDLE, you will need to call
254           writeflexhdr() yourself since the filename cannot be determined
255           (at least easily).
256

BAD VALUE SUPPORT

258       As of PDL-2.4.8, PDL::IO::FlexRaw has support for reading and writing
259       pdls with bad values in them.
260
261       On "writeflex", an ndarray argument with "$pdl->badflag == 1" will have
262       the keyword/token "badvalue" added to the header file after the
263       dimension list and an additional token with the bad value for that pdl
264       if "$pdl->badvalue != $pdl->orig_badvalue".
265
266       On "readflex", a pdl with the "badvalue" token in the header will
267       automatically have its badflag set and its badvalue as well if it is
268       not the standard default for that type.
269
270       The new badvalue support required some additions to the header
271       structure.  However, the interface is still being finalized.  For
272       reference the current $hdr looks like this:
273
274           $hdr = {
275                    Type => 'byte',    # data type
276                    NDims => 2,        # number of dimensions
277                    Dims => [640,480], # dims
278                    BadFlag => 1,      # is set/set badflag
279                    BadValue => undef, # undef==default
280                  };
281
282           $badpdl = readflex('badpdl', [$hdr]);
283
284       If you use bad values and try the new PDL::IO::FlexRaw bad value
285       support, please let us know via the perldl mailing list.  Suggestions
286       and feedback are also welcome.
287

BUGS

289       The test on two dimensional byte arrays fail using g77 2.7.2, but not
290       Sun f77.  I hope this isn't my problem!
291
292       Assumes gzip is on the PATH.
293
294       Can't auto-swap compressed files, because it can't seek on them.
295
296       The header format may not agree with that used elsewhere.
297
298       Should it handle handles?
299
300       Mapflex should warn and fallback to reading on SEGV?  Would have to
301       make sure that the data was written back after it was `destroyed'.
302

AUTHOR

304       Copyright (C) Robin Williams <rjrw@ast.leeds.ac.uk> 1997.  All rights
305       reserved. There is no warranty. You are allowed to redistribute this
306       software / documentation under certain conditions. For details, see the
307       file COPYING in the PDL distribution. If this file is separated from
308       the PDL distribution, the copyright notice should be included in the
309       file.
310
311       Documentation contributions copyright (C) David Mertens, 2010.
312
313
314
315perl v5.36.0                      2023-01-20                        FlexRaw(3)
Impressum