1FastRaw(3) User Contributed Perl Documentation FastRaw(3)
2
6 PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.
7
9 use PDL;
10 use PDL::IO::FastRaw;
11 writefraw($pdl,"fname"); # write a raw file
13 $pdl2 = readfraw("fname"); # read a raw file
15 $pdl2 = PDL->readfraw("fname");
16 gluefraw($pdlx, "fname"); # append to existing file
18 $pdlx->gluefraw("fname");
19 $pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
21 $pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.
23
25 This is a very simple and fast io format for PerlDL. The disk data
26 consists of two files, a header metadata file in ASCII and a binary
27 file consisting simply of consecutive bytes, shorts or whatever.
28 It is hoped that this will not only make for a simple PerlDL module for
30 saving and retrieving these files but also make it easy for other
31 programs to use these files.
32 The format of the ASCII header is simply
34 <typeid>
36 <ndims>
37 <dim0> <dim1> ...
38 You should probably stick with the default header name. You may want
40 to specify your own header, however, such as when you have a large
41 collection of data files with identical dimensions and data types.
42 Under these circumstances, simply specify the "Header" option in the
43 options hash.
44 The binary files are in general NOT interchangeable between different
46 architectures since the binary file is simply dumped from the memory
47 region of the ndarray. This is what makes the approach efficient.
48 It is also possible to mmap the file which can give a large speedup in
50 certain situations as well as save a lot of memory by using a disk file
51 as virtual memory. When a file is mapped, parts of it are read only as
52 they are accessed in the memory (or as the kernel decides: if you are
53 reading the pages in order, it may well preread some for you).
54 Note that memory savings and copy-on-write are operating-system
56 dependent - see Core.xs and your operating system documentation for
57 exact semantics of whatever. Basically, if you write to a mmapped file
58 without "ReadOnly", the change will be reflected in the file
59 immediately. "ReadOnly" doesn't really make it impossible to write to
60 the ndarray but maps the memory privately so the file will not be
61 changed when you change the ndarray. Be aware though that mmapping a
62 40Mb file without "ReadOnly" spends no virtual memory but with
63 "ReadOnly" it does reserve 40Mb.
64 Example: Converting ASCII to raw
66 You have a whole slew of data files in ASCII from an experiment that
67 you ran in your lab. You're still tweaking the analysis and plots, so
68 you'd like if your data could load as fast as possible. Eventually
69 you'll read the data into your scripts using "readfraw", but the first
70 thing you might do is create a script that converts all the data files
71 to raw files:
72 #!/usr/bin/perl
74 # Assumes that the data files end with a .asc or .dat extension
75 # and saves the raw file output with a .bdat extension.
76 # call with
77 # >./convert_to_raw.pl file1.dat file2.dat ...
78 # or
79 # >./convert_to_raw.pl *.dat
80 use PDL;
82 use PDL::IO::FastRaw; # for saving raw files
83 use PDL::IO::Misc; # for reading ASCII files with rcols
84 while(shift) { # run through the entire supplied list of file names
85 ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
86 print "Saving contents of $_ to $newName\n";
87 $data = rcols($_);
88 writefraw($data, $newName);
89 }
90 Example: readfraw
92 Now that you've gotten your data into a raw file format, you can start
93 working on your analysis scripts. If you scripts used "rcols" in the
94 past, the reading portion of the script should go much, much faster
95 now:
96 #!/usr/bin/perl
98 # My plotting script.
99 # Assume I've specified the files to plot on the command line like
100 # >./plot_script.pl file1.bdat file2.bdat ...
101 # or
102 # >./plot_script.pl *.bdat
103 use PDL;
105 use PDL::IO::FastRaw;
106 while(shift) { # run through the entire supplied list of file names
107 $data = readfraw($_);
108 my_plot_func($data);
109 }
110 Example: Custom headers
112 In the first example, I allow "writefraw" to use the standard header
113 file name, which would be "file.bdat.hdr". However, I often measure
114 time series that have identical length, so all of those header files
115 are redundant. To fix that, I simply pass the Header option to the
116 "writefraw" command. A modified script would look like this:
117 #!/usr/bin/perl
119 # Assumes that the data files end with a .asc or .dat extension
120 # and saves the raw file output with a .bdat extension.
121 # call with
122 # >./convert_to_raw.pl [-hHeaderFile] <fileglob> [-hHeaderFile] <fileglob> ...
123 use PDL;
125 use PDL::IO::FastRaw; # for saving raw files
126 use PDL::IO::Misc; # for reading ASCII files with rcols
127 my $header_file = undef;
128 CL_OPTION: while($_ = shift @ARGV) { # run through the entire list of command-line options
129 if(/-h(.*)/) {
130 $header_file = $1;
131 next CL_OPTION;
132 }
133 ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
134 print "Saving contents of $_ to $newName\n";
135 $data = rcols($_);
136 writefraw($data, $newName, {Header => $header_file});
137 }
138 Modifying the read script is left as an exercise for the reader. :]
140 Example: Using mapfraw
142 Sometimes you'll want to use "mapfraw" rather than the read/write
143 functions. In fact, the original author of the module doesn't use the
144 read/write functions anymore, prefering to always use "mapfraw". How
145 would you go about doing this?
146 Assuming you've already saved your data into the raw format, the only
148 change you would have to make to the script in example 2 would be to
149 change the call to "readfraw" to "mapfraw". That's it. You will
150 probably see differences in performance, though I (David Mertens)
151 couldn't tell you about them because I haven't played around with
152 "mapfraw" much myself.
153 What if you eschew the use of "writefraw" and prefer to only use
155 "mapfraw"? How would you save your data to a raw format? In that
156 case, you would have to create a "mapfraw" ndarray with the correct
157 dimensions first using
158 $ndarray_on_hd = mapfraw('fname', {Creat => 1, Dims => [dim1, dim2, ...]});
160 Note that you must specify the dimensions and you must tell "mapfraw"
162 to create the new ndarray for you by setting the "Creat" option to a
163 true value, not "Create" (note the missing final 'e').
164
166 writefraw
167 Write a raw format binary file
168 writefraw($pdl,"fname");
170 writefraw($pdl,"fname", {Header => 'headerfname'});
171 The "writefraw" command supports the following option:
173 Header Specify the header file name.
175 readfraw
177 Read a raw format binary file
178 $pdl2 = readfraw("fname");
180 $pdl2 = PDL->readfraw("fname");
181 $pdl2 = readfraw("fname", {Header => 'headerfname'});
182 The "readfraw" command supports the following option:
184 Header Specify the header file name.
186 gluefraw
188 Append a single data item to an existing binary file written by
189 "writefraw". Error if dims not compatible with existing data.
190 gluefraw($file, $pdl[, $opts]);
192 mapfraw
194 Memory map a raw format binary file (see the module docs also)
195 $pdl3 = mapfraw("fname2",{ReadOnly => 1});
197 The "mapfraw" command supports the following options (not all
199 combinations make sense):
200 Dims, Datatype
202 If creating a new file or if you want to specify your own
203 header data for the file, you can give an array reference and a
204 scalar, respectively.
205 Creat Create the file. Also writes out a header for the file.
207 Trunc Set the file size. Automatically enabled with "Creat". NOTE:
209 This also clears the file to all zeroes.
210 ReadOnly
212 Disallow writing to the file.
213 Header Specify the header file name.
215 maptextfraw
217 Memory map a text file (see the module docs also).
218 Note that this function maps the raw format so if you are using an
220 operating system which does strange things to e.g. line delimiters
221 upon reading a text file, you get the raw (binary) representation.
222 The file doesn't really need to be text but it is just mapped as one
224 large binary chunk.
225 This function is just a convenience wrapper which firsts "stat"s the
227 file and sets the dimensions and datatype.
228 $pdl4 = maptextfraw("fname", {options}
230 The options other than Dims, Datatype of "mapfraw" are supported.
232
234 Should be documented better. "writefraw" and "readfraw" should also
235 have options (the author nowadays only uses "mapfraw" ;)
236
238 Copyright (C) Tuomas J. Lukka 1997. All rights reserved. There is no
239 warranty. You are allowed to redistribute this software / documentation
240 under certain conditions. For details, see the file COPYING in the PDL
241 distribution. If this file is separated from the PDL distribution, the
242 copyright notice should be included in the file.
243perl v5.36.0 2023-01-20 FastRaw(3)