1FastRaw(3) User Contributed Perl Documentation FastRaw(3)
2
6 PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.
7
9 This documentation refers to PDL::IO::FastRaw version 0.0.2, I guess.
10
12 use PDL;
13 use PDL::IO::FastRaw;
14 writefraw($pdl,"fname"); # write a raw file
16 $pdl2 = readfraw("fname"); # read a raw file
18 $pdl2 = PDL->readfraw("fname");
19 $pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
21 $pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.
23
25 This is a very simple and fast io format for PerlDL. The disk data
26 consists of two files, a header metadata file in ASCII and a binary
27 file consisting simply of consecutive bytes, shorts or whatever.
28 It is hoped that this will not only make for a simple PerlDL module for
30 saving and retrieving these files but also make it easy for other
31 programs to use these files.
32 The format of the ASCII header is simply
34 <typeid>
36 <ndims>
37 <dim0> <dim1> ...
38 You should probably stick with the default header name. You may want
40 to specify your own header, however, such as when you have a large
41 collection of data files with identical dimensions and data types.
42 Under these circumstances, simply specify the "Header" option in the
43 options hash.
44 The binary files are in general NOT interchangeable between different
46 architectures since the binary file is simply dumped from the memory
47 region of the piddle. This is what makes the approach efficient.
48 It is also possible to mmap the file which can give a large speedup in
50 certain situations as well as save a lot of memory by using a disk file
51 as virtual memory. When a file is mapped, parts of it are read only as
52 they are accessed in the memory (or as the kernel decides: if you are
53 reading the pages in order, it may well preread some for you).
54 Note that memory savings and copy-on-write are operating-system
56 dependent - see Core.xs and your operating system documentation for
57 exact semantics of whatever. Basically, if you write to a mmapped file
58 without "ReadOnly", the change will be reflected in the file
59 immediately. "ReadOnly" doesn't really make it impossible to write to
60 the piddle but maps the memory privately so the file will not be
61 changed when you change the piddle. Be aware though that mmapping a
62 40Mb file without "ReadOnly" spends no virtual memory but with
63 "ReadOnly" it does reserve 40Mb.
64 Example: Converting ASCII to raw
66 You have a whole slew of data files in ASCII from an experiment that
67 you ran in your lab. You're still tweaking the analysis and plots, so
68 you'd like if your data could load as fast as possible. Eventually
69 you'll read the data into your scripts using "readfraw", but the first
70 thing you might do is create a script that converts all the data files
71 to raw files:
72 #!/usr/bin/perl
74 # Assumes that the data files end with a .asc or .dat extension
75 # and saves the raw file output with a .bdat extension.
76 # call with
77 # >./convert_to_raw.pl file1.dat file2.dat ...
78 # or
79 # >./convert_to_raw.pl *.dat
80 use PDL;
82 use PDL::IO::FastRaw; # for saving raw files
83 use PDL::IO::Misc; # for reading ASCII files with rcols
84 while(shift) { # run through the entire supplied list of file names
85 ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
86 print "Saving contents of $_ to $newName\n";
87 $data = rcols($_);
88 writefraw($data, $newName);
89 }
90 Example: readfraw
92 Now that you've gotten your data into a raw file format, you can start
93 working on your analysis scripts. If you scripts used "rcols" in the
94 past, the reading portion of the script should go much, much faster
95 now:
96 #!/usr/bin/perl
98 # My plotting script.
99 # Assume I've specified the files to plot on the command line like
100 # >./plot_script.pl file1.bdat file2.bdat ...
101 # or
102 # >./plot_script.pl *.bdat
103 use PDL;
105 use PDL::IO::FastRaw;
106 while(shift) { # run through the entire supplied list of file names
107 $data = readfraw($_);
108 my_plot_func($data);
109 }
110 Example: Custom headers
112 In the first example, I allow "writefraw" to use the standard header
113 file name, which would be "file.bdat.hdr". However, I often measure
114 time series that have identical length, so all of those header files
115 are redundant. To fix that, I simply pass the Header option to the
116 "writefraw" command. A modified script would look like this:
117 #!/usr/bin/perl
119 # Assumes that the data files end with a .asc or .dat extension
120 # and saves the raw file output with a .bdat extension.
121 # call with
122 # >./convert_to_raw.pl [-hHeaderFile] <fileglob> [-hHeaderFile] <fileglob> ...
123 use PDL;
125 use PDL::IO::FastRaw; # for saving raw files
126 use PDL::IO::Misc; # for reading ASCII files with rcols
127 my $header_file = undef;
128 CL_OPTION: while($_ = shift @ARGV) { # run through the entire list of command-line options
129 if(/-h(.*)/) {
130 $header_file = $1;
131 next CL_OPTION;
132 }
133 ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
134 print "Saving contents of $_ to $newName\n";
135 $data = rcols($_);
136 writefraw($data, $newName, {Header => $header_file});
137 }
138 Modifying the read script is left as an exercise for the reader. :]
140 Example: Using mapfraw
142 Sometimes you'll want to use "mapfraw" rather than the read/write
143 functions. In fact, the original author of the module doesn't use the
144 read/write functions anymore, prefering to always use "mapfraw". How
145 would you go about doing this?
146 Assuming you've already saved your data into the raw format, the only
148 change you would have to make to the script in example 2 would be to
149 change the call to "readfraw" to "mapfraw". That's it. You will
150 probably see differences in performance, though I (David Mertens)
151 couldn't tell you about them because I haven't played around with
152 "mapfraw" much myself.
153 What if you eschew the use of "writefraw" and prefer to only use
155 "mapfraw"? How would you save your data to a raw format? In that
156 case, you would have to create a "mapfraw" piddle with the correct
157 dimensions first using
158 $piddle_on_hd = mapfraw('fname', {Creat => 1, Dims => [dim1, dim2, ...]});
160 Note that you must specify the dimensions and you must tell "mapfraw"
162 to create the new piddle for you by setting the "Creat" option to a
163 true value, not "Create" (note the missing final 'e').
164
166 readfraw
167 Read a raw format binary file
168 $pdl2 = readfraw("fname");
170 $pdl2 = PDL->readfraw("fname");
171 $pdl2 = readfraw("fname", {Header => 'headerfname'});
172 The "readfraw" command supports the following option:
174 Header Specify the header file name.
176 writefraw
178 Write a raw format binary file
179 writefraw($pdl,"fname");
181 writefraw($pdl,"fname", {Header => 'headerfname'});
182 The "writefraw" command supports the following option:
184 Header Specify the header file name.
186 mapfraw
188 Memory map a raw format binary file (see the module docs also)
189 $pdl3 = mapfraw("fname2",{ReadOnly => 1});
191 The "mapfraw" command supports the following options (not all
193 combinations make sense):
194 Dims, Datatype
196 If creating a new file or if you want to specify your own
197 header data for the file, you can give an array reference and a
198 scalar, respectively.
199 Creat Create the file. Also writes out a header for the file.
201 Trunc Set the file size. Automatically enabled with "Creat". NOTE:
203 This also clears the file to all zeroes.
204 ReadOnly
206 Disallow writing to the file.
207 Header Specify the header file name.
209 maptextfraw
211 Memory map a text file (see the module docs also).
212 Note that this function maps the raw format so if you are using an
214 operating system which does strange things to e.g. line delimiters
215 upon reading a text file, you get the raw (binary) representation.
216 The file doesn't really need to be text but it is just mapped as one
218 large binary chunk.
219 This function is just a convenience wrapper which firsts "stat"s the
221 file and sets the dimensions and datatype.
222 $pdl4 = maptextfraw("fname", {options}
224 The options other than Dims, Datatype of "mapfraw" are supported.
226
228 Should be documented better. "writefraw" and "readfraw" should also
229 have options (the author nowadays only uses "mapfraw" ;)
230
232 Copyright (C) Tuomas J. Lukka 1997. All rights reserved. There is no
233 warranty. You are allowed to redistribute this software / documentation
234 under certain conditions. For details, see the file COPYING in the PDL
235 distribution. If this file is separated from the PDL distribution, the
236 copyright notice should be included in the file.
237perl v5.30.2 2020-04-02 FastRaw(3)