1FastRaw(3)            User Contributed Perl Documentation           FastRaw(3)
2
3
4

NAME

6       PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.
7

VERSION

9       This documentation refers to PDL::IO::FastRaw version 0.0.2, I guess.
10

SYNOPSIS

12        use PDL;
13        use PDL::IO::FastRaw;
14
15        writefraw($pdl,"fname");         # write a raw file
16
17        $pdl2 = readfraw("fname");       # read a raw file
18        $pdl2 = PDL->readfraw("fname");
19
20        $pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
21
22        $pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.
23

DESCRIPTION

25       This is a very simple and fast io format for PerlDL.  The disk data
26       consists of two files, a header metadata file in ASCII and a binary
27       file consisting simply of consecutive bytes, shorts or whatever.
28
29       It is hoped that this will not only make for a simple PerlDL module for
30       saving and retrieving these files but also make it easy for other
31       programs to use these files.
32
33       The format of the ASCII header is simply
34
35               <typeid>
36               <ndims>
37               <dim0> <dim1> ...
38
39       You should probably stick with the default header name.  You may want
40       to specify your own header, however, such as when you have a large
41       collection of data files with identical dimensions and data types.
42       Under these circumstances, simply specify the "Header" option in the
43       options hash.
44
45       The binary files are in general NOT interchangeable between different
46       architectures since the binary file is simply dumped from the memory
47       region of the ndarray.  This is what makes the approach efficient.
48
49       It is also possible to mmap the file which can give a large speedup in
50       certain situations as well as save a lot of memory by using a disk file
51       as virtual memory. When a file is mapped, parts of it are read only as
52       they are accessed in the memory (or as the kernel decides: if you are
53       reading the pages in order, it may well preread some for you).
54
55       Note that memory savings and copy-on-write are operating-system
56       dependent - see Core.xs and your operating system documentation for
57       exact semantics of whatever. Basically, if you write to a mmapped file
58       without "ReadOnly", the change will be reflected in the file
59       immediately. "ReadOnly" doesn't really make it impossible to write to
60       the ndarray but maps the memory privately so the file will not be
61       changed when you change the ndarray. Be aware though that mmapping a
62       40Mb file without "ReadOnly" spends no virtual memory but with
63       "ReadOnly" it does reserve 40Mb.
64
65   Example: Converting ASCII to raw
66       You have a whole slew of data files in ASCII from an experiment that
67       you ran in your lab.  You're still tweaking the analysis and plots, so
68       you'd like if your data could load as fast as possible.  Eventually
69       you'll read the data into your scripts using "readfraw", but the first
70       thing you might do is create a script that converts all the data files
71       to raw files:
72
73        #!/usr/bin/perl
74        # Assumes that the data files end with a .asc or .dat extension
75        # and saves the raw file output with a .bdat extension.
76        # call with
77        #  >./convert_to_raw.pl file1.dat file2.dat ...
78        # or
79        #  >./convert_to_raw.pl *.dat
80
81        use PDL;
82        use PDL::IO::FastRaw;  # for saving raw files
83        use PDL::IO::Misc;             # for reading ASCII files with rcols
84        while(shift) {                 # run through the entire supplied list of file names
85                ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
86                print "Saving contents of $_ to $newName\n";
87                $data = rcols($_);
88                writefraw($data, $newName);
89        }
90
91   Example: readfraw
92       Now that you've gotten your data into a raw file format, you can start
93       working on your analysis scripts.  If you scripts used "rcols" in the
94       past, the reading portion of the script should go much, much faster
95       now:
96
97        #!/usr/bin/perl
98        # My plotting script.
99        # Assume I've specified the files to plot on the command line like
100        #  >./plot_script.pl file1.bdat file2.bdat ...
101        # or
102        #  >./plot_script.pl *.bdat
103
104        use PDL;
105        use PDL::IO::FastRaw;
106        while(shift) {                 # run through the entire supplied list of file names
107                $data = readfraw($_);
108                my_plot_func($data);
109        }
110
111   Example: Custom headers
112       In the first example, I allow "writefraw" to use the standard header
113       file name, which would be "file.bdat.hdr".  However, I often measure
114       time series that have identical length, so all of those header files
115       are redundant.  To fix that, I simply pass the Header option to the
116       "writefraw" command.  A modified script would look like this:
117
118        #!/usr/bin/perl
119        # Assumes that the data files end with a .asc or .dat extension
120        # and saves the raw file output with a .bdat extension.
121        # call with
122        #  >./convert_to_raw.pl [-hHeaderFile] <fileglob> [-hHeaderFile] <fileglob> ...
123
124        use PDL;
125        use PDL::IO::FastRaw;  # for saving raw files
126        use PDL::IO::Misc;             # for reading ASCII files with rcols
127        my $header_file = undef;
128        CL_OPTION: while($_ = shift @ARGV) {   # run through the entire list of command-line options
129                if(/-h(.*)/) {
130                        $header_file = $1;
131                        next CL_OPTION;
132                }
133                ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
134                print "Saving contents of $_ to $newName\n";
135                $data = rcols($_);
136                writefraw($data, $newName, {Header => $header_file});
137        }
138
139       Modifying the read script is left as an exercise for the reader.  :]
140
141   Example: Using mapfraw
142       Sometimes you'll want to use "mapfraw" rather than the read/write
143       functions.  In fact, the original author of the module doesn't use the
144       read/write functions anymore, prefering to always use "mapfraw".  How
145       would you go about doing this?
146
147       Assuming you've already saved your data into the raw format, the only
148       change you would have to make to the script in example 2 would be to
149       change the call to "readfraw" to "mapfraw".  That's it.  You will
150       probably see differences in performance, though I (David Mertens)
151       couldn't tell you about them because I haven't played around with
152       "mapfraw" much myself.
153
154       What if you eschew the use of "writefraw" and prefer to only use
155       "mapfraw"?  How would you save your data to a raw format?  In that
156       case, you would have to create a "mapfraw" ndarray with the correct
157       dimensions first using
158
159        $ndarray_on_hd = mapfraw('fname', {Creat => 1, Dims => [dim1, dim2, ...]});
160
161       Note that you must specify the dimensions and you must tell "mapfraw"
162       to create the new ndarray for you by setting the "Creat" option to a
163       true value, not "Create" (note the missing final 'e').
164

FUNCTIONS

166   readfraw
167       Read a raw format binary file
168
169        $pdl2 = readfraw("fname");
170        $pdl2 = PDL->readfraw("fname");
171        $pdl2 = readfraw("fname", {Header => 'headerfname'});
172
173       The "readfraw" command supports the following option:
174
175       Header  Specify the header file name.
176
177   writefraw
178       Write a raw format binary file
179
180        writefraw($pdl,"fname");
181        writefraw($pdl,"fname", {Header => 'headerfname'});
182
183       The "writefraw" command supports the following option:
184
185       Header  Specify the header file name.
186
187   mapfraw
188       Memory map a raw format binary file (see the module docs also)
189
190        $pdl3 = mapfraw("fname2",{ReadOnly => 1});
191
192       The "mapfraw" command supports the following options (not all
193       combinations make sense):
194
195       Dims, Datatype
196               If creating a new file or if you want to specify your own
197               header data for the file, you can give an array reference and a
198               scalar, respectively.
199
200       Creat   Create the file. Also writes out a header for the file.
201
202       Trunc   Set the file size. Automatically enabled with "Creat". NOTE:
203               This also clears the file to all zeroes.
204
205       ReadOnly
206               Disallow writing to the file.
207
208       Header  Specify the header file name.
209
210   maptextfraw
211       Memory map a text file (see the module docs also).
212
213       Note that this function maps the raw format so if you are using an
214       operating system which does strange things to e.g.  line delimiters
215       upon reading a text file, you get the raw (binary) representation.
216
217       The file doesn't really need to be text but it is just mapped as one
218       large binary chunk.
219
220       This function is just a convenience wrapper which firsts "stat"s the
221       file and sets the dimensions and datatype.
222
223        $pdl4 = maptextfraw("fname", {options}
224
225       The options other than Dims, Datatype of "mapfraw" are supported.
226

BUGS

228       Should be documented better. "writefraw" and "readfraw" should also
229       have options (the author nowadays only uses "mapfraw" ;)
230

AUTHOR

232       Copyright (C) Tuomas J. Lukka 1997.  All rights reserved. There is no
233       warranty. You are allowed to redistribute this software / documentation
234       under certain conditions. For details, see the file COPYING in the PDL
235       distribution. If this file is separated from the PDL distribution, the
236       copyright notice should be included in the file.
237
238
239
240perl v5.34.0                      2022-02-28                        FastRaw(3)
Impressum