1FastRaw(3)            User Contributed Perl Documentation           FastRaw(3)
2
3
4

NAME

6       PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.
7

SYNOPSIS

9        use PDL;
10        use PDL::IO::FastRaw;
11
12        writefraw($pdl,"fname");         # write a raw file
13
14        $pdl2 = readfraw("fname");       # read a raw file
15        $pdl2 = PDL->readfraw("fname");
16
17        gluefraw($pdlx, "fname");        # append to existing file
18        $pdlx->gluefraw("fname");
19
20        $pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
21
22        $pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.
23

DESCRIPTION

25       This is a very simple and fast io format for PerlDL.  The disk data
26       consists of two files, a header metadata file in ASCII and a binary
27       file consisting simply of consecutive bytes, shorts or whatever.
28
29       It is hoped that this will not only make for a simple PerlDL module for
30       saving and retrieving these files but also make it easy for other
31       programs to use these files.
32
33       The format of the ASCII header is simply
34
35               <typeid>
36               <ndims>
37               <dim0> <dim1> ...
38
39       You should probably stick with the default header name.  You may want
40       to specify your own header, however, such as when you have a large
41       collection of data files with identical dimensions and data types.
42       Under these circumstances, simply specify the "Header" option in the
43       options hash.
44
45       The binary files are in general NOT interchangeable between different
46       architectures since the binary file is simply dumped from the memory
47       region of the ndarray.  This is what makes the approach efficient.
48
49       It is also possible to mmap the file which can give a large speedup in
50       certain situations as well as save a lot of memory by using a disk file
51       as virtual memory. When a file is mapped, parts of it are read only as
52       they are accessed in the memory (or as the kernel decides: if you are
53       reading the pages in order, it may well preread some for you).
54
55       Note that memory savings and copy-on-write are operating-system
56       dependent - see Core.xs and your operating system documentation for
57       exact semantics of whatever. Basically, if you write to a mmapped file
58       without "ReadOnly", the change will be reflected in the file
59       immediately. "ReadOnly" doesn't really make it impossible to write to
60       the ndarray but maps the memory privately so the file will not be
61       changed when you change the ndarray. Be aware though that mmapping a
62       40Mb file without "ReadOnly" spends no virtual memory but with
63       "ReadOnly" it does reserve 40Mb.
64
65   Example: Converting ASCII to raw
66       You have a whole slew of data files in ASCII from an experiment that
67       you ran in your lab.  You're still tweaking the analysis and plots, so
68       you'd like if your data could load as fast as possible.  Eventually
69       you'll read the data into your scripts using "readfraw", but the first
70       thing you might do is create a script that converts all the data files
71       to raw files:
72
73        #!/usr/bin/perl
74        # Assumes that the data files end with a .asc or .dat extension
75        # and saves the raw file output with a .bdat extension.
76        # call with
77        #  >./convert_to_raw.pl file1.dat file2.dat ...
78        # or
79        #  >./convert_to_raw.pl *.dat
80
81        use PDL;
82        use PDL::IO::FastRaw;  # for saving raw files
83        use PDL::IO::Misc;             # for reading ASCII files with rcols
84        while(shift) {                 # run through the entire supplied list of file names
85                ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
86                print "Saving contents of $_ to $newName\n";
87                $data = rcols($_);
88                writefraw($data, $newName);
89        }
90
91   Example: readfraw
92       Now that you've gotten your data into a raw file format, you can start
93       working on your analysis scripts.  If you scripts used "rcols" in the
94       past, the reading portion of the script should go much, much faster
95       now:
96
97        #!/usr/bin/perl
98        # My plotting script.
99        # Assume I've specified the files to plot on the command line like
100        #  >./plot_script.pl file1.bdat file2.bdat ...
101        # or
102        #  >./plot_script.pl *.bdat
103
104        use PDL;
105        use PDL::IO::FastRaw;
106        while(shift) {                 # run through the entire supplied list of file names
107                $data = readfraw($_);
108                my_plot_func($data);
109        }
110
111   Example: Custom headers
112       In the first example, I allow "writefraw" to use the standard header
113       file name, which would be "file.bdat.hdr".  However, I often measure
114       time series that have identical length, so all of those header files
115       are redundant.  To fix that, I simply pass the Header option to the
116       "writefraw" command.  A modified script would look like this:
117
118        #!/usr/bin/perl
119        # Assumes that the data files end with a .asc or .dat extension
120        # and saves the raw file output with a .bdat extension.
121        # call with
122        #  >./convert_to_raw.pl [-hHeaderFile] <fileglob> [-hHeaderFile] <fileglob> ...
123
124        use PDL;
125        use PDL::IO::FastRaw;  # for saving raw files
126        use PDL::IO::Misc;             # for reading ASCII files with rcols
127        my $header_file = undef;
128        CL_OPTION: while($_ = shift @ARGV) {   # run through the entire list of command-line options
129                if(/-h(.*)/) {
130                        $header_file = $1;
131                        next CL_OPTION;
132                }
133                ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
134                print "Saving contents of $_ to $newName\n";
135                $data = rcols($_);
136                writefraw($data, $newName, {Header => $header_file});
137        }
138
139       Modifying the read script is left as an exercise for the reader.  :]
140
141   Example: Using mapfraw
142       Sometimes you'll want to use "mapfraw" rather than the read/write
143       functions.  In fact, the original author of the module doesn't use the
144       read/write functions anymore, prefering to always use "mapfraw".  How
145       would you go about doing this?
146
147       Assuming you've already saved your data into the raw format, the only
148       change you would have to make to the script in example 2 would be to
149       change the call to "readfraw" to "mapfraw".  That's it.  You will
150       probably see differences in performance, though I (David Mertens)
151       couldn't tell you about them because I haven't played around with
152       "mapfraw" much myself.
153
154       What if you eschew the use of "writefraw" and prefer to only use
155       "mapfraw"?  How would you save your data to a raw format?  In that
156       case, you would have to create a "mapfraw" ndarray with the correct
157       dimensions first using
158
159        $ndarray_on_hd = mapfraw('fname', {Creat => 1, Dims => [dim1, dim2, ...]});
160
161       Note that you must specify the dimensions and you must tell "mapfraw"
162       to create the new ndarray for you by setting the "Creat" option to a
163       true value, not "Create" (note the missing final 'e').
164

FUNCTIONS

166   writefraw
167       Write a raw format binary file
168
169        writefraw($pdl,"fname");
170        writefraw($pdl,"fname", {Header => 'headerfname'});
171
172       The "writefraw" command supports the following option:
173
174       Header  Specify the header file name.
175
176   readfraw
177       Read a raw format binary file
178
179        $pdl2 = readfraw("fname");
180        $pdl2 = PDL->readfraw("fname");
181        $pdl2 = readfraw("fname", {Header => 'headerfname'});
182
183       The "readfraw" command supports the following option:
184
185       Header  Specify the header file name.
186
187   gluefraw
188       Append a single data item to an existing binary file written by
189       "writefraw". Error if dims not compatible with existing data.
190
191         gluefraw($file, $pdl[, $opts]);
192
193   mapfraw
194       Memory map a raw format binary file (see the module docs also)
195
196        $pdl3 = mapfraw("fname2",{ReadOnly => 1});
197
198       The "mapfraw" command supports the following options (not all
199       combinations make sense):
200
201       Dims, Datatype
202               If creating a new file or if you want to specify your own
203               header data for the file, you can give an array reference and a
204               scalar, respectively.
205
206       Creat   Create the file. Also writes out a header for the file.
207
208       Trunc   Set the file size. Automatically enabled with "Creat". NOTE:
209               This also clears the file to all zeroes.
210
211       ReadOnly
212               Disallow writing to the file.
213
214       Header  Specify the header file name.
215
216   maptextfraw
217       Memory map a text file (see the module docs also).
218
219       Note that this function maps the raw format so if you are using an
220       operating system which does strange things to e.g.  line delimiters
221       upon reading a text file, you get the raw (binary) representation.
222
223       The file doesn't really need to be text but it is just mapped as one
224       large binary chunk.
225
226       This function is just a convenience wrapper which firsts "stat"s the
227       file and sets the dimensions and datatype.
228
229        $pdl4 = maptextfraw("fname", {options}
230
231       The options other than Dims, Datatype of "mapfraw" are supported.
232

BUGS

234       Should be documented better. "writefraw" and "readfraw" should also
235       have options (the author nowadays only uses "mapfraw" ;)
236

AUTHOR

238       Copyright (C) Tuomas J. Lukka 1997.  All rights reserved. There is no
239       warranty. You are allowed to redistribute this software / documentation
240       under certain conditions. For details, see the file COPYING in the PDL
241       distribution. If this file is separated from the PDL distribution, the
242       copyright notice should be included in the file.
243
244
245
246perl v5.36.0                      2023-01-20                        FastRaw(3)
Impressum