1dirfile(5)                       DATA FORMATS                       dirfile(5)
2
3
4

NAME

6       dirfile  —  a  filesystem-based database format for time-ordered binary
7       data
8

DESCRIPTION

10       The dirfile database format is designed to provide a fast, simple  for‐
11       mat  for storing and reading binary time-ordered data.  Dirfiles can be
12       read using the GetData Library, which provides  a  reference  implemen‐
13       taiton of these Standards.
14
15       The  dirfile  database  is centred around one or more time-ordered data
16       streams (a time stream).  Each time stream is written to the filesystem
17       in  a  separate  file,  as binary data.  The name of these binary files
18       correspond to the time stream's field name.   Dirfiles  support  binary
19       data  fields  for signed and unsigned integer types of 8 to 64 bits, as
20       well as single and double precision floating-point real or complex data
21       types.
22
23       Two  time  streams may have different constant sampling frequencies and
24       mechanisms exist within the dirfile format to ensure these time streams
25       remain properly sequenced in time.
26
27       To do this, the time streams in the dirfile are subdivided into frames.
28       Each frame contains a fixed integer number  of  samples  of  each  time
29       stream.   Two  time streams in the same dirfile may have different num‐
30       bers of samples per frame, but the number of samples per frame  of  any
31       given time stream is fixed.
32
33       When  synchronous  retrieval  of data from more than one time stream is
34       required, position in the dirfile can be  specified  in  frames,  which
35       will ensure synchronicity.
36
37       The  binary  files  are all located in one ore more filesystem directo‐
38       ries, rooted around a central directory, known as  the  dirfile  direc‐
39       tory.   The dirfile as a whole may be referred to by its dirfile direc‐
40       tory path.
41
42       Included in the dirfile along with the time streams is the dirfile for‐
43       mat specification, which is one or more ASCII text files containing the
44       dirfile database metadata.  The primary file is the file called  format
45       located  in  the dirfile directory.  This file and any additional files
46       that it names, fully specify the dirfile's metadata.  For the syntax of
47       these files, see dirfile-format(5).
48
49       Version  3 of the Dirfile Standards introduced the large dirfile exten‐
50       sion.  This extension added the ability to distribute the dirfile meta‐
51       data  among multiple files (called fragments) in addition to the format
52       file, as well as the ability to house portions of the database in  sub‐
53       dirfiles.  These subdirfiles may be fully fledged dirfiles in their own
54       right, but may also be contained within a larger, parent dirfile.   See
55       dirfile-format(5) for information on specifying these subdirfiles.
56
57       In addition to the raw fields on disk, the dirfile format specification
58       may also specify derived fields which are calculated by performing sim‐
59       ple  element-wise  operations  on  one  or  more input fields.  Derived
60       fields behave identically to raw fields when  read  via  GetData.   See
61       dirfile-format(5) for a complete list of derived field types.  Dirfiles
62       may also contain both numerical and character  string  constant  scalar
63       fields, also further outlined in dirfile-format(5).
64
65       Dirfiles  are  designed  to  be written to and read simultaneously. The
66       dirfile specification dictates that one particular raw field (specified
67       either  explicitly or implicitly by the dirfile metadata) is to be used
68       as the reference field: all other vector fields are assumed to have  at
69       least  as  many  frames  as  the  reference field has, and the size (in
70       frames) of the reference field is used as the size of the dirfile as  a
71       whole.
72
73       Version  6  of  the  Dirfile  Standards added the ability to encode the
74       binary files on disk.  Each fragment may have its own encoding  scheme.
75       Most  commonly,  encodings  are used to compress the data files to same
76       space.  See dirfile-encoding(5) for information on encoding schemes.
77
78
79   Complex Number Storage Format
80       Version 7 of the Dirfile Standards added  support  for  complex  valued
81       data.   Two  types  of complex valued data are supported by the Dirfile
82       Standards:
83
84       ·   A 64-bit complex number consisting of a  IEEE-754  standard  32-bit
85           single  precision  floating point real part and a IEEE-754 standard
86           32-bit single precision floating point imaginary part, and
87
88       ·   A 128-bit complex number consisting of a IEEE-754  standard  64-bit
89           double  precision  floating point real part and a IEEE-754 standard
90           64-bit double precision floating point imaginary part.
91
92       No integer-type complex numbers are supported.
93
94       Unencoded complex numbers are stored on disk in "Fortran  order",  that
95       is with the IEEE-754 real part followed by the IEEE-754 imaginary part.
96       The specified endianness of the two components follows that  of  purely
97       real  floating  point numbers.  Endianness does not affect the ordering
98       of the real and imaginary parts.  This format also conforms to the  C99
99       and C++11 standards.
100
101       To  aid in using complex valued data, dirfile field codes may contain a
102       representation suffix which specifies a function to apply to  the  com‐
103       plex  valued  data  to  map it into purely real data.  See dirfile-for‐
104       mat(5).
105
106

AUTHORS

108       The   Dirfile   format   was   created    by    C.    B.    Netterfield
109       <netterfield@astro.utoronto.ca>.   It  is now maintained by D. V. Wiebe
110       <getdata@ketiltrout.net>.
111
112

SEE ALSO

114       dirfile-encoding(5), dirfile-format(5)
115
116       For an introduction to the GetData  Library  reference  implementation,
117       see  gd_open(3),or gd_getdata(3), or visit the GetData Project website:
118       <http://getdata.sourceforge.net/>.
119
120
121
122Standards Version 10           19 November 2016                     dirfile(5)
Impressum