1dirfile(5) DATA FORMATS dirfile(5)
2
6 dirfile — a filesystem-based database format for time-ordered binary
7 data
8
10 The dirfile database format is designed to provide a fast, simple for‐
11 mat for storing and reading binary time-ordered data. Dirfiles can be
12 read using the GetData Library, which provides a reference implemen‐
13 taiton of these Standards.
14 The dirfile database is centred around one or more time-ordered data
16 streams (a time stream). Each time stream is written to the filesystem
17 in a separate file, as binary data. The name of these binary files
18 correspond to the time stream's field name. Dirfiles support binary
19 data fields for signed and unsigned integer types of 8 to 64 bits, as
20 well as single and double precision floating-point real or complex data
21 types.
22 Two time streams may have different constant sampling frequencies and
24 mechanisms exist within the dirfile format to ensure these time streams
25 remain properly sequenced in time.
26 To do this, the time streams in the dirfile are subdivided into frames.
28 Each frame contains a fixed integer number of samples of each time
29 stream. Two time streams in the same dirfile may have different num‐
30 bers of samples per frame, but the number of samples per frame of any
31 given time stream is fixed.
32 When synchronous retrieval of data from more than one time stream is
34 required, position in the dirfile can be specified in frames, which
35 will ensure synchronicity.
36 The binary files are all located in one ore more filesystem directo‐
38 ries, rooted around a central directory, known as the dirfile direc‐
39 tory. The dirfile as a whole may be referred to by its dirfile direc‐
40 tory path.
41 Included in the dirfile along with the time streams is the dirfile for‐
43 mat specification, which is one or more ASCII text files containing the
44 dirfile database metadata. The primary file is the file called format
45 located in the dirfile directory. This file and any additional files
46 that it names, fully specify the dirfile's metadata. For the syntax of
47 these files, see dirfile-format(5).
48 Version 3 of the Dirfile Standards introduced the large dirfile exten‐
50 sion. This extension added the ability to distribute the dirfile meta‐
51 data among multiple files (called fragments) in addition to the format
52 file, as well as the ability to house portions of the database in sub‐
53 dirfiles. These subdirfiles may be fully fledged dirfiles in their own
54 right, but may also be contained within a larger, parent dirfile. See
55 dirfile-format(5) for information on specifying these subdirfiles.
56 In addition to the raw fields on disk, the dirfile format specification
58 may also specify derived fields which are calculated by performing sim‐
59 ple element-wise operations on one or more input fields. Derived
60 fields behave identically to raw fields when read via GetData. See
61 dirfile-format(5) for a complete list of derived field types. Dirfiles
62 may also contain both numerical and character string constant scalar
63 fields, also further outlined in dirfile-format(5).
64 Dirfiles are designed to be written to and read simultaneously. The
66 dirfile specification dictates that one particular raw field (specified
67 either explicitly or implicitly by the dirfile metadata) is to be used
68 as the reference field: all other vector fields are assumed to have at
69 least as many frames as the reference field has, and the size (in
70 frames) of the reference field is used as the size of the dirfile as a
71 whole.
72 Version 6 of the Dirfile Standards added the ability to encode the
74 binary files on disk. Each fragment may have its own encoding scheme.
75 Most commonly, encodings are used to compress the data files to same
76 space. See dirfile-encoding(5) for information on encoding schemes.
77 Complex Number Storage Format
80 Version 7 of the Dirfile Standards added support for complex valued
81 data. Two types of complex valued data are supported by the Dirfile
82 Standards:
83 · A 64-bit complex number consisting of a IEEE-754 standard 32-bit
85 single precision floating point real part and a IEEE-754 standard
86 32-bit single precision floating point imaginary part, and
87 · A 128-bit complex number consisting of a IEEE-754 standard 64-bit
89 double precision floating point real part and a IEEE-754 standard
90 64-bit double precision floating point imaginary part.
91 No integer-type complex numbers are supported.
93 Unencoded complex numbers are stored on disk in "Fortran order", that
95 is with the IEEE-754 real part followed by the IEEE-754 imaginary part.
96 The specified endianness of the two components follows that of purely
97 real floating point numbers. Endianness does not affect the ordering
98 of the real and imaginary parts. This format also conforms to the C99
99 and C++11 standards.
100 To aid in using complex valued data, dirfile field codes may contain a
102 representation suffix which specifies a function to apply to the com‐
103 plex valued data to map it into purely real data. See dirfile-for‐
104 mat(5).
105
108 The Dirfile format was created by C. B. Netterfield
109 <netterfield@astro.utoronto.ca>. It is now maintained by D. V. Wiebe
110 <getdata@ketiltrout.net>.
111
114 dirfile-encoding(5), dirfile-format(5)
115 For an introduction to the GetData Library reference implementation,
117 see gd_open(3),or gd_getdata(3), or visit the GetData Project website:
118 <http://getdata.sourceforge.net/>.
119Standards Version 10 19 November 2016 dirfile(5)