1MS_INTRO(3) Library Functions Manual MS_INTRO(3)
2
6 ms_intro - Introduction to libmseed
7
10 The Mini-SEED library provides a framework for manipulation of SEED
11 data records including the unpacking and packing of data records.
12 Functionality is also included for managing waveform data as continuous
13 traces. All structures of SEED 2.4 data records are supported with the
14 following exceptions: Blockette 2000 opaque data which has an unknown
15 data structure by definition and Blockette 405 which depends on full
16 SEED (SEED including full ASCII headers) for a full data description.
17 The primary purpose of the library is to hide the details of Mini-SEED
19 in order to allow rapid development of Mini-SEED reading/writing soft‐
20 ware. The framework allows everything from manipulation of Mini-SEED
21 on a record-by-record basis to reading of Mini-SEED into continuous
22 trace segments to packing of large continuous traces using a record
23 template.
24 Certain common tasks have, through library design, been streamlined,
26 for example: reading Mini-SEED records from a file, adding data from
27 unpacked records to a group of traces or packing a group of continuous
28 traces into Mini-SEED records.
29 The following data encoding formats are supported for both unpacking
31 and packing: ASCII, INT16, INT32, FLOAT32, FLOAT64, STEIM1 and STEIM2.
32 The INT and FLOAT encodings each have two versions for quantities with
33 a different number of bits of representation. The STEIM decompression
34 produces 32-bit integers; likewise the compression routines require
35 32-bit integers as input. The following data encoding formats are sup‐
36 ported for unpacking only: GEOSCOPE (24-bit, 16/3 and 16/4 gain
37 ranged), CDSN, SRO and DWWSSN.
38
41 A Mini-SEED record is represented in the library using the data struc‐
42 ture given below. This structure is used for both unpacking and pack‐
43 ing of Mini-SEED records. When unpacking with msr_unpack(3) this
44 structure is populated. When packing with msr_pack(3) this structure
45 is used as a template for the resulting data records and as a source of
46 samples to be packed.
47 Blockettes following the fixed section of the header are contained in
49 the blockette chain of BlktLink structures. Shortcut pointers to com‐
50 monly used blockettes are maintained for types 100, 1000 and 1001.
51 Many common header fields which are not easily accessible/usable in the
53 raw header are available directly from the structure. When this struc‐
54 ture is used as a packing template, these common header fields are
55 packed into the appropriate place in the fixed section or blockette.
56 As examples, the ASCII stream identifiers (network, station, location
57 and channel) are available as NULL terminated strings, the start time
58 is available as a high precision epoch time (see ms_time(3)) and the
59 sample rate is available as a double precision floating point value.
60 The MSRecord data structure:
62 typedef struct MSRecord_s {
64 char *record; /* Mini-SEED record */
65 int32_t reclen; /* Length of Mini-SEED record */
66 /* Pointers to SEED data record structures */
68 struct fsdh_s *fsdh; /* Fixed Section of Data Header */
69 struct BlktLink *blkts; /* Root of blockette chain */
70 struct blkt_100_s *Blkt100; /* Blockette 100, if present */
71 struct blkt_1000_s *Blkt1000; /* Blockette 1000, if present */
72 struct blkt_1001_s *Blkt1001; /* Blockette 1001, if present */
73 /* Common header fields in accessible form */
75 int32_t sequence_number; /* SEED record sequence number */
76 char dataquality; /* Data quality indicator */
77 char network[11]; /* Network designation */
78 char station[11]; /* Station designation */
79 char location[11]; /* Location designation */
80 char channel[11]; /* Channel designation */
81 hptime_t starttime; /* Record start time */
82 double samprate; /* Nominal sample rate (Hz) */
83 int64_t samplecnt; /* Number of samples in record */
84 int8_t encoding; /* Data encoding format */
85 int8_t byteorder; /* Byte order of record */
86 /* Data sample fields */
88 void *datasamples; /* Data samples */
89 int64_t numsamples; /* Number of data samples */
90 char sampletype; /* Sample type code: a, i, f, d */
91 /* Stream oriented state information */
93 StreamState *ststate; /* Stream processing state information */
94 }
95 MSRecord;
96 Explanation of fields
99 record:
100 Pointer to the Mini-SEED record which was unpacked into the
101 MSRecord.
102 reclen:
105 When unpacking this is the record length in bytes of the record
106 pointed to by the 'record' pointer. When packing this is the
107 length of records to pack.
108 fsdh: A pointer to the Fixed Section of the Data Header, all appropri‐
111 ate multi-byte quantities are in host byte order.
112 blkts: The root of the blockette chain. The chain is constructed from
115 linked BlktLink structures. All appropriate multi-byte quanti‐
116 ties in the blockettes are in host byte order. The
117 msr_addblockette(3) routine can be used to add blockettes to
118 this chain. The BlktLink structure and SEED blockette struc‐
119 tures are defined in libmseed.h.
120 Blkt100:
123 Blkt1000:
125 Blkt1001:
127 Shortcut pointers to common blockettes in the blockette chain.
128 If a given blockette does not exist in the blockette chain the
129 shortcut pointer will be 0. If more than one of these blockette
130 types exist in the chain this pointer will point to the last
131 one.
132 sequence_number:
135 SEED record sequence number, should be between 0 and 999999.
136 dataquality:
139 Data record and quality indicator, should be 'D', 'R', 'Q' or
140 'M'.
141 network:
144 station:
146 location:
148 channel:
150 SEED stream identifiers as a NULL terminated strings.
151 starttime:
154 Record start time, the time of the first sample, as a high pre‐
155 cision epoch time (see ms_time(3)). This time can be converted
156 using the various ms_hptime2<X> functions.
157 samprate:
160 The sample rate in samples per second in double precision. Dur‐
161 ing unpacking this value will be set to the sample rate given in
162 the 100 blockette if it is present, otherwise the sample rate
163 derived from the factor and multiplier in the fixed section of
164 the header. In a packing template this value will be used to
165 derive a factor and multiplier for the fixed section of the
166 header and will be written into 100 blockettes if any are in the
167 blockette chain.
168 samplecnt:
171 The sample count, i.e. number of data samples in the record.
172 encoding:
175 The SEED data sample encoding format. During packing this dic‐
176 tates what format will be used to pack the data samples. Sup‐
177 ported packing formats are 0 (DE_ASCII), 1 (DE_INT16), 3
178 (DE_INT32), 4 (DE_FLOAT32), 5 (DE_FLOAT64), 10 (DE_STEIM1) and
179 11 (DE_STEIM2).
180 byteorder:
183 Byte order of multi-byte quantities in the record. A value of 0
184 indicates little endian and a value of 1 indicates big endian.
185 During packing this dictates the byte order of the final
186 records.
187 datasamples:
190 A pointer to the unpacked data samples. If no data samples were
191 unpacked this will be 0. The 'numsamples' field indicates how
192 many samples are in this array and the 'sampletype' field indi‐
193 cates what type of samples they are.
194 numsamples:
197 The number of samples pointed to by the 'datasamples' pointer.
198 sampletype:
201 The type of samples pointed to by the 'datasamples' pointer.
202 Supported types are 'a' (ASCII), 'i' (integer), 'f' (float) and
203 'd' (double). The size of each sample type in bytes is returned
204 by the get_samplesize(3) lookup routine.
205 ststate:
208 Pointer to a StreamState struct used internally to track stream
209 oriented state variables. Memory for this only allocated when
210 needed.
211
214 The library includes two facilities to manage collections of continuous
215 trace segments, each represented by their top most structure: MSTrace‐
216 Group and MSTraceList. The MSTraceList facility is a next generation
217 version of the MSTraceGroup facility. Whereas the MSTraceGroup facil‐
218 ity uses a single linked list of time segments the MSTraceList facility
219 is slightly more complex with two levels of linked lists and common
220 access pointers. The advantages are that the MSTraceList structure is
221 faster to populate, especially when there are many segments (gappy
222 data), and the list is always maintained in a sorted order.
223
226 MSTraceList data structures allow the grouping of MSTraceID structures
227 which are themselves the root of MSTraceSeg structures, see libmseed.h
228 as a reference to these structures.
229
232 MSTraceGroup data structures allow the grouping of MSTrace structures.
233 While a MSTrace structure is normally used to hold trace information
234 and associated data samples it can also be used without data samples as
235 a means to keep trace of data coverage without actual samples.
236 Numerous routines are provided for basic management of MSTrace struc‐
238 tures, including the creation of new MSTrace structures, adding data
239 from Mini-SEED data structures to MSTrace structures, printing trace
240 information, etc.
241 The MSTraceGroup data structure acts as a very simple place to begin a
243 chain of MSTrace structures and keep track of the number of traces.
244 The MSTrace and MSTraceGroup data structures:
246 typedef struct MSTrace_s {
248 char network[11]; /* Network designation */
249 char station[11]; /* Station designation */
250 char location[11]; /* Location designation */
251 char channel[11]; /* Channel designation */
252 char dataquality; /* Data quality indicator */
253 char type; /* MSTrace type code */
254 hptime_t starttime; /* Time of first sample */
255 hptime_t endtime; /* Time of last sample */
256 double samprate; /* Nominal sample rate (Hz) */
257 int64_t samplecnt; /* Num. in trace coverage */
258 void *datasamples; /* Data samples */
259 int64_t numsamples; /* Num. samples in datasamples */
260 char sampletype; /* Sample type code: a, i, f, d */
261 void *prvtptr /* Private pointer for general use */
262 struct MSTrace_s *next; /* Pointer to next trace */
263 }
264 MSTrace;
265 typedef struct MSTraceGroup_s {
267 int32_t numtraces; /* Number of MSTraces in trace chain */
268 struct MSTrace_s *traces; /* Root of the trace chain */
269 }
270 MSTraceGroup;
271 Explanation of fields
274 dataquality:
275 SEED data quality indicator, either 'D', 'R', 'Q' or 'M'. This
277 value will be (binary) 0 when the quality is unknown or mixed.
278 network:
281 station:
283 location:
285 channel:
287 MSTrace identifiers as a NULL terminated strings.
288 type: A single character trace type indicator. This field is not used
291 by libmseed but could be used for application specific trace
292 identification.
293 starttime:
296 MSTrace start time, the time of the first sample, as a high pre‐
297 cision epoch time (see ms_time(3)). This time can be converted
298 using the various ms_hptime2<X> functions.
299 endtime:
302 MSTrace end time, the time of the last sample, as a high preci‐
303 sion epoch time (see ms_time(3)). This time can be converted
304 using the various ms_hptime2<X> functions.
305 samprate:
308 The sample rate in samples per second in double precision.
309 samplecnt:
312 The sample count, i.e. number of data samples in the trace.
313 datasamples:
316 A pointer to the data samples. If no data samples are included
317 this will be 0. The 'numsamples' field indicates how many sam‐
318 ples are in this array and the 'sampletype' field indicates what
319 type of samples they are.
320 numsamples:
323 The number of samples pointed to by the 'datasamples' pointer.
324 sampletype:
327 The type of samples pointed to by the 'datasamples' pointer.
328 Supported types are 'a' (ASCII), 'i' (integer), 'f' (float) and
329 'd' (double). The size of each sample type in bytes is returned
330 by the get_samplesize(3) lookup routine.
331 prvtptr:
334 A private pointer for general use. This pointer is not used by
335 libmseed and can safely be used by the calling program.
336 ststate:
339 Pointer to a StreamState struct used internally to track stream
340 oriented state variables. Memory for this only allocated when
341 needed.
342 next: A pointer to the next MSTrace structure. The value will be 0
345 for the last link in a chain of MSTrace structures.
346
349 All of the log and diagnostic messages emitted by the library functions
350 use the same interface. The output from this interface can be con‐
351 trolled. This is useful when the library will be embedded in a larger
352 system with a custom logging facility. See the man page for more
353 details.
354 ms_log() : the central logging facility. Behavior is controlled by
356 the settings specified with ms_loginit().
357 ms_loginit() : set the functions and prefixes used for log,
359 diagnostic and error messages.
360 The default destination for log messages is standard output (stdout),
362 while all diagnostic (including error) messages go to standard error
363 (stderr). Most of the internal messages emmited by the library are
364 considered diagnostic and will, by default, go to standard error.
365 The default prefix for log and diagnostic messages is nothing. The
367 default prefix for diagnostic error messages is "Error: ".
368 There are reentrant versions of these functions that operate directly
370 on a logging parameter MSLogParam struct. These are intended for use
371 in threaded programs or where a complex logging scheme is desired. See
372 the man pages for more details.
373
377 Waveform data samples are managed by libmseed in a couple of different
378 formats depending on how they are unpacked or will be packed. An array
379 of samples is completely represented by an array of sample values, the
380 number of samples and a sample type. The number of samples is always
381 the actual number of sample values, not the number of bytes needed for
382 storing the values. Samples can be either ASCII, 32-bit integer,
383 32-bit floats or 64-bit double precision floats.
384 Sample types are identified by a single ASCII type character:
386 "a" - ASCII (8 bits)
387 "i" - integer (32 bits)
388 "f" - float (32 bits)
389 "d" - double (64 bits)
390 The size of each sample type in bytes is returned by the get_sample‐
392 size(3) lookup routine.
393
396 The SEED 2.4 standard allows data only SEED (Mini-SEED) to be either in
397 big (most significant byte first) or little (least significant byte
398 first) endian byte order. Unfortunately it is not well defined what
399 little endian Mini-SEED really means. While libmseed supports all four
400 combinations of big and little endian header and data the surest way to
401 avoid compatibility problems is to always create big endian Mini-SEED
402 records (header and data).
403 Reading MiniSEED - how libmseed determines the byte order of a record:
405 The byte order of a record header is determined by checking if the
407 record start year is a sane value (e.g. between 1920 and 2020). The
408 byte order of (compressed) data samples is determined by the byte order
409 flag in the Blockette 1000, if a Blockette 1000 is not present the byte
410 order is assumed to be the same as the header. To force the byte order
411 determination of either the header or data section of a record the fol‐
412 lowing environment variables can be set:
413 UNPACK_HEADER_BYTEORDER
415 UNPACK_DATA_BYTEORDER
416 These variables should be set to either 0 (little endian) or 1 (big
418 endian). A programmatic equivalent of setting these environment vari‐
419 ables is provided via the following macros:
420 MS_UNPACKHEADERBYTEORDER(X)
422 MS_UNPACKDATABYTEORDER(X)
423 Writing MiniSEED - in what byte order libmseed creates records:
425 Normally the byte order of MiniSEED created by libmseed is controlled
427 via a flag in the API. This byte order flag determines the ordering
428 for both the header and data sections of a record. To force the byte
429 order of either the header or data section of a record the following
430 environment variables can be set:
431 PACK_HEADER_BYTEORDER
433 PACK_DATA_BYTEORDER
434 These variables should be set to either 0 (little endian) or 1 (big
436 endian). A programmatic equivalent of setting these environment vari‐
437 ables is provided via the following macros:
438 MS_PACKHEADERBYTEORDER(X)
440 MS_PACKDATABYTEORDER(X)
441 Note that some interpretations of the SEED 2.4 format imply that so-
443 called little endian MiniSEED means that the record header is little
444 endian but that the data section is big endian (as the only defined
445 data encodings must be based on the SEED DDL which, in turn, must be
446 defined in terms of big endian). Libmseed will not create MiniSEED of
447 this flavor by default but can be configured to do so by setting the
448 environment variables described above appropriately.
449
452 Example programs using libmseed are provided in the 'examples' direc‐
453 tory of the source code distribution.
454 One of the most common tasks is to read a file of Mini-SEED records and
456 either perform some action based on the header values or apply some
457 process to the data samples. This task is greatly simplified by using
458 the library functions ms_readmsr(3) and ms_readtraces(3). The
459 ms_readmsr(3) routine will open a specified file and return MSRecord
460 structures for each Mini-SEED record it reads from the file. The
461 ms_readtraces(3) routine will do the same except add all the data read
462 to a MSTraceGroup, this is ideal for quickly reading data for process‐
463 ing. Both of these routines are able to automatically detect record
464 length.
465 If your application is not designed to read Mini-SEED from files the
467 library also provides functions to detect and parse Mini-SEED records
468 in memory buffers. For more information see ms_detect(3) and
469 msr_parse(3).
470 Skeleton code for reading a file with ms_readmsr(3):
472 main() {
474 MSRecord *msr = NULL;
475 int retcode;
476 while ( (retcode = ms_readmsr (&msr, filename, 0, NULL, NULL, 1, 0, verbose)) == MS_NOERROR )
478 {
479 /* Do something with the record here, e.g. print */
480 msr_print (msr, verbose);
481 }
482 if ( retcode != MS_ENDOFFILE )
484 ms_log (2, "Cannot read %s: %s0, filename, ms_errorstr(retcode));
485 /* Cleanup memory and close file */
487 ms_readmsr (&msr, NULL, 0, NULL, NULL, 0, 0);
488 }
489 For reading two files with ms_readtraces(3):
491 main() {
493 MSTraceGroup *mstg = NULL;
494 int retcode;
495 retcode = ms_readtraces (&mstg, filename, 0, -1.0, -1.0, 0, 1, 0, verbose);
497 if ( retcode != MS_ENDOFFILE )
499 ms_log (2, "Cannot read %s: %s0, filename, ms_errorstr(retcode));
500 retcode = ms_readtraces (&mstg, filename2, 0, -1.0, -1.0, 0, 1, 0, verbose);
502 if ( retcode != MS_ENDOFFILE )
504 ms_log (2, "Cannot read %s: %s0, filename2, ms_errorstr(retcode));
505 if ( ! mstg )
507 {
508 fprintf (stderr, "Error reading file\n");
509 return -1;
510 }
511 /* Do something with the traces here, e.g. print */
513 mst_printtracelist (mstg, 0, verbose, 0);
514 mst_freegroup (&mstg);
516 }
517 Another common task is to create (pack) Mini-SEED records. The library
519 supports packing of Mini-SEED either from MSRecord structures, MSTrace
520 structures or MSTraceGroup collections using, respectively,
521 msr_pack(3), mst_pack(3) or mst_packgroup(3). In each case the appro‐
522 priate data structure and parameters are provided to the routine along
523 with a function pointer to a routine that will be called each time a
524 record is complete and should be disposed of.
525 When packing Mini-SEED records the concept of a record header template
527 is used, the template is always in the form of a MSRecord structure.
528 This allows the calling program to dictate the contents, with a few
529 exceptions, of the header in the final data records.
530 Skeleton code for creating (packing) Mini-SEED records with
532 mst_pack(3):
533 static void record_handler (char *record, int reclen, void *srcname) {
535 if ( fwrite(record, reclen, 1, outfile) != 1 )
536 {
537 ms_log (2, "Error writing %s to output file0, (char *)srcname);
538 }
539 }
540 main() {
542 int64_t psamples;
543 int precords;
544 MSTrace *mst;
545 char srcname[50];
546 mst = mst_init (NULL);
548 /* Populate MSTrace values */
550 strcpy (mst->network, "XX");
551 strcpy (mst->station, "TEST");
552 strcpy (mst->channel, "BHE");
553 mst->starttime = ms_seedtimestr2hptime ("2004,350,00:00:00.000000");
554 mst->samprate = 40.0;
555 /* The datasamples pointer and numsamples counter will be adjusted by
557 the packing routine, the datasamples array must be dynamic memory
558 allocated by the malloc() family of routines. */
559 mst->datasamples = dataptr; /* pointer to 32-bit integer data samples */
560 mst->numsamples = 1234;
561 mst->sampletype = 'i'; /* declare type to be 32-bit integers */
562 mst_srcname (mst, srcname, 0);
564 /* Pack 4096 byte, big-endian records, using Steim-2 compression */
566 precords = mst_pack (mst, &record_handler, srcname, 4096, DE_STEIM2,
567 1, &psamples, 1, verbose, NULL);
568 ms_log (0, "Packed %"PRId64" samples into %d records0,
570 psamples, precords);
571 /* Disconnect datasamples pointer, otherwise mst_free() will free it */
573 mst->datasamples = NULL;
574 mst_free (&mst);
576 }
577
580 msr_unpack(3), ms_time(3) and msr_pack(3)
581
584 Chad Trabant
585 IRIS Data Management Center
586 2013/07/17 MS_INTRO(3)