1MGD77SNIFFER(1) Generic Mapping Tools MGD77SNIFFER(1)
2
6 mgd77sniffer - Scan for errors in MGD77 data
7
9 mgd77sniffer NGDC-ids [ -Afieldabbrev,scale,offset ] [ -Cmaxspd ] [
10 -Dd|e|E|f|l|m|s|v[r] ] [ -gfieldabbrev,imggrid,scale,mode ] [ -Gfield‐
11 abbrev,grid ] [ -H[i][nrec] ] [ -Ifieldabbrev,rec1,recN ] [ -K ] [
12 -Lcustom limits file ] [ -N ] [ -Q[b|c|l|n][[/]threshold] ] [
13 -Rwest/east/south/north[r] ] [ -Sd|s|t ] [ -Tgap ] [ -V ] [
14 -Wc|g|o|s|t|v|x ] [ -bo[s|S|d|D[ncol]|c[var1/...]] ]
15
17 mgd77sniffer scans old (pre-Y2K) and new format ASCII MGD77 files for
18 errors using point-by-point sanity checking, along-track detection of
19 excessive slopes, and optional comparison of cruise data with global
20 gravity and predicted bathymetry grids. Detected data problems are out‐
21 put by default as verbose descriptions of each detected error, often
22 resulting in multiple messages per scanned record. Data problems are
23 optionally output (-De option) using a computer-parseable format (see
24 E77 ERROR FORMAT description below). Default error thresholds are
25 derived from histograms of all MGD77 geophysical data collected between
26 1952 and January, 2006. Thresholds are adjustable with the -L option.
27 NGDC-ids
29 Can be one or more of five kinds of specifiers:
30 1) 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
31 2) 2-character <agency> codes which will return all cruises from
32 each agency.
33 3) 4-character <agency><vessel> codes, which will return all
34 cruises from those vessels.
35 4) =<list>, where <list> is a table with NGDC IDs, one per line.
36 5) If nothing is specified we return all cruises in the data
37 base.
38 (See mgd77info -L for agency and vessel codes). The ".mgd77" or
39 ".nc" extensions will automatically be appended, if needed (use
40 -I to ignore certain file types). Cruise files will be looked
41 for first in the current directory and second in all directories
42 listed in $MGD77_HOME/mgd77_paths.txt [If $MGD77_HOME is not set
43 it will default to $GMT_SHAREDIR/mgd77].
44
46 The mgd77sniffer links with Generic Mapping Tools 4.0 or later along
47 with the supplemental GMT packages x2sys and mgd77. See
48 http://gmt.soest.hawaii.edu for GMT details. Grids for comparison with
49 cruise data may be downloaded via the web.
50
52 -A Apply scale factor and DC adjustment to specified data field.
53 Allows adjustment of cruise data prior to along-track analysis.
54 CAUTION: data must be thoroughly examined before applying these
55 global data adjustments. May not be used for multiple cruises.
56 -C Set maximum ship speed in m/s, or knots with -N option. Ship
58 speeds exceeding 10 m/s (~20 knots) are flagged as excessive by
59 default.
60 -D Suppress default warning output and only dump cruise data row-
62 by-row such as values, gradients, grid-cruise differences, E77
63 error summaries for each record, re-created MGD77 records or
64 sniffer limits. Append r to include all records (default omits
65 records where navigation errors were detected).
66 -Dd output differences between cruise and grid data. Requires -G
68 option. Output columns include:
69 lat lon dist cruiseZ gridZ diff [cruiseZ2 gridZ2 diff2 ...]
71 Note: grid values are subtracted from cruise data so a positive
73 difference implies cruise > grid. For multiple grid comparison,
74 cruiseZ gridZ diff are repeated for each grid comparison in com‐
75 mand line order.
76 -De output E77 error classification format. Error output is
78 divided into (1) a header containing information globally appli‐
79 cable to the cruise and (2) individual error records summarizing
80 all errors encountered in each cruise record. mgd77sniffer
81 writes E77 directly to <ngdc_id.e77> file handle. See E77 ERROR
82 FORMAT below for additional details.
83 -DE Same as -De but no regression tests will be carried out.
85 -Df output delta Z (change in geophysical field) column and
87 delta S (change in distance) for each geophysical field. Dis‐
88 tance between observations often differ for different fields
89 depending on instrument sampling rate, so ds is included for
90 each geophysical observation. Output columns include:
91 d[twt] ds d[depth] ds d[mtf1] ds d[mtf2] ds d[mag] ds d[diur] ds
93 d[msd] ds d[gobs] ds d[eot] ds d[faa] ds
94 -Dl display mgd77sniffer limits. Customize this output to create
96 a custom limits file for the -L option. No additional arguments
97 are required. Output columns include:
98 fieldabbrev min max maxSlope maxArea
100 -Dm output MGD77 format records in Y2K-compliant MGD77 format
102 -Dn output distance to coast for each record. Requires -Gnav or
104 -gnav option. Output columns include:
105 lat lon dist distToCoast
107 -Ds output calculated gradients for speed and geophysical
109 fields. Gradients correspond to the gradient type selected in
110 the -S option (spatial derivatives by default). Output columns
111 include:
112 speed d[twt] d[depth] d[mtf1] d[mtf2] d[mag] d[diur] d[msd]
114 d[gobs] d[eot] d[faa]
115 See MGD77 FIELD INFO below for field and abbreviations descrip‐
117 tions.
118 -Dv display values for the twelve position and geophysical
120 fields for each MGD77 data record (in this order):
121 lat lon twt depth mtf1 mtf2 mag diur msens gobs eot faa
123 See below for MGD77 FIELD INFO.
125 -g Compare cruise data to the specified grid in Sandwell/Smith Mer‐
127 cator format. Requires a valid MGD77 field abbreviation (see
128 MGD77 FIELD INFO below) followed by a comma, the path (if not in
129 current directory) and grid filename, a scale to multiply the
130 data (1 or 0.1), and mode which stand for the following: (0) Img
131 files with no constraint code, returns data at all points, (1)
132 Img file with constraints coded, return data at all points, (2)
133 Img file with constraints coded, return data only at constrained
134 points and NaN elsewhere, and (3) Img file with constraints
135 coded, return 1 at constraints and 0 elsewhere.
136 -G Compare cruise data to the specified grid. Requires a valid
138 MGD77 field abbreviation (see MGD77 FIELD INFO below) followed
139 by a comma, then the path (if not in current directory) and grid
140 filename. Multiple grid comparison is supported by using sepa‐
141 rate -g or -G calls for each grid. See GRID FILE INFO below.
142 Grid comparison activates several additional error checks. (1)
144 Re-weighted Least Squares Regression of ship versus grid data
145 determines slope and DC shift, which when differing from
146 expected 1 and 0, respectively, may indicate incorrectly scaled
147 ship data, including incorrect units or instrument drift as well
148 as erroneous gravity tie-in. (2) Accumulated ship grid offsets
149 are computed along-track and excessive offsets are flagged
150 according to maxArea threshold (use -L option to adjust
151 maxArea). Warning: predicted bathymetry grids are constrained by
152 cruise data so grids and cruise data are not always independent.
153 Comparison of cruise bathymetry with predicted bathymetry grids
154 also activates a "navigation crossing over land" check.
155 -H (with -G|g only) disable (or force) decimation during RLS analy‐
158 sis of ship and gridded data. By default mgd77sniffer analyses
159 both the full and decimated data sets then reports RLS statis‐
160 tics for the higher correlation regression.
161 -Hb analyze both (default), report better of two.
163 -Hd to disable data decimation (equivalent to -H with no argu‐
165 ment).
166 -Hf to force data decimation.
168 -I Append a field abbreviation and the first and last record in a
170 range of records that should be flagged as bad (and set to NaN
171 prior to the analysis). Repeat as many times as needed. May not
172 be used for multiple cruises.
173 -K Reverse navigation quality flags (good to bad and vice versa).
175 May be necessary when a majority of navigation fixes are erro‐
176 neously flagged bad, which can happen when a cruise's first nav‐
177 igation fix is extremely erroneous. Caution! This will affect
178 sniffer output and should only be attempted after careful manual
179 navigation review.
180 -L Override mgd77sniffer default error detection limits. Supply
182 path and filename to the custom limits file. Rows not beginning
183 with a valid MGD77 field abbreviation are ignored. Field abbre‐
184 viations are listed below in exact form under MGD77 FIELD INFO.
185 Multiple field limits may be modified using one default file,
186 one field per line. Field min, max, max slope and max area may
187 be changed for each field. Max slope pertains to the gradient
188 type selected using the -S option. Max area is used by the -G
189 option as the threshold for flagging excessive offsets from the
190 specified grid. Dump defaults -Dl to view syntax or to quickly
191 create an editable custom limits file.
192 Example custom default file contents (see below for units):
194 # abbrev min max maxSlope maxArea
195 twt 0 15 1 0
196 depth 0 11000 500 5000
197 mag -800 800 - -
198 faa -300 300 100 2500
199 Use a dash '-' to retain a default limit. Hint: to test your
201 custom limits, try: mgd77sniffer -Dl -L<yourlimitsfile>
202 -N Use nautical units.
204 -P Flag regression statistics that are outside the specified confi‐
206 dence level. (i.e., -P5 flags coefficients m, b, rms, and r
207 that fall outside 95%.)
208 -Q Quick mode, use bilinear rather than bicubic interpolation
210 [Default]. Alternatively, select the interpolation mode by
211 adding b for B-spline smoothing, c for bicubic interpolation, l
212 for bilinear interpolation or n for nearest-neighbor value.
213 Optionally, append threshold in the range [0,1]. This parameter
214 controls how close to nodes with NaN values the interpolation
215 will go. E.g., a threshold of 0.5 will interpolate about half
216 way from a non-NaN to a NaN node, whereas 0.1 will go about 90%
217 of the way, etc. [Default is 1, which means none of the (4 or
218 16) nearby nodes may be NaN]. -Q0 will just return the value of
219 the nearest node instead of interpolating. This is the same as
220 using -Qn.
221 -R west, east, south, and north specify the Region of interest, and
223 you may specify them in decimal degrees or in
224 [+-]dd:mm[:ss.xxx][W|E|S|N] format. Append r if lower left and
225 upper right map coordinates are given instead of w/e/s/n. The
226 two shorthands -Rg and -Rd stand for global domain (0/360 and
227 -180/+180 in longitude respectively, with -90/+90 in latitude).
228 Alternatively, specify the name of an existing grid file and the
229 -R settings (and grid spacing, if applicable) are copied from
230 the grid.
231 -S Specify gradient type for along-track excessive slope checking.
233 -Sd Calculate change in z values along track (dz). Output is
234 given in geophysical units, e.g. mGal.
235 -Ss Calculate spatial gradients (dz/ds). Output is given in geo‐
236 physical units per km along the survey track, e.g. mGal/km.
237 -St Calculate time gradients (dz/dt) [default]. Output is given
238 in geophysical units per second along the survey track, e.g.
239 mGal/sec.
240 -T Adjusts mgd77sniffer gap handling. By default, data gaps greater
242 than 5 km are skipped. Set to zero to de-activate gap skipping.
243 -W Print out only certain warning types for verbose error messages.
245 Comma delimit any combination of c|g|o|s|t|v|x: where (c) type
246 code warnings, (g)radient out of range, (o)ffsets from grid
247 (requires -G|g), (s)peed out of range, (t)ime warnings, (v)alue
248 out of range, (x) warning summaries. By default ALL warning mes‐
249 sages are printed.Not compatible with any -D options.
250 -V Selects verbose mode, which will send progress reports to stderr
252 [Default runs "silently"].
253 -b output binary data for -Dd|f|s|v option. Append s for single
255 and d for double precision [Default is double].
256
258 Field Abbreviation Units
259 Two-way Travel Time twt sec
260 Corrected Depth depth m
261 Mag Total Field1 mtf1 nT
262 Mag Total Field2 mtf2 nT
263 Residual Magnetic mag nT
264 Diurnal Correction diur nT
265 Mag Sensor Depth/Alt msens m
266 Observed Gravity gobs mGal
267 Eotvos Correction eot mGal
268 Free Air Anomaly faa mGal
269
271 For -g the grids must be in the format used by Sandwell & Smith, which
272 is a spherical Mercator 2-byte grid with no header. For -G the grid
273 files can be of any grid type supported by GMT and therefore must con‐
274 tain a GMT header. A correctly formatted *.i2 grid file can be gener‐
275 ated using grdraster as shown below.
276 gmtset GRIDFILE_SHORTHAND TRUE
278 Create/edit .gmt_io file to include the following rows:
280 # GMT I/O shorthand file
281 # suffix format_id scale offset NaN
282 grd 0 - - -
283 i2 2 - - 32767
284 grdraster 1 -R0/359:55/-90/90 -Getopo5_hdr.i2
286 The new grid, etopo5_hdr.i2 in this example, contains a GMT header and
288 can be used in the -G option to compare cruise depth with grid values.
289
291 Header Information pertaining to an entire cruise, such as NGDC and
292 survey institution identification codes, cruise examination
293 time, two-way travel time corrector information, data precision
294 warnings, as well as systematic scales, DC shifts and correla‐
295 tion coefficients from global grid comparisons are reported as
296 E77 header information.
297 Sample
299 # Cruise 08010039 ID 74010908 MGD77 FILE VERSION: 19801230
300 N_RECS: 3066
301 # Examined: Wed Oct 3 16:30:13 2007 by mtchandl
302 # Arguments: -De -Gdepth,/data/GRIDS/etopo5_hdr.i2
303 N Errata table verification status
304 # mgd77manage applies corrections if the errata table is veri‐
305 fied (toggle 'N' above to 'Y' after review)
306 # For instructions on E77 format and usage, see
307 http://gmt.soest.hawaii.edu/mgd77/errata.php
308 # Verified by:
309 # Comments:
310 # Errata: Header
311 Y-E-08010039-H13-02: Invalid Magnetics Sampling Rate: (99) [ ]
312 Y-W-08010039-H13-10: Survey year (1975) outside magnetic refer‐
313 ence field IGRF 1965 time range (1965-1970)
314 Y-I-08010039-depth-00: RLS m: 1.00053 b: 0 rms: 127.851 r:
315 0.973422 significant: 1 decimation: 0
316 Y-W-08010039-twt-09: More recent bathymetry correction table
317 available
318 Y-W-08010039-mtf1-10: Integer precision
319 Y-W-08010039-mag-10: Integer precision
320 Error Record
323 Individual error records have strict format. Included is a time
324 or distance column followed by record number, a formatted error
325 code string, and finally a verbose description of errors
326 detected in the record. Three error classes are encoded into the
327 error code string with different alphabetic characters repre‐
328 senting unique error types. See below for error code format
329 description.
330 Format
332 <time/distance> <record number> <error code string> <descrip‐
333 tion>
334 Sample
336 # Errata: Data
337 Y 08010039 1975-05-10T22:16:05.88 74 C-0-0 NAV:
338 excessive speed
339 Error Code Description
342 Each of the three error classes is separated by a dash '-' and
343 described by a combination of alphabetic characters or 0 signi‐
344 fying no detected problems.
345 Error classes: NAV-VAL-GRAD
347 Error Class Descriptions
349 NAV (navigation):
350 0 - fine
351 A - time out of range
352 B - time decreasing
353 C - excessive speed
354 D - above sea level
355 E - lat undefined
356 F - lon undefined
357 VAL (value):
359 0 - fine
360 K - twt invalid
361 L - depth invalid
362 O - mtf1 invalid
363 etc.
364 GRAD (gradient):
366 0 - fine
367 K - d[twt] excessive
368 L - d[depth] excessive
369 O - d[mtf1] excessive
370 etc.
371 The NAV error class has unique cases while VAL and GRAD classes
373 are described by alphabetic characters for each of the 24
374 numeric fields in MGD77 format order.
375 MGD77 bit-pattern w/ E77 alpha characters
377 |-------------------------------------------------|----------|
378 | X W V U T S R Q P O N M L K J I H G F E D C B A | E77 Code |
379 | - - - - - - - - - - - - - - - - - - - - - - - - | - - - - -|
380 | n f e g m d m m m m b b d t p l l m h d m y t d | F I |
381 | q a o o s i s a t t t c e w t o a i o a o e z r | i D |
382 | c a t b d u e g f f c c p t c n t n u y n a t | e |
383 | s r n 2 1 t r t r | l |
384 | s h h | d |
385 | - - - - - - - - - - - - - - - - - - - - - - - - | - - - - -|
386 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | Bit place|
387 | - G C G C C - G G G - - G G - - - T T T T T - - | Bit type |
388 |-------------------------------------------------|----------|
389 Bit types: (G)eophysical, (C)orrection, (T)ime
390
393 To scan for excessive values or gradients, try
394 mgd77sniffer 08010001
396 To dump cruise gradients, try
398 mgd77sniffer 08010001 -Ds
400 To compare cruise depth with ETOPO5 bathymetry and gravity with
402 Sandwell/Smith 2 min gravity version 11, try
403 mgd77sniffer 08010001 -Gdepth,/data/GRIDS/etopo5_hdr.i2
405 -gfaa,/data/GRIDS/grav.11.2.img,0.1,1
406
409 mgd77list(1), mgd77track(1) x2sys_init(1)
410
412 Wessel, P., and W. H. F. Smith, 2009, The Generic Mapping Tools (GMT)
413 version 4.5.0 Technical Reference & Cookbook, SOEST/NOAA.
414 Wessel, P., and W. H. F. Smith, 1998, New, Improved Version of Generic
415 Mapping Tools Released, EOS Trans., AGU, 79 (47), p. 579.
416 Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Map‐
417 ping Tools Released, EOS Trans., AGU, 76 (33), p. 329.
418 Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Map‐
419 ping Tools Released, http://www.agu.org/eos_elec/95154e.html, Copyright
420 1995 by the American Geophysical Union.
421 Wessel, P., and W. H. F. Smith, 1991, Free Software Helps Map and Dis‐
422 play Data, EOS Trans., AGU, 72 (41), p. 441.
423 The Marine Geophysical Data Exchange Format - "MGD77", see
424 http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt
425GMT 4.5.6 10 Mar 2011 MGD77SNIFFER(1)