1MGD77MANAGE(1) GMT MGD77MANAGE(1)
2
6 mgd77manage - Manage the content of MGD77+ files
7
9 mgd77manage NGDC-ids [ -A[+]a|c|d|D|e|E|g|i|n|t|Tfileinfo ] [ -Cf|g|e
10 ] [ -Dabbrev1,abbrev2,...) ] [ -Eempty ] [ -F ] [ -Iab‐
11 brev/name/unit/t/scale/offset/comment ] [ -Nunit ] [ -Rregion ] [
12 -V[level] ] [ -bibinary ] [ -dinodata ] [ -nflags ]
13 Note: No space is allowed between the option flag and the associated
15 arguments.
16
18 mgd77manage deals with maintaining extra custom columns in MGD77+
19 netCDF files. You can either delete one or more columns, add a new col‐
20 umn, update an existing column with new data, or supply error correc‐
21 tion information (*.e77 files). New data may come from a table (ASCII
22 unless -bi is used), be based on existing columns and certain theoreti‐
23 cal expressions, or they may be obtained by sampling a grid (choose
24 between GMT grid or a Sandwell/Smith Mercator *.img grid) along track.
25 The new data will be appended to the MGD77+ file in the form of an
26 extra data column of specified type. The data file will be modified; no
27 new file will be created. For the big issues, see the DISCUSSION sec‐
28 tion below.
29
31 NGDC-ids
32 Can be one or more of five kinds of specifiers:
33 1. 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
35 2. 2-character agency codes which will return all cruises from
37 each agency.
38 3. 4-character <agency><vessel> codes, which will return all
40 cruises from those vessels.
41 4. =list, where list is a table with NGDC IDs, one per line.
43 5. If nothing is specified we return all cruises in the data
45 base.
46 (See mgd77info -L for agency and vessel codes). If no file
48 extension is given then we search for files with one of the four
49 known extensions. The search order (and the extensions) tried
50 is MGD77+ (".nc"), MGD77T (".m77t"), MGD77 (".mgd77" ) and plain
51 text file (".dat"). Use -I to ignore one or more of these file
52 types). Cruise files will be looked for first in the current
53 directory and second in all directories listed in
54 $MGD77_HOME/mgd77_paths.txt [If $MGD77_HOME is not set it will
55 default to $GMT_SHAREDIR/mgd77].
56
58 -A[+]a|c|d|D|e|E|g|i|n|t|Tfileinfo
59 Add a new data column. If an existing column with the same
60 abbreviation already exists in the file we will cowardly refuse
61 to update the file. Specifying -A+ overcomes this reluctance
62 (However, sometimes an existing column cannot be upgraded with‐
63 out first deleting it; if so you will be warned). Select a col‐
64 umn source code among a, c, d, D, e, g, i, n, t, or T; detailed
65 descriptions for each choice follow:
66 a Append filename of a single column table to add. File must
68 have the same number of rows as the MGD77+ file. If no file is
69 given we read from stdin instead.
70 c Create a new column that derives from existing data or formu‐
72 las for corrections and reference fields. Append c for the
73 Carter corrections subtracted from uncorrected depths, g for the
74 IGF gravity reference field (a.k.a "normal gravity"), m for the
75 IGRF total field magnetic reference field, and r for recomputed
76 magnetic anomaly (append 1 or 2 to specify which total field
77 column to use [1]). For gravity we choose the reference field
78 based on the parameter Gravity Theoretical Formula Code in the
79 cruise's MGD77 header. If this is not set or is invalid we
80 default to the IGF 1980. You can override this behavior by
81 appending the desired code: 1 = Heiskanen 1924, 2 = Interna‐
82 tional 1930, 3 = IGF1967, or 4 = IGF1980.
83 d Append filename of a two-column table with the first column
85 holding distances along track and the second column holding data
86 values. If no file is given we read from stdin instead. Records
87 with matching distances in the MGD77+ file will be assigned the
88 new values; at other distances we set them to NaN. Alterna‐
89 tively, give upper case D instead and we will interpolate the
90 column at all record distances. See -N for choosing distance
91 units and -C for choosing how distances are calculated.
92 e Expects to find an e77 error/correction log from mgd77sniffer
94 with the name NGDC_ID.e77 in the current directory or in
95 $MGD77_HOME/E77; this file will examined and used to make modi‐
96 fications to the header values, specify a systematic correction
97 for certain columns (such as scale and offset), specify that a
98 certain anomaly should be recalculated from the observations
99 (e.g., recalculate mag from mtf1 and the latest IGRF), and add
100 or update the special column flag which may hold bitflags (0 =
101 GOOD, 1 = BAD) for each data field in the standard MGD77 data
102 set. Any fixed correction terms found (such as needing to scale
103 a field by 0.1 or 10 because the source agency used incorrect
104 units) will be written as attributes to the netCDF MGD77+ file
105 and applied when the data are read by mgd77list. Ephemeral cor‐
106 rections such as those determined by crossover analysis are not
107 kept in the data files but reside in correction tables (see
108 mgd77list for details). By default, the first character of each
109 header line in the e77 file (which is ?, Y or N) will be con‐
110 sulted to see if the corresponding adjustment should be applied.
111 If any undecided settings are found (i.i, ?) we will abort and
112 make no changes. Only records marked Y will be processed. You
113 can override this behavior by appending one or more modifiers to
114 the -Ae command: h will ignore all header corrections, f will
115 ignore all fixed systematic trend corrections, n, v, and s will
116 ignore bitflags pertaining to navigation, data values, and data
117 slopes, respectively. Use -A+e to replace any existing E77 cor‐
118 rections in the file with the new values. Finally, e77 correc‐
119 tions will not be applied if the E77 file has not been verified.
120 Use -AE to ignore the verification status.
121 g Sample a GMT geographic (lon, lat) grid along the track given
123 by the MGD77+ file using bicubic interpolation (however, see
124 -n). Append name of a GMT grid file.
125 i Sample a Sandwell/Smith Mercator *.img grid along the track
127 given by the MGD77+ file using bicubic interpolation (however,
128 see -n). Append the img grid filename, followed by the
129 comma-separated data scale (typically 1 or 0.1), the IMG file
130 mode (0-3), and optionally the img grid max latitude [80.738].
131 The modes stand for the following: (0) Img files with no con‐
132 straint code, returns data at all points, (1) Img file with con‐
133 straints coded, return data at all points, (2) Img file with
134 constraints coded, return data only at constrained points and
135 NaN elsewhere, and (3) Img file with constraints coded, return 1
136 at constraints and 0 elsewhere.
137 n Append filename of a two-column table with the first column
139 holding the record number (0 to nrows - 1) and the second column
140 holding data values. If no file is given we read from stdin
141 instead. Records with matching record numbers in the MGD77+
142 file will be assigned the new values; at other records we set
143 them to NaN.
144 t Append filename of a two-column table with the first column
146 holding absolute times along track and the second column holding
147 data values. If no file is given we read from stdin instead.
148 Records with matching times in the MGD77+ file will be assigned
149 the new values; at other times we set them to NaN. Alterna‐
150 tively, give upper case T instead and we will interpolate the
151 column at all record times.
152 -Cf|g|e
154 Append a one-letter code to select the procedure for along-track
155 distance calculation when using -Ad|D (see -N for selecting dis‐
156 tance units):
157 f Flat Earth distances.
159 g Great circle distances [Default].
161 e Geodesic distances on current GMT ellipsoid.
163 -Dabbrev1,abbrev2,...)
165 Give a comma-separated list of column abbreviations that you
166 want to delete from the MGD77+ files. Do NOT use this option to
167 remove columns that you are replacing with new data (use -A+
168 instead). Because we cannot remove variables from netCDF files
169 we must create a new file without the columns to be deleted.
170 Once the file is successfully created we temporarily rename the
171 old file, change the new filename to the old filename, and
172 finally remove the old, renamed file.
173 -Eempty
175 Give a single character that will be repeated to fill empty
176 string values, e.g., "9" will yield a string like "99999..."
177 [9].
178 -F Force mode. When this mode is active you are empowered to delete
180 or replace even the standard MGD77 set of columns. You better
181 know what you are doing!
182 -Iabbrev/name/unit/t/scale/offset/comment
184 In addition to file information we must specify additional
185 information about the extra column. Specify a short (16 char or
186 less, using lower case letters, digits, or underscores only)
187 abbreviation for the selected data, its more descriptive name,
188 the data unit, the data type 1-character code (byte, short,
189 float, int, double, or text) you want used for storage in the
190 netCDF file, any scale and offset we should apply to the data to
191 make them fit inside the range implied by the chosen storage
192 type, and a general comment (< 128 characters) regarding what
193 these data represent. Note: If text data type is selected then
194 the terms "values" in the -A discussion refer to your text data.
195 Furthermore, the discussion on interpolation does not apply and
196 the NaN value becomes a "no string" value (see -E for what this
197 is). Place quotes around terms with more than one word (e.g.,
198 "Corrected Depth").
199 -Nunit Append the distance unit (see UNITS). [Default is -Nk (km)].
201 Only relevant when -Ag|i is selected.
202 -Rxmin/xmax/ymin/ymax[+r][+uunit] (more ...)
204 Specify the region of interest. Only relevant when -Ag|i is
205 selected.
206 -V[level] (more ...)
208 Select verbosity level [c].
209 -bi[ncols][t] (more ...)
211 Select native binary input. This applies to the input 1- or
212 2-column data files specified under some of the -A options. The
213 binary input option is only available for numerical data col‐
214 umns.
215 -dinodata (more ...)
217 Replace input columns that equal nodata with NaN.
218 -n[b|c|l|n][+a][+bBC][+c][+tthreshold] (more ...)
220 Select interpolation mode for grids.
221 -^ or just -
223 Print a short message about the syntax of the command, then
224 exits (NOTE: on Windows just use -).
225 -+ or just +
227 Print an extensive usage (help) message, including the explana‐
228 tion of any module-specific option (but not the GMT common
229 options), then exits.
230 -? or no arguments
232 Print a complete usage (help) message, including the explanation
233 of all options, then exits.
234
236 For map distance unit, append unit d for arc degree, m for arc minute,
237 and s for arc second, or e for meter [Default], f for foot, k for km, M
238 for statute mile, n for nautical mile, and u for US survey foot. By
239 default we compute such distances using a spherical approximation with
240 great circles. Prepend - to a distance (or the unit is no distance is
241 given) to perform "Flat Earth" calculations (quicker but less accurate)
242 or prepend + to perform exact geodesic calculations (slower but more
243 accurate).
244
246 Resample or sampling of grids will use various algorithms (see -n) that
247 may lead to possible distortions or unexpected results in the resampled
248 values. One expected effect of resampling with splines is the tendency
249 for the new resampled values to slightly exceed the global min/max lim‐
250 its of the original grid. If this is unacceptable, you can impose
251 clipping of the resampled values values so they do not exceed the input
252 min/max values by adding +c to your -n option.
253
255 To append Geosat/ERS-1 gravity version 11.2 as an extra data column in
256 the cruises 01010047.nc and 01010008.nc, storing the values as mGal*10
257 in a 2-byte short integer, try
258 gmt mgd77manage 01010047 01010008 -Ai10/1/grav.11.2.img \
260 -Isatgrav/"Geosat/ERS-1 gravity"/"mGal"/s/10/0/"Sandwell/Smith version 11.2" -V
261 To append a filtered version of magnetics as an extra data column of
263 type float for the cruise 01010047.nc, and interpolate the filtered
264 data at the times given in the MGD77+ file, try
265 gmt mgd77manage 01010047 -ATmymag.tm -Ifiltmag/"Intermediate-wavelength \
267 magnetic residuals"/"nTesla"/f/1/0/"Useful for looking for isochrons" -V
268 To delete the existing extra columns satfaa, coastdist, and satvgg from
270 all MGD77+ files, try
271 gmt mgd77manage =allmgd77.lis -Dsatfaa,coastdist,satvgg -V
273 To create a 4-byte float column with the correct IGRF reference field
275 in all MGD77+ files, try
276 gmt mgd77manage =allmgd77.lis -Acm -Iigrf/"IGRF reference \
278 field"/"nTesla"/f/1/0/"IGRF version 10 for 1990-2010" -V
279
281 1. Preamble
282 The mgd77 supplement is an attempt to (1) improve on the limited func‐
284 tionality of the existing mgg supplement, (2) incorporate some of the
285 ideas from Scripps' gmt+ supplement by allowing extra data columns, and
286 (3) add new capabilities for managing marine geophysical trackline data
287 stored in an architecture-independent CF-1.0- and COARDS-compliant
288 netCDF file format. Here are some of the underlying ideas and steps you
289 need to take to maintain your files.
290 2. Introduction
292 Our starting point is the MGD77 ASCII data files distributed from NGDC
294 on CD-ROMS, DVD-ROMS, and via FTP. Using Geodas to install the files
295 locally we choose the "Carter corrected depth" option which will fill
296 in the depth column using the two-way travel-times and the Carter
297 tables if twt is present. This step yields ~5000 individual cruise
298 files. Place these in one or more sub-directories of your choice, list
299 these sub-directories (one per line) in the file mgd77_paths.txt, and
300 place that file in the directory pointed to by $MGD77_HOME; if not set
301 this variable defaults to $GMT_SHAREDIR/mgd77.
302 3. Conversion
304 Convert the ASCII MGD77 files to the new netCDF MGD77+ format using
306 mgd77convert. Typically, you will make a list of all the cruises to be
307 converted (with or without extension), and you then run
308 mgd77convert =cruises.lis -Fa -Tc -V -Lwe+ > log.txt
309 The verbose settings will ensure that all problems found during conver‐
311 sion will be reported. The new *.nc files may also be placed in one or
312 more separate sub-directories and these should also be listed in the
313 mgd77_paths.txt file. We suggest you place the directories with *.nc
314 files ahead of the *.mgd77 directories. When you later want to limit a
315 search to files of a certain extension you should use the -I option.
316 4. Adding new columns
318 mgd77manage will allow you to add additional data columns to your *.nc
320 files. These can be anything, including text strings, but most likely
321 are numerical values sampled along the track from a supplied grid or an
322 existing column that have been filtered or manipulated for a particular
323 purpose. The format supports up to 32 such extra columns. See this man
324 page for how to add columns. You may later decide to remove some of
325 these columns or update the data associated with a certain column. Data
326 extraction tools such as mgd77list can be used to extract a mix of
327 standard MGD77 columns (navigation, time, and the usual geophysical
328 observations) and your custom columns.
329 5. Error sources
331 Before we discuss how to correct errors we will first list the differ‐
333 ent classes of errors associated with MGD77 data: (1) Header record
334 errors occur when some of the information fields in the header do not
335 comply with the MGD77 specification or required information is missing.
336 mgd77convert will list these errors when the extended verbose setting
337 is selected. These errors typically do not affect the data and are
338 instead errors in the meta-data (2). Fixed systematic errors occur when
339 a particular data column, despite the MGD77 specification, has been
340 encoded incorrectly. This usually means the data will be off by a con‐
341 stant factor such as 10 or 0.1, or in some cases even 1.8288 which con‐
342 verts fathoms to meters. (3) Unknown systematic errors occur when the
343 instrument that recorded the data or the processing that followed
344 introduced signals that appear to be systematic functions of time along
345 track, latitude, heading, or some other combination of terms that have
346 a physical or logical explanation. These terms may sometimes be
347 resolved by data analysis techniques such as along-track and
348 across-track investigations, and will result in correction terms that
349 when applied to the data will remove these unwanted signals in an opti‐
350 mal way. Because these correction terms may change when new data are
351 considered in their determination, such corrections are considered to
352 be ephemeral. (4) Individual data points or sequences of data may vio‐
353 late rules such as being outside of possible ranges or in other ways
354 violate sanity. Furthermore, sequences of points that may be within
355 valid ranges may give rise to data gradients that are unreasonable. The
356 status of every point can therefore be determined and this gives rise
357 to bitflags GOOD or BAD. Our policy is that error sources 1, 2, and 4
358 will be corrected by supplying the information as meta-data in the rel‐
359 evant *.nc files, whereas the corrections for error source 3 (because
360 they will constantly be improved) will be maintained in a separate list
361 of corrections.
362 6. Finding errors
364 The mgd77sniffer is a tool that does a thorough along-track sanity
366 check of the original MGD77 ASCII files and produces a corresponding
367 *.e77 error log. All problems found are encoded in the error log, and
368 recommended fixed correction terms are given, if needed. An analyst may
369 verify that the suggested corrections are indeed valid (we only want to
370 correct truly obvious unit errors), edit these error logs and modify
371 such correction terms and activate them by changing the relevant code
372 key (see mgd77sniffer for more details). mgd77manage can ingest these
373 error logs and (1) correct bad header records given the suggestions in
374 the log, (2) insert scale/offset correction terms to be used when read‐
375 ing certain columns, and (3) insert any bit-flags found. Rerun this
376 step if you later find other problems as all E77 settings or flags will
377 be recreated based on the latest E77 log.
378 7. Error corrections
380 The extraction program mgd77list allows for corrections to be applied
382 on-the-fly when data are requested. First, data with BAD bitflags are
383 suppressed. Second, data with fixed systematic correction terms are
384 corrected accordingly. Third, data with ephemeral correction terms will
385 have those corrections applied (if a correction table is supplied). All
386 of these steps require the presence of the relevant meta-data and all
387 can be overruled by the user. In addition, users may add their own bit‐
388 flags as separate data columns and use mgd77list's logical tests to
389 further dictate which data are suppressed from output.
390
392 The IGRF calculations are based on a Fortran program written by Susan
393 Macmillan, British Geological Survey, translated to C via f2c by
394 Joaquim Luis, and adapted to GMT style by Paul Wessel.
395
397 mgd77convert, mgd77list, mgd77info, mgd77sniffer mgd77track x2sys_init
398
400 The Marine Geophysical Data Exchange Format - MGD77, see
401 http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt
402 IGRF, see http://www.ngdc.noaa.gov/IAGA/vmod/igrf.html
404
406 2019, P. Wessel, W. H. F. Smith, R. Scharroo, J. Luis, and F. Wobbe
4075.4.5 Feb 24, 2019 MGD77MANAGE(1)