1MGD77MANAGE(1) Generic Mapping Tools MGD77MANAGE(1)
2
6 mgd77manage - Manage extra columns in MGD77+ files
7
9 mgd77manage NGDC-ids [ -A[+]a|c|d|D|e|E|g|i|n|t|Tfileinfo ] [ -Cf|g|e ]
10 [ -Dabbrev1,abbrev2,... ] [ -Eempty ] [ -F ] [ -Iab‐
11 brev/name/unit/t/scale/offset/comment ] [ -Ne|k|m|n ] [
12 -Q[b|c|l|n][[/]threshold] ] [ -V ] [ -bi[s|S|d|D[ncol]|c[var1/...]] ]
13
15 mgd77manage deals with maintaining extra custom columns in MGD77+
16 netCDF files. You can either delete one or more columns, add a new
17 column, update an existing column with new data, or supply error cor‐
18 rection information (*.e77 files). New data may come from a table
19 (ASCII unless -b is used), be based on existing columns and certain
20 theoretical expressions, or they may be obtained by sampling a grid
21 (choose between GMT grid or a Sandwell/Smith Mercator *.img grid) along
22 track. The new data will be appended to the MGD77+ file in the form of
23 an extra data column of specified type. The data file will be modi‐
24 fied; no new file will be created. For the big issues, see the DISCUS‐
25 SION section below.
26 NGDC-ids
28 Can be one or more of five kinds of specifiers:
29 1) 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
30 2) 2-character <agency> codes which will return all cruises from
31 each agency.
32 3) 4-character <agency><vessel> codes, which will return all
33 cruises from those vessels.
34 4) =<list>, where <list> is a table with NGDC IDs, one per line.
35 5) If nothing is specified we return all cruises in the data
36 base.
37 (See mgd77info -L for agency and vessel codes). The ".mgd77" or
38 ".nc" extensions will automatically be appended, if needed (use
39 -I to ignore certain file types). Cruise files will be looked
40 for first in the current directory and second in all directories
41 listed in $MGD77_HOME/mgd77_paths.txt [If $MGD77_HOME is not set
42 it will default to $GMT_SHAREDIR/mgd77].
43
45 No space between the option flag and the associated arguments
46 -A Add a new data column. If an existing column with the same
48 abbreviation already exists in the file we will cowardly refuse
49 to update the file. Specifying -A+ overcomes this reluctance
50 (However, sometimes an existing column cannot be upgraded with‐
51 out first deleting it; if so you will be warned). Select a col‐
52 umn source code among a, c, d, D, e, g, i, n, t, or T; detailed
53 descriptions for each choice follow:
54 a Append filename of a single column table to add. File must
56 have the same number of rows as the MGD77+ file. If no file is
57 given we read from stdin instead.
58 c Create a new column that derives from existing data or formu‐
60 las for corrections and reference fields. Append c for the
61 Carter corrections subtracted from uncorrected depths, g for the
62 IGF gravity reference field (a.k.a "normal gravity"), m for the
63 IGRF total field magnetic reference field, and r for recomputed
64 magnetic anomaly (append 1 or 2 to specify which total field
65 column to use [1]). For gravity we choose the reference field
66 based on the parameter Gravity Theoretical Formula Code in the
67 cruise's MGD77 header. If this is not set or is invalid we
68 default to the IGF 1980. You can override this behaviour by
69 appending the desired code: 1 = Heiskanen 1924, 2 = Interna‐
70 tional 1930, 3 = IGF1967, or 4 = IGF1980.
71 d Append filename of a two-column table with the first column
73 holding distances along track and the second column holding data
74 values. If no file is given we read from stdin instead.
75 Records with matching distances in the MGD77+ file will be
76 assigned the new values; at other distances we set them to NaN.
77 Alternatively, give upper case D instead and we will interpolate
78 the column at all record distances. See -N for choosing dis‐
79 tance units and -C for choosing how distances are calculated.
80 e Expects to find an e77 error/correction log from mgd77sniffer
82 with the name NGDC_ID.e77 in the current directory or in
83 $MGD77_HOME/E77; this file will examined and used to make modi‐
84 fications to the header values, specify a systematic correction
85 for certain columns (such as scale and offset), specify that a
86 certain anomaly should be recalculated from the observations
87 (e.g., recalculate mag from mtf1 and the latest IGRF), and add
88 or update the special column flag which may hold bitflags (0 =
89 GOOD, 1 = BAD) for each data field in the standard MGD77 data
90 set. Any fixed correction terms found (such as needing to scale
91 a field by 0.1 or 10 because the source agency used incorrect
92 units) will be written as attributes to the netCDF MGD77+ file
93 and applied when the data are read by mgd77list. Ephemeral cor‐
94 rections such as those determined by crossover analysis are not
95 kept in the data files but reside in correction tables (see
96 mgd77list for details). By default, the first character of each
97 header line in the e77 file (which is ?, Y or N) will be con‐
98 sulted to see if the corresponding adjustment should be applied.
99 If any undecided settings are found (i.i, ?) we will abort and
100 make no changes. Only records marked Y will be porcessed. You
101 can override this behavior by appending one or more modifiers to
102 the -Ae command: h will ignore all header corrections, f will
103 ignore all fixed systematic trend corrections, n, v, and s will
104 ignore bitflags pertaining to navigation, data values, and data
105 slopes, respectively. Use -A+e to replace any existing E77 cor‐
106 rections in the file with the new values. Finally, e77 correc‐
107 tions will not be applied if the E77 file has not been verified.
108 Use -AE to ignore the verification status.
109 g Sample a GMT geographic (lon, lat) grid along the track given
111 by the MGD77+ file using bicubic interpolation (however, see
112 -Q). Append name of a GMT grid file.
113 i Sample a Sandwell/Smith Mercator *.img grid along the track
115 given by the MGD77+ file using bicubic interpolation (however,
116 see -Q). Append the img grid filename, followed by the comma-
117 separated data scale (typically 1 or 0.1), the IMG file mode
118 (0-3), and optionally the img grid max latitude [80.738]. The
119 modes stand for the following: (0) Img files with no constraint
120 code, returns data at all points, (1) Img file with constraints
121 coded, return data at all points, (2) Img file with constraints
122 coded, return data only at constrained points and NaN elsewhere,
123 and (3) Img file with constraints coded, return 1 at constraints
124 and 0 elsewhere.
125 n Append filename of a two-column table with the first column
127 holding the record number (0 to nrows - 1) and the second column
128 holding data values. If no file is given we read from stdin
129 instead. Records with matching record numbers in the MGD77+
130 file will be assigned the new values; at other records we set
131 them to NaN.
132 t Append filename of a two-column table with the first column
134 holding absolute times along track and the second column holding
135 data values. If no file is given we read from stdin instead.
136 Records with matching times in the MGD77+ file will be assigned
137 the new values; at other times we set them to NaN. Alterna‐
138 tively, give upper case T instead and we will interpolate the
139 column at all record times.
140 -C Append a one-letter code to select the procedure for along-track
142 distance calculation when using -Ad|D (see -N for selecting dis‐
143 tance units):
144 f Flat Earth distances.
145 g Great circle distances [Default].
146 e Geodesic distances on current GMT ellipsoid.
147 -D Give a comma-separated list of column abbreviations that you
149 want to delete from the MGD77+ files. Do NOT use this option to
150 remove columns that you are replacing with new data (use -A+
151 instead). Because we cannot remove variables from netCDF files
152 we must create a new file without the columns to be deleted.
153 Once the file is successfully created we temporarily rename the
154 old file, change the new filename to the old filename, and
155 finally remove the old, renamed file.
156 -E Give a single character that will be repeated to fill empty
158 string values, e.g., '9' will yield a string like "99999..."
159 [9].
160 -F Force mode. When this mode is active you are empowered to
162 delete or replace even the standard MGD77 set of columns. You
163 better know what you are doing!
164 -I In addition to file information we must specify additional
166 information about the extra column. Specify a short (16 char or
167 less, using lower case letters, digits, or underscores only)
168 abbreviation for the selected data, its more descriptive name,
169 the data unit, the data type 1-character code (byte, short,
170 float, int, double, or text) you want used for storage in the
171 netCDF file, any scale and offset we should apply to the data to
172 make them fit inside the range implied by the chosen storage
173 type, and a general comment (< 128 characters) regarding what
174 these data represent. Note: If text data type is selected then
175 the terms "values" in the -A discussion refer to your text data.
176 Furthermore, the discussion on interpolation does not apply and
177 the NaN value becomes a "no string" value (see -E for what this
178 is). Place quotes around terms with more than one word (e.g.,
179 "Corrected Depth").
180 -N Specify the distance unit used when using -Ad|D by appending e
182 (meter), k (km), m (miles), or n (nautical miles). [Default is
183 -Nk (km)].
184 -Q Quick mode, use bilinear rather than bicubic interpolation
186 [Default]. Alternatively, select the interpolation mode by
187 adding b for B-spline smoothing, c for bicubic interpolation, l
188 for bilinear interpolation or n for nearest-neighbor value.
189 Optionally, append threshold in the range [0,1]. This parameter
190 controls how close to nodes with NaN values the interpolation
191 will go. E.g., a threshold of 0.5 will interpolate about half
192 way from a non-NaN to a NaN node, whereas 0.1 will go about 90%
193 of the way, etc. [Default is 1, which means none of the (4 or
194 16) nearby nodes may be NaN]. -Q0 will just return the value of
195 the nearest node instead of interpolating. This is the same as
196 using -Qn. Only relevant when -Ag|i is selected.
197 -V Selects verbose mode, which will send progress reports to stderr
199 [Default runs "silently"].
200 -bi Selects binary input. Append s for single precision [Default is
202 d (double)]. Uppercase S or D will force byte-swapping.
203 Optionally, append ncol, the number of columns in your binary
204 input file if it exceeds the columns needed by the program. Or
205 append c if the input file is netCDF. Optionally, append
206 var1/var2/... to specify the variables to be read. This applies
207 to the input 1- or 2-column data files specified under some of
208 the -A options. The binary input option is only available for
209 numerical data columns.
210
212 To append Geosat/ERS-1 gravity version 11.2 as an extra data column in
213 the cruises 01010047.nc and 01010008.nc, storing the values as mGal*10
214 in a 2-byte short integer, try
215 mgd77manage 01010047 01010008 -Ai10/1/grav.11.2.img -Isat‐
217 grav/"Geosat/ERS-1 gravity"/"mGal"/s/10/0/"Sandwell/Smith version 11.2"
218 -V
219 To append a filtered version of magnetics as an extra data column of
221 type float for the cruise 01010047.nc, and interpolate the filtered
222 data at the times given in the MGD77+ file, try
223 mgd77manage 01010047 -ATmymag.tm -Ifiltmag/"Intermediate-wavelength
225 magnetic residuals"/"nTesla"/f/1/0/"Useful for looking for isochrons"
226 -V
227 To delete the existing extra columns satfaa, coastdist, and satvgg from
229 all MGD77+ files, try
230 mgd77manage `cat allmgd77.lis` -Dsatfaa,coastdist,satvgg -V
232 To create a 4-byte float column with the correct IGRF reference field
234 in all MGD77+ files, try
235 mgd77manage `cat allmgd77.lis` -Acm -Iigrf/"IGRF reference
237 field"/"nTesla"/f/1/0/"IGRF version 10 for 1990-2010" -V
238
241 1. Preamble
242 The mgd77 supplement is an attempt to (1) improve on the limited func‐
243 tionality of the existing mgg supplement, (2) incorporate some of the
244 ideas from Scripps' gmt+ supplement by allowing extra data columns, and
245 (3) add new capabilities for managing marine geophysical trackline data
246 stored in an architecture-independent CF-1.0- and COARDS-compliant
247 netCDF file format. Here are some of the underlying ideas and steps
248 you need to take to maintain your files.
249 2. Introduction
251 Our starting point is the MGD77 ASCII data files distributed from NGDC
252 on CD-ROMS, DVD-ROMS, and via FTP. Using Geodas to install the files
253 locally we choose the "Carter corrected depth" option which will fill
254 in the depth column using the two-way traveltimes and the Carter tables
255 if twt is present. This step yields ~5000 individual cruise files.
256 Place these in one or more sub-directories of your choice, list these
257 sub-directories (one per line) in the file mgd77_paths.txt, and place
258 that file in the directory pointed to by $MGD77_HOME; if not set this
259 variable defaults to $GMT_SHAREDIR/mgd77.
260 3. Conversion
262 Convert the ASCII MGD77 files to the new netCDF MGD77+ format using
263 mgd77convert. Typically, you will make a list of all the cruises to be
264 converted (with or without extension), and you then run
265 mgd77convert -Fa -Tc -V -Lwe+ `cat cruises.lis` > log.txt
267 The verbose settings will ensure that all problems found during conver‐
269 sion will be reported. The new *.nc files may also be placed in one or
270 more separate sub-directories and these should also be listed in the
271 mgd77_paths.txt file. We suggest you place the directories with *.nc
272 files ahead of the *.mgd77 directories. When you later want to limit a
273 search to files of a certain extension you should use the -I option.
274 4. Adding new columns
276 mgd77manage will allow you to add additional data columns to your *.nc
277 files. These can be anything, including text strings, but most likely
278 are numerical values sampled along the track from a supplied grid or an
279 existing column that have been filtered or manipulated for a particular
280 purpose. The format supports up to 32 such extra columns. See this
281 man page for how to add columns. You may later decide to remove some
282 of these columns or update the data associated with a certain column.
283 Data extraction tools such as mgd77list can be used to extract a mix of
284 standard MGD77 columns (navigation, time, and the usual geophysical
285 observations) and your custom columns.
286 5. Error sources
288 Before we discuss how to correct errors we will first list the differ‐
289 ent classes of errors associated with MGD77 data: (1) Header record
290 errors occur when some of the information fields in the header do not
291 comply with the MGD77 specification or required information is missing.
292 mgd77convert will list these errors when the extended verbose setting
293 is selected. These errors typically do not affect the data and are
294 instead errors in the meta-data (2) Fixed systematic errors occur when
295 a particular data column, despite the MGD77 specification, has been
296 encoded incorrectly. This usually means the data will be off by a con‐
297 stant factor such as 10 or 0.1, or in some cases even 1.8288 which con‐
298 verts fathoms to meters. (3) Unknown systematic errors occur when the
299 instrument that recorded the data or the processing that followed
300 introduced signals that appear to be systematic functions of time along
301 track, latitude, heading, or some other combination of terms that have
302 a physical or logical explanation. These terms may sometimes be
303 resolved by data analysis techniques such as along-track and across-
304 track investigations, and will result in correction terms that when
305 applied to the data will remove these unwanted signals in an optimal
306 way. Because these correction terms may change when new data are con‐
307 sidered in their determination, such corrections are considered to be
308 ephemeral. (4) Individual data points or sequences of data may violate
309 rules such as being outside of possible ranges or in other ways violate
310 sanity. Furthermore, sequences of points that may be within valid
311 ranges may give rise to data gradients that are unreasonable. The sta‐
312 tus of every point can therefore be determined and this gives rise to
313 bitflags GOOD or BAD. Our policy is that error sources 1, 2, and 4
314 will be corrected by supplying the information as meta-data in the rel‐
315 evant *.nc files, whereas the corrections for error source 3 (because
316 they will constantly be improved) will be maintained in a separate list
317 of corrections.
318 6. Finding errors
320 The mgd77sniffer is a tool that does a thorough along-track sanity
321 check of the original MGD77 ASCII files and produces a corresponding
322 *.e77 error log. All problems found are encoded in the error log, and
323 recommended fixed correction terms are given, if needed. An analyst
324 may verify that the suggested corrections are indeed valid (we only
325 want to correct truly obvious unit errors), edit these error logs and
326 modify such correction terms and activate them by changing the relevant
327 code key (see mgd77sniffer for more details). mgd77manage can ingest
328 these error logs and (1) correct bad header records given the sugges‐
329 tions in the log, (2) insert scale/offset correction terms to be used
330 when reading certain columns, and (3) insert any bit-flags found.
331 Rerun this step if you later find other problems as all E77 settings or
332 flags will be recreated based on the latest E77 log.
333 7. Error corrections
335 The extraction program mgd77list allows for corrections to be applied
336 on-the-fly when data are requested. First, data with BAD bitflags are
337 suppressed. Second, data with fixed systematic correction terms are
338 corrected accordingly. Third, data with ephemeral correction terms
339 will have those corrections applied (if a correction table is sup‐
340 plied). All of these steps require the presence of the relevant meta-
341 data and all can be overruled by the user. In addition, users may add
342 their own bitflags as separate data columns and use mgd77list's logical
343 tests to further dictate which data are suppressed from output.
344
346 The IGRF calculations are based on a Fortran program written by Susan
347 Macmillan, British Geological Survey, translated to C via f2c by
348 Joaquim Luis, and adapted to GMT style by Paul Wessel.
349
351 mgd77convert(1), mgd77list(1), mgd77info(1), mgd77sniffer(1)
352 mgd77track(1) x2sys_init(1)
353
355 Wessel, P., and W. H. F. Smith, 2009, The Generic Mapping Tools (GMT)
356 version 4.5.0 Technical Reference & Cookbook, SOEST/NOAA.
357 Wessel, P., and W. H. F. Smith, 1998, New, Improved Version of Generic
358 Mapping Tools Released, EOS Trans., AGU, 79 (47), p. 579.
359 Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Map‐
360 ping Tools Released, EOS Trans., AGU, 76 (33), p. 329.
361 Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Map‐
362 ping Tools Released, http://www.agu.org/eos_elec/95154e.html, Copyright
363 1995 by the American Geophysical Union.
364 Wessel, P., and W. H. F. Smith, 1991, Free Software Helps Map and Dis‐
365 play Data, EOS Trans., AGU, 72 (41), p. 441.
366 The Marine Geophysical Data Exchange Format - "MGD77", see
367 http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt
368 IGRF, see http://www.ngdc.noaa.gov/IAGA/vmod/igrf.html
369GMT 4.5.6 10 Mar 2011 MGD77MANAGE(1)