1MGD77MANAGE(1) Generic Mapping Tools MGD77MANAGE(1)
2
6 mgd77manage - Manage extra columns in MGD77+ files
7
9 mgd77manage NGDC-ids [ -A[+]a|c|d|D|e|g|i|n|t|Tfileinfo ] [ -Cf|g|e ] [
10 -Dabbrev1,abbrev2,... ] [ -Eempty ] [ -Iabbrev/name/unit/t/scale/off‐
11 set/comment ] [ -Ne|k|m|n ] [ -Q[b|c|l|n][[/]threshold] ] [ -V ] [
12 -bi[s|S|d|D[ncol]|c[var1/...]] ]
13
15 mgd77manage deals with maintaining extra custom columns in MGD77+
16 netCDF files. You can either delete one or more columns, add a new
17 column, update an existing column with new data, or supply error cor‐
18 rection information (*.e77 files). New data may come from a table
19 (ASCII unless -b is used), be based on existing columns and certain
20 theoretical expressions, or they may be obtained by sampling a grid
21 (choose between GMT grid or a Sandwell/Smith Mercator *.img grid) along
22 track. The new data will be appended to the MGD77+ file in the form of
23 an extra data column of specified type. The data file will be modi‐
24 fied; no new file will be created. For the big issues, see the DISCUS‐
25 SION section below.
26 NGDC-ids
28 Can be one or more of five kinds of specifiers:
29 1) 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
30 2) 2-character <agency> codes which will return all cruises from
31 each agency.
32 3) 4-character <agency><vessel> codes, which will return all
33 cruises from those vessels.
34 4) =<list>, where <list> is a table with NGDC IDs, one per line.
35 5) If nothing is specified we return all cruises in the data
36 base.
37 (See mgd77info -F for agency and vessel codes). The ".mgd77" or
38 ".nc" extensions will automatically be appended, if needed (use
39 -I to ignore certain file types). Cruise files will be looked
40 for first in the current directory and second in all directories
41 listed in $MGD77_HOME/mgd77_paths.txt [If $MGD77_HOME is not set
42 it will default to $GMT_SHAREDIR/mgd77].
43
45 No space between the option flag and the associated arguments
46 -A Add a new data column. If an existing column with the same
48 abbreviation already exists in the file we will cowardly refuse
49 to update the file. Specifying -A+ overcomes this reluctance
50 (However, sometimes an existing column cannot be upgraded with‐
51 out first deleting it; if so you will be warned). Select a col‐
52 umn source code among a, c, d, D, e, g, i, n, t, or T; detailed
53 descriptions for each choice follow:
54 a Append filename of a single column table to add. File must
56 have the same number of rows as the MGD77+ file. If no file is
57 given we read from stdin instead.
58 c Create a new column that derives from existing data or formu‐
60 las for corrections and reference fields. Append c for the
61 Carter corrections subtracted from uncorrected depths, g for the
62 IGF gravity reference field (a.k.a "normal gravity"), m for the
63 IGRF total field magnetic reference field, and r for recomputed
64 magnetic anomaly (append 1 or 2 to specify which total field
65 column to use [1]). For gravity we choose the reference field
66 based on the parameter Gravity Theoretical Formula Code in the
67 cruise's MGD77 header. If this is not set or is invalid we
68 default to the IGF 1980. You can override this behaviour by
69 appending the desired code: 1 = Heiskanen 1924, 2 = Interna‐
70 tional 1930, 3 = IGF1967, or 4 = IGF1980.
71 d Append filename of a two-column table with the first column
73 holding distances along track and the second column holding data
74 values. If no file is given we read from stdin instead.
75 Records with matching distances in the MGD77+ file will be
76 assigned the new values; at other distances we set them to NaN.
77 Alternatively, give upper case D instead and we will interpolate
78 the column at all record distances. See -N for chosing distance
79 units and -C for chosing how distances are calculated.
80 e Expects to find an e77 error/correction log from mgd77sniffer
82 with the name NGDC_ID.e77 in the current directory or in
83 $MGD77_HOME/E77; this file will examined and used to make modi‐
84 fications to the header values, specify a systematic correction
85 for certain columns (such as scale and offset), and add or
86 update the special column flag which may hold bitflags (0 =
87 GOOD, 1 = BAD) for each data field in the standard MGD77 data
88 set. Any fixed correction terms found (such as needing to scale
89 a field by 0.1 or 10 because the source agency used incorrect
90 units) will be written as attributes to the netCDF MGD77+ file
91 and applied when the data are read by mgd77list. Emphemeral
92 corrections such as those determined by crossover analysis are
93 not kept in the data files but reside in correction tables (see
94 mgd77list for details). By default, the first character of each
95 header line in the e77 file (which is ?, Y or N) will be con‐
96 sulted to see if the corresponding adjustment should be applied.
97 If any undecided settings are found (i.i, ?) we will abort and
98 make no changes. Only records marked Y will be porcessed. You
99 can override this behavior by appending one or more modifiers to
100 the -Ae command: h will ignore all header corrections, f will
101 ignore all fixed systematic trend corrections, n, v, and s will
102 ignore bitflags pertaining to navigation, data values, and data
103 slopes, respectively. Use -A+e to replace any existing E77 cor‐
104 rections in the file with the new values.
105 g Sample a GMT geographic (lon, lat) grid along the track given
107 by the MGD77+ file using bicubic interpolation (however, see
108 -Q). Append name of a GMT grid file.
109 i Sample a Sandwell/Smith Mercator *.img grid along the track
111 given by the MGD77+ file using bicubic interpolation (however,
112 see -Q). Append the gridded data scale (typically 1 or 0.1),
113 the IMG file mode (0-3), and finally the img grid filename. The
114 modes stand for the following: (0) Img files with no constraint
115 code, returns data at all points, (1) Img file with constraints
116 coded, return data at all points, (2) Img file with constraints
117 coded, return data only at constrained points and NaN elsewhere,
118 and (3) Img file with constraints coded, return 1 at constraints
119 and 0 elsewhere.
120 n Append filename of a two-column table with the first column
122 holding the record number (0 to nrows - 1) and the second column
123 holding data values. If no file is given we read from stdin
124 instead. Records with matching record numbers in the MGD77+
125 file will be assigned the new values; at other records we set
126 them to NaN.
127 t Append filename of a two-column table with the first column
129 holding absolute times along track and the second column holding
130 data values. If no file is given we read from stdin instead.
131 Records with matching times in the MGD77+ file will be assigned
132 the new values; at other times we set them to NaN. Alterna‐
133 tively, give upper case T instead and we will interpolate the
134 column at all record times.
135 -C Append a one-letter code to select the procedure for along-track
137 distance calculation when using -Ad|D (see -N for selecting dis‐
138 tance units):
139 f Flat Earth distances.
140 g Great circle distances [Default].
141 e Geodesic distances on current GMT ellipsoid.
142 -D Give a comma-separated list of column abbreviations that you
144 want to delete from the MGD77+ files. Do NOT use this option to
145 remove columns that you are replacing with new data (use -A+
146 instead). Because we cannot remove variables from netCDF files
147 we must create a new file without the columns to be deleted.
148 Once the file is successfully created we temporarily rename the
149 old file, change the new filename to the old filename, and
150 finally remove the old, renamed file.
151 -E Give a single character that will be repeated to fill empty
153 string values, e.g., '9' will yield a string like "99999..."
154 [9].
155 -I In addition to file information we must specify additional
157 information about the extra column. Specify a short (16 char or
158 less, using lower case letters, digits, or underscores only)
159 abbreviation for the selected data, its more descriptive name,
160 the data unit, the data type 1-character code (byte, short,
161 float, int, double, or text) you want used for storage in the
162 netCDF file, any scale and offset we should apply to the data to
163 make them fit inside the range implied by the chosen storage
164 type, and a general comment (< 128 characters) regarding what
165 these data represent. Note: If text data type is selected then
166 the terms "values" in the -A discussion refer to your text data.
167 Furthermore, the discussion on interpolation does not apply and
168 the NaN value becomes a "no string" value (see -E for what this
169 is). Place quotes around terms with more than one word (e.g.,
170 "Corrected Depth").
171 -N Specify the distance unit used when using -Ad|D by appending e
173 (meter), k (km), m (miles), or n (nautical miles). [Default is
174 -Nk (km)].
175 -Q Quick mode, use bilinear rather than bicubic interpolation
177 [Default]. Alternatively, select the interpolation mode by
178 adding b for B-spline smooting, c for bicubic interpolation, l
179 for bilinear interpolation or n for nearest-neighbor value.
180 Optionally, append threshold in the range [0,1]. This parameter
181 controls how close to nodes with NaN values the interpolation
182 will go. E.g., a threshold of 0.5 will interpolate about half
183 way from a non-NaN to a NaN node, whereas 0.1 will go about 90%
184 of the way, etc. [Default is 1, which means none of the (4 or
185 16) nearby nodes may be NaN]. -Q0 will just return the value of
186 the nearest node instead of interpolating. This is the same as
187 using -Qn. Only relevant when -Ag|i is selected.
188 -V Selects verbose mode, which will send progress reports to stderr
190 [Default runs "silently"].
191 -bi Selects binary input. Append s for single precision [Default is
193 d (double)]. Uppercase S or D will force byte-swapping.
194 Optionally, append ncol, the number of columns in your binary
195 input file if it exceeds the columns needed by the program. Or
196 append c if the input file is netCDF. Optionally, append
197 var1/var2/... to specify the variables to be read. This applies
198 to the input 1- or 2-column data files specified under some of
199 the -A options. The binary input option is only available for
200 numerical data columns.
201
203 To append Geosat/ERS-1 gravity version 11.2 as an extra data column in
204 the cruises 01010047.nc and 01010008.nc, storing the values as mGal*10
205 in a 2-byte short integer, try
206 mgd77manage 01010047 01010008 -Ai10/1/grav.11.2.img -Isat‐
208 grav/"Geosat/ERS-1 gravity"/"mGal"/s/10/0/"Sandwell/Smith version 11.2"
209 -V
210 To append a filtered version of magnetics as an extra data column of
212 type float for the cruise 01010047.nc, and interpolate the filtered
213 data at the times given in the MGD77+ file, try
214 mgd77manage 01010047 -ATmymag.tm -Ifiltmag/"Intermediate-wavelength
216 magnetic residuals"/"nTesla"/f/1/0/"Useful for looking for isochrons"
217 -V
218 To delete the existing extra columns satfaa, coastdist, and satvgg from
220 all MGD77+ files, try
221 mgd77manage `cat allmgd77.lis` -Dsatfaa,coastdist,satvgg -V
223 To create a 4-byte float column with the correct IGRF reference field
225 in all MGD77+ files, try
226 mgd77manage `cat allmgd77.lis` -Acm -Iigrf/"IGRF reference
228 field"/"nTesla"/f/1/0/"IGRF version 10 for 1990-2010" -V
229
232 1. Preamble
233 The mgd77 supplement is an attempt to (1) improve on the limited func‐
234 tionality of the existing mgg supplement, (2) incorporate some of the
235 ideas from Scripps' gmt+ supplement by allowing extra data columns, and
236 (3) add new capabilities for managing marine geophysical trackline data
237 stored in an architecture-independent CF-1.0- and COARDS-compliant
238 netCDF file format. Here are some of the underlying ideas and steps
239 you need to take to maintain your files.
240 2. Introduction
242 Our starting point is the MGD77 ASCII data files distributed from NGDC
243 on CD-ROMS, DVD-ROMS, and via FTP. Using Geodas to install the files
244 locally we choose the "Carter corrected depth" option which will fill
245 in the depth column using the two-way traveltimes and the Carter tables
246 if twt is present. This step yields ~5000 individual cruise files.
247 Place these in one or more sub-directories of your choice, list these
248 sub-directories (one per line) in the file mgd77_paths.txt, and place
249 that file in the directory pointed to by $MGD77_HOME; if not set this
250 variable defaults to $GMT_SHAREDIR/mgd77.
251 3. Conversion
253 Convert the ASCII MGD77 files to the new netCDF MGD77+ format using
254 mgd77convert. Typically, you will make a list of all the cruises to be
255 converted (with or without extension), and you then run
256 mgd77convert -Fa -Tc -V -Lwe+ `cat cruises.lis` > log.txt
258 The verbose settings will ensure that all problems found during conver‐
260 sion will be reported. The new *.nc files may also be placed in one or
261 more separate sub-directories and these should also be listed in the
262 mgd77_paths.txt file. We suggest you place the directories with *.nc
263 files ahead of the *.mgd77 directories. When you later want to limit a
264 search to files of a certain extension you should use the -I option.
265 4. Adding new columns
267 mgd77manage will allow you to add additional data columns to your *.nc
268 files. These can be anything, including text strings, but most likely
269 are numerical values sampled along the track from a supplied grid or an
270 existing column that have been filtered or manipulated for a particular
271 purpose. The format supports up to 32 such extra columns. See this
272 man page for how to add columns. You may later decide to remove some
273 of these columns or update the data associated with a certain column.
274 Data extraction tools such as mgd77list can be used to extract a mix of
275 standard MGD77 columns (navigation, time, and the usual geophysical
276 observations) and your custom columns.
277 5. Error sources
279 Before we discuss how to correct errors we will first list the differ‐
280 ent classes of errors associated with MGD77 data: (1) Header record
281 errors occur when some of the information fields in the header do not
282 comply with the MGD77 specification or required information is missing.
283 mgd77convert will list these errors when the extended verbose setting
284 is selected. These errors typically do not affect the data and are
285 instead errors in the meta-data (2) Fixed systematic errors occur when
286 a particular data column, despite the MGD77 specification, has been
287 encoded incorrectly. This usually means the data will be off by a con‐
288 stant factor such as 10 or 0.1, or in some cases even 1.8288 which con‐
289 verts fathoms to meters. (3) Unknown systematic errors occur when the
290 instrument that recorded the data or the processing that followed
291 introduced signals that appear to be systematic functions of time along
292 track, latitude, heading, or some other combination of terms that have
293 a physical or logical explanation. These terms may sometimes be
294 resolved by data analysis techniques such as along-track and across-
295 track investigations, and will result in correction terms that when
296 applied to the data will remove these unwanted signals in an optimal
297 way. Because these correction terms may change when new data are con‐
298 sidered in their determination, such corrections are considered to be
299 ephemeral. (4) Individual data points or sequences of data may violate
300 rules such as being outside of possible ranges or in other ways violate
301 sanity. Furthermore, sequences of points that may be within valid
302 ranges may give rise to data gradients that are unreasonable. The sta‐
303 tus of every point can therefore be determined and this gives rise to
304 bitflags GOOD or BAD. Our policy is that error sources 1, 2, and 4
305 will be corrected by suppliying the information as meta-data in the
306 relevant *.nc files, whereas the corrections for error source 3
307 (because they will constantly be improved) will be maintained in a sep‐
308 arate list of corrections.
309 6. Finding errors
311 The mgd77sniffer is a tool that does a thorough along-track sanity
312 check of the original MGD77 ASCII files and produces a corresponding
313 *.e77 error log. All problems found are encoded in the error log, and
314 recommended fixed correction terms are given, if needed. An analyst
315 may verify that the suggested corrections are indeed valid (we only
316 want to correct truly obvious unit errors), edit these error logs and
317 modify such correction terms and activate them by changing the relevant
318 code key (see mgd77sniffer for more details). mgd77manage can ingest
319 these error logs and (1) correct bad header records given the sugges‐
320 tions in the log, (2) insert scale/offset correction terms to be used
321 when reading certain columns, and (3) insert any bit-flags found.
322 Rerun this step if you later find other problems as all E77 settings or
323 flags will be recreated based on the latest E77 log.
324 7. Error corrections
326 The extraction program mgd77list allows for corrections to be applied
327 on-the-fly when data are requested. First, data with BAD bitflags are
328 suppressed. Second, data with fixed systematic correction terms are
329 corrected accordingly. Third, data with ephemeral correction terms
330 will have those corrections applied (if a correction table is sup‐
331 plied). All of these steps require the presence of the relevant meta-
332 data and all can be overruled by the user. In addition, users may add
333 their own bitflags as separate data columns and use mgd77list's logical
334 tests to further dictate which data are suppressed from output.
335
337 The IGRF calculations are based on a Fortran program written by Susan
338 Macmillan, British Geological Survey, translated to C via f2c by
339 Joaquim Luis, and adapted to GMT style by Paul Wessel.
340
342 mgd77convert(1), mgd77list(1), mgd77info(1), mgd77sniffer(1)
343 mgd77track(1) x2sys_init(1)
344
346 Wessel, P., and W. H. F. Smith, 2005, The Generic Mapping Tools (GMT)
347 version 4.1 Technical Reference & Cookbook, SOEST/NOAA.
348 Wessel, P., and W. H. F. Smith, 1998, New, Improved Version of Generic
349 Mapping Tools Released, EOS Trans., AGU, 79 (47), p. 579.
350 Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Map‐
351 ping Tools Released, EOS Trans., AGU, 76 (33), p. 329.
352 Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Map‐
353 ping Tools Released, http://www.agu.org/eos_elec/95154e.html, Copyright
354 1995 by the American Geophysical Union.
355 Wessel, P., and W. H. F. Smith, 1991, Free Software Helps Map and Dis‐
356 play Data, EOS Trans., AGU, 72 (41), p. 441.
357 The Marine Geophysical Data Exchange Format - "MGD77", see
358 http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt
359 IGRF, see http://www.ngdc.noaa.gov/IAGA/vmod/igrf.html
360GMT 4.3.1 15 May 2008 MGD77MANAGE(1)