1WARTS(5) BSD File Formats Manual WARTS(5)
2
4 warts — format for scamper's warts storage.
5
7 This document describes the warts binary file format used by scamper(1).
8 A warts file consists of a set of records, each of which begin with an 8
9 byte object header.
10
12 The header consists of a 2-byte magic number (0x1205), a 2-byte type
13 value, and a 4-byte length value defining the size of the object that
14 follows. All 16-bit and 32-bit numbers are written in network byte
15 order. The currently defined types, and the types of scamper(1) object
16 they map to, are as follows:
17 - 0x0001: List (scamper_list_t)
18 - 0x0002: Cycle start (scamper_cycle_t)
19 - 0x0003: Cycle definition (scamper_cycle_t)
20 - 0x0004: Cycle stop (scamper_cycle_t)
21 - 0x0005: Address (scamper_addr_t) -- deprecated
22 - 0x0006: Traceroute (scamper_trace_t)
23 - 0x0007: Ping (scamper_ping_t)
24 - 0x0008: MDA traceroute (scamper_tracelb_t)
25 - 0x0009: Alias resolution (scamper_dealias_t)
26 - 0x000a: Neighbour Discovery (scamper_neighbourdisc_t)
27 - 0x000b: TCP Behaviour Inference Tool (scamper_tbit_t)
28 - 0x000c: Sting (scamper_sting_t)
29 - 0x000d: Sniff (scamper_sniff_t)
30 A new type number can be requested by emailing the author of scamper.
31 The structure of each warts record beyond the header is arbitrary, though
32 some conventions have been established to promote extensibility.
33
35 The warts routines in scamper provide the ability to conditionally store
36 arbitrary data in a forwards compatible method. A set of flags and
37 parameters begins with a sequence of bytes that denote which items are
38 included. If any flags are set, then after the flags is a 2-byte field
39 that records the length of the parameters that follow. Finally, the data
40 follows. The following figure illustrates how flags are recorded:
41 Byte zero Byte one Byte two
43 8 7 6 5 4 3 2 1 8 7 6 5 4 3 2 1 8 7 6 5 4 3 2 1
44 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
45 |1 | |1 | |0 |
46 +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
47 The most significant bit of each byte is the `link' bit; it determines if
48 the next byte in the sequence contains flags. The low-order 7 bits of
49 each byte signal if the corresponding field is written out in the parame‐
50 ters that follow. In the figure, the link bit is set to one in the first
51 two bytes, and zero in the final byte, signifying that three flag-bytes
52 are included.
53 The rest of each byte is used to record flags, whose position in the
55 sequence signifies if a particular parameter is included. For example,
56 if bit 6 of byte zero is set, then parameter 6 is included, and if bit 5
57 of byte one is set, then parameter 12 is included, and if bit 2 of byte
58 two is set, then parameter 16 is included.
59
61 A warts file may have addresses embedded in two ways. The first is dep‐
62 recated: the address is written as a data object that can be globally
63 referenced before the data object that uses it is written. A reader
64 therefore must keep a copy of all addresses it reads in order to be able
65 to decode data objects that subsequently reference it, which can consume
66 a significant amount of memory. The format of the address is
67 Warts header ID Modulo Type Address
69 +--------------+-----------+--------+----//----+
70 | 8 bytes | 1 byte | 1 byte | |
71 | type = 5 | | | |
72 +--------------+-----------+--------+----//----+
73 A reader determines the ID number of each address by the order in which
74 it appears, and can sanity check the ID number it determines by comparing
75 the lower 8 bits of the computed ID with the ID that is embedded in the
76 record. Address ID numbers start at one; zero is reserved for when no
77 address is embedded. The type corresponds to the type of address that
78 follows. The currently defined types are as follows:
79 - 0x01 IPv4 address
80 - 0x02 IPv6 address
81 - 0x03 48-bit Ethernet MAC address
82 - 0x04 64-bit Firewire link address
83 The second method to embed an address is to embed the address in each
85 data object that requires that address. The format of that address can
86 take one of two forms, depending on whether or not the address is being
87 defined or referenced. A defined address declares a new address that has
88 scope for the data object being embedded; a reader adds the address to
89 the end of a table so that it can be later referenced without having to
90 re-define the address. In this method, ID numbers start from zero. The
91 format of a defined address is:
92 Address length Type Address
94 +----------------+---------+----//-----+
95 | uint8_t | uint8_t | |
96 | value > 0 | | |
97 +----------------+---------+-----------+
98 The format of a referenced address is:
100 Magic value ID number
102 +--------------+------------+
103 | uint8_t | uint32_t |
104 | value == 0 | |
105 +--------------+------------+
106
108 Bytes, unsigned 16-bit integers, and unsigned 32 bit integers are embed‐
109 ded directly, using network byte order where appropriate. ASCII strings
110 are also embedded directly, including the trailing null byte to terminate
111 the string. Time values (timeval) are embedded as two unsigned 32 bit
112 integers; the first number counts the number of seconds that have elapsed
113 since the Unix epoch, the second number counts the number of microseconds
114 that have elapsed in the current second. Round-trip-time (RTT) values
115 are embedded as a single unsigned 32 bit integer that counts the number
116 of microseconds that elapsed.
117 Several measurements record ICMP extension data, so there is a standard‐
119 ised method to record a set of ICMP extensions. Individual ICMP exten‐
120 sion records are written in the following format: The format of a list
121 structure is:
122 - uint16_t: length of data that follows
123 - uint8_t: ICMP extension class number
124 - uint8_t: ICMP extension type number
125 - Bytes: ICMP extension data, if any.
126 A set of ICMP extension records is written in the following format:
128 Total Length Extension #1 .. Extension #N
130 +--------------+------//------+-------//------+
131 | uint16_t | | |
132 | | | |
133 +--------------+------//------+-------//------+
134
136 The format of a list structure is:
137 - 8 bytes: Warts header, type 0x0001
138 - uint32_t: List ID assigned by warts from a counter
139 - uint32_t: List ID assigned by a person.
140 - String: List Name assigned by a person
141 - Variable: Flags
142 - uint16_t: Parameter length (optional, included if any flags are set)
143 - String: Description, included if flag 1 is set
144 - String: Monitor name, included if flag 2 is set
145 The List ID assigned by warts is subsequently used by objects that refer‐
146 ence the list to identify which list they refer to.
147
149 Three types of cycle records may be written: a start record denoting the
150 starting point for a new cycle, a definition record declaring a cycle
151 record whose corresponding start record is in a different file, and a
152 cycle stop record, denoting the end point for a cycle. The format of the
153 cycle start and definition structures is:
154 - 8 bytes: Warts header, type 0x0002 or 0x0003
155 - uint32_t: Cycle ID, assigned by warts from a counter
156 - uint32_t: List ID, referencing the list this cycle is over
157 - uint32_t: Cycle ID, assigned by a human
158 - uint32_t: Start time of the cycle, seconds since Unix epoch
159 - Variable: Flags
160 - uint16_t: Parameter length, included if any flags are set
161 - uint32_t: Stop time of the cycle in seconds since Unix epoch,
162 included if flag 1 is set
163 - String: Hostname at cycle start point, included if flag 2 is set
164 The format of the cycle stop structure is:
166 - 8 bytes: Warts header, type 0x0004
167 - uint32_t: Cycle ID, assigned by warts from a counter, referencing
168 the cycle structure that is being updated.
169 - uint32_t: Stop time of the cycle, seconds since Unix epoch
170 - Variable: Flags. currently set to zero.
171
173 Traceroute structures consist of traceroute parameters, hop records, and
174 an optional series of additional data types for special types of tracer‐
175 oute invokation. The general form of a traceroute recorded in warts is
176 as follows:
177 - 8 bytes: Warts header, type 0x0006
178 - Variable: Flags describing traceroute parameters and high-level out‐
179 comes
180 - uint16_t: Parameter length, included if any flags are set
181 - Variable: Traceroute parameters, depending on flags
182 - uint16_t: Hop record count
183 - Variable: Hop records, if hop record count > 0
184 - Variable: Optional traceroute data; pmtud, doubletree
185 - uint16_t: End of traceroute record; value is zero.
186 The flags and data types that describe traceroute are as follows:
188 - uint32_t: List ID assigned by warts, included if flag 1 is set
189 - uint32_t: Cycle ID assigned by warts, included if flag 2 is set
190 - uint32_t: Src IP address ID assigned by warts, included if flag 3 is
191 set
192 - uint32_t: Dst IP address ID assigned by warts, included if flag 4 is
193 set
194 - timeval: Time traceroute commenced, included if flag 5 is set
195 - uint8_t: Stop reason, included if flag 6 is set
196 - uint8_t: Stop data, included if flag 7 is set
197 - uint8_t: Trace flags, included if flag 8 is set
198 - uint8_t: Attempts, included if flag 9 is set
199 - uint8_t: Hoplimit, included if flag 10 is set
200 - uint8_t: Trace type, included if flag 11 is set
201 - uint16_t: Probe size, included if flag 12 is set
202 - uint16_t: Source port, included if flag 13 is set
203 - uint16_t: Destination port, included if flag 14 is set
204 - uint8_t: TTL of first probe, included if flag 15 is set
205 - uint8_t: IP ToS set in probe packets, included if flag 16 is set
206 - uint8_t: Timeout length for each probe in seconds, included if flag
207 17 is set
208 - uint8_t: How many loops are allowed before probing halts, included
209 if flag 18 is set
210 - uint16_t: Number of hops probed, included if flag 19 is set
211 - uint8_t: Gap limit before probing halts, included if flag 20 is set
212 - uint8_t: What to do when the gap limit is reached, included if flag
213 21 is set
214 - uint8_t: What to do when a loop is found, included if flag 22 is set
215 - uint16_t: Number of probes sent, included if flag 23 is set
216 - uint8_t: Minimum time to wait between probes in centiseconds,
217 included if flag 24 is set
218 - uint8_t: Confidence level to attain that all hops have replied at a
219 given distance in the path, included if flag 25 is set
220 - address: Source address used in probes, included if flag 26 is set
221 - address: Destination address used in probes, included if flag 27 is
222 set
223 - uint32_t: User ID assigned to the traceroute, included if flag 28 is
224 set
225 The traceroute flags field has the following fields:
227 - If bit 1 is set, traceroute sent all allotted attempts.
228 - If bit 2 is set, traceroute was instructed to conduct path MTU dis‐
229 covery.
230 - If bit 3 is set, traceroute should use the datalink to obtain time‐
231 stamps.
232 - If bit 4 is set, traceroute should not halt probing if a TTL expired
233 message is received from the destination.
234 - If bit 5 is set, traceroute should use Doubletree to reduce redun‐
235 dant probing.
236 - If bit 6 is set, the ICMP checksum used in echo probes can be found
237 is stored where the UDP destination port value is.
238 Hop records are written in series. Each hop record takes the following
240 form:
241 - Variable: Flags describing which hop parameters are recorded
242 - uint16_t: Parameter length, included if any flags are set
243 - uint32_t: Hop address, ID corresponding to global warts address;
244 included if flag 1 is set
245 - uint8_t: IP TTL of probe packet, included if flag 2 is set
246 - uint8_t: IP TTL of reply packet, included if flag 3 is set
247 - uint8_t: Hop flags, included if flag 4 is set
248 - uint8_t: Hop probe ID - how many probes have been sent for the given
249 TTL. Included if flag 5 is set.
250 - RTT: Round trip time - the length of time in microseconds it took
251 this reply to arrive after the probe was transmitted. Included if
252 flag 6 is set.
253 - uint16_t: ICMP type, code. The first byte is the ICMP type of the
254 response, the second byte is the ICMP code. Included if flag 7 is
255 set.
256 - uint16_t: Probe size - the size of the probe sent. Included if flag
257 9 is set.
258 - uint16_t: Reply size - the size of the response received. Included
259 if flag 10 is set.
260 - uint16_t: IPID - the IP identifier value set in the response packet.
261 Included if flag 11 is set.
262 - uint8_t: Type of Service - the value of the ToS byte in the IP
263 header, including ECN bits. Included if flag 12 is set.
264 - uint16_t: Next-hop MTU - the value of the next-hop MTU field if the
265 response is an ICMP packet too big message. Included if flag 13 is
266 set.
267 - uint16_t: Quoted IP length - the value of the IP length field found
268 in the ICMP quotation. Included if flag 14 is set, else it is the
269 same as the probe size.
270 - uint8_t: Quoted TTL - the value of the IP TTL field found in the
271 ICMP quotation. Included if flag 15 is set, else it is one.
272 - uint8_t: TCP flags - the value of the TCP flags received in response
273 to TCP probes. Included if flag 16 is set.
274 - uint8_t: Quoted TOS - the value of the IP ToS byte found in the ICMP
275 quotation. Included if flag 17 is set.
276 - icmpext: ICMP extension data, included if flag 18 is set.
277 - address: Hop address, included if flag 19 is set.
278 Optional traceroute data, such as PMTUD and doubletree control and result
280 structures are included after hop records. Optional traceroute data
281 begins with a 16-bit header; the first four bits define the type of
282 record, and the remaining 12 bits specify the length of the record. Cur‐
283 rently, three types of optional data are defined: PMTUD data (1), Last-
284 ditch probing results (2), and doubletree (3).
285 The format of the last-ditch data is:
287 - uint16_t: traceroute optional data header, type = 1.
288 - Variable: Flags describing which last-ditch parameters are recorded.
289 Currently, no flags are defined.
290 - uint16_t: Parameter length, included if any flags are set.
291 - uint16_t: Number of responses received to last-ditch probing,
292 recorded as hop records.
293 - Variable: Hop records.
294 The format of PMTUD data is:
296 - uint16_t: traceroute optional data header, type = 2.
297 - Variable: PMTUD flags and parameters
298 - uint16_t: Number of hop records that follow
299 - Variable: Hop Records, if any
300 - Variable: Notes, if any
301 The format of the PMTUD flags and attributes is:
302 - Variable: Flags describing which hop parameters are recorded
303 - uint16_t: Parameter length, included if any flags are set
304 - uint16_t: MTU of the interface, included if flag 1 is set.
305 - uint16_t: Path MTU, included if flag 2 is set.
306 - uint16_t: MTU to the first hop, included if flag 3 is set and if it
307 differs to the MTU of the interface.
308 - uint8_t: verson of the PMTUD attribute, included if flag 4 is set,
309 otherwise version 1 can be assumed.
310 - uint8_t: note count - number of PMTUD note structures that follow
311 the hops.
312 The format of the PMTUD notes is:
313 - Variable: Flags describing which hop parameters are recorded
314 - uint16_t: Parameter length, included if any flags are set
315 - uint8_t: type of note, included if flag 1 is set.
316 - uint16_t: next-hop MTU inferred, included if flag 2 is set.
317 - uint16_t: Index of corresponding hop record in the PMTUD hops,
318 included if flag 3 is set.
319 The tree types of PMTUD notes are: ordinary PTB (1), PTB with invalid
320 next-hop MTU (2), and an inferred MTU in the absence of a PTB (3).
321
323 scamper(1), libscamperfile(3), sc_wartsdump(1),
324 M. Luckie, Scamper: a Scalable and Extensible Packet Prober for Active
326 Measurement of the Internet, Proc. ACM/SIGCOMM Internet Measurement
327 Conference 2010.
328
330 warts is written by Matthew Luckie <mjl@luckie.org.nz>.
331BSD May 16, 2011 BSD