1CARBON-C-RELAY(1) General Commands Manual CARBON-C-RELAY(1)
2
6 carbon-c-relay - graphite relay, aggregator and rewriter
7 https://travis-ci.org/grobian/carbon-c-relay
9
11 carbon-c-relay -f config-file [ options ... ]
12
14 carbon-c-relay accepts, cleanses, matches, rewrites, forwards and ag‐
15 gregates graphite metrics by listening for incoming connections and re‐
16 laying the messages to other servers defined in its configuration. The
17 core functionality is to route messages via flexible rules to the de‐
18 sired destinations.
19 carbon-c-relay is a simple program that reads its routing information
21 from a file. The command line arguments allow to set the location for
22 this file, as well as the amount of dispatchers (worker threads) to use
23 for reading the data from incoming connections and passing them onto
24 the right destination(s). The route file supports two main constructs:
25 clusters and matches. The first define groups of hosts data metrics can
26 be sent to, the latter define which metrics should be sent to which
27 cluster. Aggregation rules are seen as matches. Rewrites are actions
28 that directly affect the metric at the point in which they appear in
29 the configuration.
30 For every metric received by the relay, cleansing is performed. The
32 following changes are performed before any match, aggregate or rewrite
33 rule sees the metric:
34 ○ double dot elimination (necessary for correctly functioning consis‐
36 tent hash routing)
37 ○ trailing/leading dot elimination
39 ○ whitespace normalisation (this mostly affects output of the relay
41 to other targets: metric, value and timestamp will be separated by
42 a single space only, ever)
43 ○ irregular char replacement with underscores (_), currently irregu‐
45 lar is defined as not being in [0-9a-zA-Z-_:#], but can be overrid‐
46 den on the command line. Note that tags (when present and allowed)
47 are not processed this way.
48
52 These options control the behaviour of carbon-c-relay.
53 ○ -v: Print version string and exit.
55 ○ -d: Enable debug mode, this prints statistics to stdout and prints
57 extra messages about some situations encountered by the relay that
58 normally would be too verbose to be enabled. When combined with -t
59 (test mode) this also prints stub routes and consistent-hash ring
60 contents.
61 ○ -s: Enable submission mode. In this mode, internal statistics are
63 not generated. Instead, queue pressure and metrics drops are re‐
64 ported on stdout. This mode is useful when used as submission relay
65 which´ job is just to forward to (a set of) main relays. Statistics
66 about the submission relays in this case are not needed, and could
67 easily cause a non-desired flood of metrics e.g. when used on each
68 and every host locally.
69 ○ -t: Test mode. This mode doesn´t do any routing at all, but instead
71 reads input from stdin and prints what actions would be taken given
72 the loaded configuration. This mode is very useful for testing re‐
73 lay routes for regular expression syntax etc. It also allows to
74 give insight on how routing is applied in complex configurations,
75 for it shows rewrites and aggregates taking place as well. When -t
76 is repeated, the relay will only test the configuration for valid‐
77 ity and exit immediately afterwards. Any standard output is sup‐
78 pressed in this mode, making it ideal for start-scripts to test a
79 (new) configuration.
80 ○ -f config-file: Read configuration from config-file. A configura‐
82 tion consists of clusters and routes. See CONFIGURATION SYNTAX for
83 more information on the options and syntax of this file.
84 ○ -l log-file: Use log-file for writing messages. Without this op‐
86 tion, the relay writes both to stdout and stderr. When logging to
87 file, all messages are prefixed with MSG when they were sent to
88 stdout, and ERR when they were sent to stderr.
89 ○ -p port: Listen for connections on port port. The port number is
91 used for both TCP, UDP and UNIX sockets. In the latter case, the
92 socket file contains the port number. The port defaults to 2003,
93 which is also used by the original carbon-cache.py. Note that this
94 only applies to the defaults, when listen directives are in the
95 config, this setting is ignored.
96 ○ -w workers: Use workers number of threads. The default number of
98 workers is equal to the amount of detected CPU cores. It makes
99 sense to reduce this number on many-core machines, or when the
100 traffic is low.
101 ○ -b batchsize: Set the amount of metrics that sent to remote servers
103 at once to batchsize. When the relay sends metrics to servers, it
104 will retrieve batchsize metrics from the pending queue of metrics
105 waiting for that server and send those one by one. The size of the
106 batch will have minimal impact on sending performance, but it con‐
107 trols the amount of lock-contention on the queue. The default is
108 2500.
109 ○ -q queuesize: Each server from the configuration where the relay
111 will send metrics to, has a queue associated with it. This queue
112 allows for disruptions and bursts to be handled. The size of this
113 queue will be set to queuesize which allows for that amount of met‐
114 rics to be stored in the queue before it overflows, and the relay
115 starts dropping metrics. The larger the queue, more metrics can be
116 absorbed, but also more memory will be used by the relay. The de‐
117 fault queue size is 25000.
118 ○ -L stalls: Sets the max mount of stalls to stalls before the relay
120 starts dropping metrics for a server. When a queue fills up, the
121 relay uses a mechanism called stalling to signal the client (writ‐
122 ing to the relay) of this event. In particular when the client
123 sends a large amount of metrics in very short time (burst),
124 stalling can help to avoid dropping metrics, since the client just
125 needs to slow down for a bit, which in many cases is possible (e.g.
126 when catting a file with nc(1)). However, this behaviour can also
127 obstruct, artificially stalling writers which cannot stop that eas‐
128 ily. For this the stalls can be set from 0 to 15, where each stall
129 can take around 1 second on the client. The default value is set to
130 4, which is aimed at the occasional disruption scenario and max ef‐
131 fort to not loose metrics with moderate slowing down of clients.
132 ○ -C CAcertpath: Read CA certs (for use with TLS/SSL connections)
134 from given path or file. When not given, the default locations are
135 used. Strict verfication of the peer is performed, so when using
136 self-signed certificates, be sure to include the CA cert in the de‐
137 fault location, or provide the path to the cert using this option.
138 ○ -T timeout: Specifies the IO timeout in milliseconds used for
140 server connections. The default is 600 milliseconds, but may need
141 increasing when WAN links are used for target servers. A relatively
142 low value for connection timeout allows the relay to quickly estab‐
143 lish a server is unreachable, and as such failover strategies to
144 kick in before the queue runs high.
145 ○ -c chars: Defines the characters that are next to [A-Za-z0-9] al‐
147 lowed in metrics to chars. Any character not in this list, is re‐
148 placed by the relay with _ (underscore). The default list of al‐
149 lowed characters is -_:#.
150 ○ -m length: Limits the metric names to be of at most length bytes
152 long. Any lines containing metric names larger than this will be
153 discarded.
154 ○ -M length Limits the input to lines of at most length bytes. Any
156 excess lines will be discarded. Note that -m needs to be smaller
157 than this value.
158 ○ -H hostname: Override hostname determined by a call to gethost‐
160 name(3) with hostname. The hostname is used mainly in the statis‐
161 tics metrics carbon.relays.<hostname>.<...> sent by the relay.
162 ○ -B backlog: Sets TCP connection listen backlog to backlog connec‐
164 tions. The default value is 32 but on servers which receive many
165 concurrent connections, this setting likely needs to be increased
166 to avoid connection refused errors on the clients.
167 ○ -U bufsize: Sets the socket send/receive buffer sizes in bytes, for
169 both TCP and UDP scenarios. When unset, the OS default is used. The
170 maximum is also determined by the OS. The sizes are set using set‐
171 sockopt with the flags SO_RCVBUF and SO_SNDBUF. Setting this size
172 may be necessary for large volume scenarios, for which also -B
173 might apply. Checking the Recv-Q and the receive errors values from
174 netstat gives a good hint about buffer usage.
175 ○ -E: Disable disconnecting idle incoming connections. By default the
177 relay disconnects idle client connections after 10 minutes. It does
178 this to prevent resources clogging up when a faulty or malicious
179 client keeps on opening connections without closing them. It typi‐
180 cally prevents running out of file descriptors. For some scenarios,
181 however, it is not desirable for idle connections to be discon‐
182 nected, hence passing this flag will disable this behaviour.
183 ○ -D: Deamonise into the background after startup. This option re‐
185 quires -l and -P flags to be set as well.
186 ○ -P pidfile: Write the pid of the relay process to a file called
188 pidfile. This is in particular useful when daemonised in combina‐
189 tion with init managers.
190 ○ -O threshold: The minimum number of rules to find before trying to
192 optimise the ruleset. The default is 50, to disable the optimiser,
193 use -1, to always run the optimiser use 0. The optimiser tries to
194 group rules to avoid spending excessive time on matching expres‐
195 sions.
196
200 The config file supports the following syntax, where comments start
201 with a # character and can appear at any position on a line and sup‐
202 press input until the end of that line:
203 ``` cluster name <forward | any_of | failover [useall] | carbon_ch |
205 fnv1a_ch | jump_fnv1a_ch [replication count] [dynamic] >
206 <host[:port][=instance] [proto udp | tcp] [type linemode] [transport
207 plain | gzip | lz4 | snappy [ssl]]> ... ;
208 cluster name file [ip] </path/to/file> ... ;
210 match <* | expression ...> [validate expression else log | drop] send
212 to <cluster ... | blackhole> [stop] ;
213 rewrite expression into replacement ;
215 aggregate expression ... every interval seconds expire after expiration
217 seconds [timestamp at start | middle | end of bucket] compute sum |
218 count | max | min | average | median | percentile<% | variance | std‐
219 dev> write to metric [compute ...] [send to <cluster ...>] [stop] ;
220 send statistics to <cluster ...> [stop] ; statistics [submit every in‐
222 terval seconds] [reset counters after interval] [prefix with prefix]
223 [send to <cluster ...>] [stop] ;
224 listen type linemode [transport plain | gzip | lz4 | snappy [ssl pem‐
226 cert]] «interface[:port] | port> proto udp | tcp> ... </ptah/to/file
227 proto unix> ... ;
228 include </path/to/file/or/glob> ; ```
230 CLUSTERS
232 Multiple clusters can be defined, and need not to be referenced by a
233 match rule. All clusters point to one or more hosts, except the file
234 cluster which writes to files in the local filesystem. host may be an
235 IPv4 or IPv6 address, or a hostname. Since host is followed by an op‐
236 tional : and port, for IPv6 addresses not to be interpreted wrongly,
237 either a port must be given, or the IPv6 address surrounded by brack‐
238 ets, e.g. [::1]. Optional transport and proto clauses can be used to
239 wrap the connection in a compression or encryption later or specify the
240 use of UDP or TCP to connect to the remote server. When omitted the
241 connection defaults to an unwrapped TCP connection. type can only be
242 linemode at the moment.
243 DNS hostnames are resolved to a single address, according to the pref‐
245 erence rules in RFC 3484 https://www.ietf.org/rfc/rfc3484.txt. The
246 any_of, failover and forward clusters have an explicit useall flag that
247 enables expansion for hostnames resolving to multiple addresses. Using
248 this option Each address returned becomes a cluster destination.
249 There are two groups of cluster types, simple forwarding clusters and
251 consistent hashing clusters.
252 ○ forward and file clusters
254 The forward and file clusters simply send everything they receive
256 to the defined members (host addresses or files). When a cluster
257 has multiple members, all incoming metrics are sent to /all/ mem‐
258 bers, basically duplicating the input metric stream over all mem‐
259 bers.
260 ○ any_of cluster
262 The any_of cluster is a small variant of the forward cluster, but
264 instead of sending the input metrics to all defined members, it
265 sends each incoming metric to only one of defined members. The pur‐
266 pose of this is a load-balanced scenario where any of the members
267 can receive any metric. As any_of suggests, when any of the members
268 become unreachable, the remaining available members will immedi‐
269 ately receive the full input stream of metrics. This specifically
270 mean that when 4 members are used, each will receive approximately
271 25% of the input metrics. When one member becomes unavailable (e.g.
272 network interruption, or a restart of the service), the remaining 3
273 members will each receive about 33% of the input. When designing
274 cluster capacity, one should take into account that in the most ex‐
275 treme case, the final remaining member will receive all input traf‐
276 fic.
277 An any_of cluster can in particular be useful when the cluster
279 points to other relays or caches. When used with other relays, it
280 effectively load-balances, and adapts immediately over inavailabil‐
281 ity of targets. When used with caches, the behaviour of the any_of
282 router to send the same metrics consistently to the same destina‐
283 tion helps caches to have a high hitrate on their internal buffers
284 for the same metrics (if they use them), but still allows for a
285 rolling-restart of the caches when e.g. on the same machine.
286 ○ failover cluster
288 The failover cluster is like the any_of cluster, but sticks to the
290 order in which servers are defined. This is to implement a pure
291 failover scenario between servers. All metrics are sent to at most
292 1 member, so no hashing or balancing is taking place. A failover
293 cluster with two members will only send metrics to the second mem‐
294 ber if the first becomes unavailable.
295 ○ carbon_ch cluster
297 The carbon_ch cluster sends the metrics to the member that is re‐
299 sponsible according to the consistent hash algorithm, as used in
300 the original carbon python relay, or multiple members if replica‐
301 tion is set to more than 1. When dynamic is set, failure of any of
302 the servers does not result in metrics being dropped for that
303 server, but instead the undeliverable metrics are sent to any other
304 server in the cluster in order for the metrics not to get lost.
305 This is most useful when replication is 1.
306 The calculation of the hashring, that defines the way in which met‐
308 rics are distributed, is based on the server host (or IP address)
309 and the optional instance of the member. This means that using car‐
310 bon_ch two targets on different ports but on the same host will map
311 to the same hashkey, which means no distribution of metrics takes
312 place. The instance is used to remedy that situation. An instance
313 is appended to the memeber after the port, and separated by an
314 equals sign, e.g. 127.0.0.1:2006=a for instance a.
315 Consistent hashes are consistent in the sense that removal of a
317 member from the cluster should not result in a complete re-mapping
318 of all metrics to members, but instead only add the metrics from
319 the removed member to all remaining members, approximately each
320 gets its fair share. The other way around, when a member is added,
321 each member should see a subset of its metrics now being addressed
322 to the new member. This is an important advantage over a normal
323 hash, where each removal or addition of members (also via e.g. a
324 change in their IP address or hostname) would cause a full re-map‐
325 ping of all metrics over all available metrics.
326 ○ fnv1a_ch cluster
328 The fnv1a_ch cluster is a identical in behaviour to carbon_ch, but
330 it uses a different hash technique (FNV1a) which is faster but more
331 importantly defined to get by the beforementioned limitation of
332 carbon_ch to use both host and port from the members. This is use‐
333 ful when multiple targets live on the same host just separated by
334 port.
335 Since the instance property is no longer necessary with fnv1a_ch
337 this way, this cluster type uses it to completely override the
338 string that the hashkey should be calculated off. This allows for
339 many things, including masquerading old IP addresses, but it basi‐
340 cally can be used to make the hash key location agnostic of the
341 (physical) location of that key. For example, usage like
342 10.0.0.1:2003=4d79d13554fa1301476c1f9fe968b0ac would allow to
343 change port and/or ip address of the server that receives data for
344 the instance key. Obviously, this way migration of data can be
345 dealt with much more conveniently. Note that since the instance
346 name is used as full hash input, instances as a, b, etc. will
347 likely result in poor hash distribution, since their hashes have
348 very little input. Consider using longer and mostly differing in‐
349 stance names such as random hashes for better hash distribution be‐
350 haviour.
351 ○ jump_fnv1a_ch cluster
353 The jump_fnv1a_ch cluster is also a consistent hash cluster like
355 the previous two, but it does not take the member host, port or in‐
356 stance into account at all. Whether this is useful to you depends
357 on your scenario. The jump hash has almost perfect balancing over
358 the members defined in the cluster, at the expense of not being
359 able to remove any member but the last in order as defined in the
360 cluster. What this means is that this hash is fine to use with ever
361 growing clusters where older nodes are never removed.
362 If you have a cluster where removal of old nodes takes place often,
364 the jump hash is not suitable for you. Jump hash works with servers
365 in an ordered list without gaps. To influence the ordering, the in‐
366 stance given to the server will be used as sorting key. Without,
367 the order will be as given in the file. It is a good practice to
368 fix the order of the servers with instances such that it is ex‐
369 plicit what the right nodes for the jump hash are.
370 MATCHES
374 Match rules are the way to direct incoming metrics to one or more clus‐
375 ters. Match rules are processed top to bottom as they are defined in
376 the file. It is possible to define multiple matches in the same rule.
377 Each match rule can send data to one or more clusters. Since match
378 rules "fall through" unless the stop keyword is added, carefully
379 crafted match expression can be used to target multiple clusters or ag‐
380 gregations. This ability allows to replicate metrics, as well as send
381 certain metrics to alternative clusters with careful ordering and usage
382 of the stop keyword. The special cluster blackhole discards any metrics
383 sent to it. This can be useful for weeding out unwanted metrics in cer‐
384 tain cases. Because throwing metrics away is pointless if other matches
385 would accept the same data, a match with as destination the blackhole
386 cluster, has an implicit stop. The validation clause adds a check to
387 the data (what comes after the metric) in the form of a regular expres‐
388 sion. When this expression matches, the match rule will execute as if
389 no validation clause was present. However, if it fails, the match rule
390 is aborted, and no metrics will be sent to destinations, this is the
391 drop behaviour. When log is used, the metric is logged to stderr. Care
392 should be taken with the latter to avoid log flooding. When a validate
393 clause is present, destinations need not to be present, this allows for
394 applying a global validation rule. Note that the cleansing rules are
395 applied before validation is done, thus the data will not have dupli‐
396 cate spaces. The route using clause is used to perform a temporary mod‐
397 ification to the key used for input to the consistent hashing routines.
398 The primary purpose is to route traffic so that appropriate data is
399 sent to the needed aggregation instances.
400 REWRITES
402 Rewrite rules take a regular expression as input to match incoming met‐
403 rics, and transform them into the desired new metric name. In the re‐
404 placement, backreferences are allowed to match capture groups defined
405 in the input regular expression. A match of server\.(x|y|z)\. allows to
406 use e.g. role.\1. in the substitution. A few caveats apply to the cur‐
407 rent implementation of rewrite rules. First, their location in the con‐
408 fig file determines when the rewrite is performed. The rewrite is done
409 in-place, as such a match rule before the rewrite would match the orig‐
410 inal name, a match rule after the rewrite no longer matches the origi‐
411 nal name. Care should be taken with the ordering, as multiple rewrite
412 rules in succession can take place, e.g. a gets replaced by b and b
413 gets replaced by c in a succeeding rewrite rule. The second caveat with
414 the current implementation, is that the rewritten metric names are not
415 cleansed, like newly incoming metrics are. Thus, double dots and poten‐
416 tial dangerous characters can appear if the replacement string is
417 crafted to produce them. It is the responsibility of the writer to make
418 sure the metrics are clean. If this is an issue for routing, one can
419 consider to have a rewrite-only instance that forwards all metrics to
420 another instance that will do the routing. Obviously the second in‐
421 stance will cleanse the metrics as they come in. The backreference no‐
422 tation allows to lowercase and uppercase the replacement string with
423 the use of the underscore (_) and carret (^) symbols following directly
424 after the backslash. For example, role.\_1. as substitution will lower‐
425 case the contents of \1. The dot (.) can be used in a similar fashion,
426 or followed after the underscore or caret to replace dots with under‐
427 scores in the substitution. This can be handy for some situations where
428 metrics are sent to graphite.
429 AGGREGATIONS
431 The aggregations defined take one or more input metrics expressed by
432 one or more regular expresions, similar to the match rules. Incoming
433 metrics are aggregated over a period of time defined by the interval in
434 seconds. Since events may arrive a bit later in time, the expiration
435 time in seconds defines when the aggregations should be considered fi‐
436 nal, as no new entries are allowed to be added any more. On top of an
437 aggregation multiple aggregations can be computed. They can be of the
438 same or different aggregation types, but should write to a unique new
439 metric. The metric names can include back references like in rewrite
440 expressions, allowing for powerful single aggregation rules that yield
441 in many aggregations. When no send to clause is given, produced metrics
442 are sent to the relay as if they were submitted from the outside, hence
443 match and aggregation rules apply to those. Care should be taken that
444 loops are avoided this way. For this reason, the use of the send to
445 clause is encouraged, to direct the output traffic where possible. Like
446 for match rules, it is possible to define multiple cluster targets.
447 Also, like match rules, the stop keyword applies to control the flow of
448 metrics in the matching process.
449 STATISTICS
451 The send statistics to construct is deprecated and will be removed in
452 the next release. Use the special statistics construct instead.
453 The statistics construct can control a couple of things about the (in‐
455 ternal) statistics produced by the relay. The send to target can be
456 used to avoid router loops by sending the statistics to a certain des‐
457 tination cluster(s). By default the metrics are prefixed with car‐
458 bon.relays.<hostname>, where hostname is determinted on startup and can
459 be overridden using the -H argument. This prefix can be set using the
460 prefix with clause similar to a rewrite rule target. The input match in
461 this case is the pre-set regular expression ^(([^.]+)(\..*)?)$ on the
462 hostname. As such, one can see that the default prefix is set by car‐
463 bon.relays.\.1. Note that this uses the replace-dot-with-underscore re‐
464 placement feature from rewrite rules. Given the input expression, the
465 following match groups are available: \1 the entire hostname, \2 the
466 short hostname and \3 the domainname (with leading dot). It may make
467 sense to replace the default by something like carbon.relays.\_2 for
468 certain scenarios, to always use the lowercased short hostname, which
469 following the expression doesn´t contain a dot. By default, the metrics
470 are submitted every 60 seconds, this can be changed using the submit
471 every <interval> seconds clause.
472 To obtain a more compatible set of values to carbon-cache.py, use the
473 reset counters after interval clause to make values non-cumulative,
474 that is, they will report the change compared to the previous value.
475 LISTENERS
477 The ports and protocols the relay should listen for incoming connec‐
478 tions can be specified using the listen directive. Currently, all lis‐
479 teners need to be of linemode type. An optional compression or encryp‐
480 tion wrapping can be specified for the port and optional interface
481 given by ip address, or unix socket by file. When interface is not
482 specified, the any interface on all available ip protocols is assumed.
483 If no listen directive is present, the relay will use the default lis‐
484 teners for port 2003 on tcp and udp, plus the unix socket /tmp/.s.car‐
485 bon-c-relay.2003. This typically expands to 5 listeners on an IPv6 en‐
486 abled system. The default matches the behaviour of versions prior to
487 v3.2.
488 INCLUDES
490 In case configuration becomes very long, or is managed better in sepa‐
491 rate files, the include directive can be used to read another file. The
492 given file will be read in place and added to the router configuration
493 at the time of inclusion. The end result is one big route configura‐
494 tion. Multiple include statements can be used throughout the configura‐
495 tion file. The positioning will influence the order of rules as normal.
496 Beware that recursive inclusion (include from an included file) is sup‐
497 ported, and currently no safeguards exist for an inclusion loop. For
498 what is worth, this feature likely is best used with simple configura‐
499 tion files (e.g. not having include in them).
500
502 carbon-c-relay evolved over time, growing features on demand as the
503 tool proved to be stable and fitting the job well. Below follow some
504 annotated examples of constructs that can be used with the relay.
505 Clusters can be defined as much as necessary. They receive data from
507 match rules, and their type defines which members of the cluster fi‐
508 nally get the metric data. The simplest cluster form is a forward clus‐
509 ter:
510 cluster send-through forward 10.1.0.1 ;
512 Any metric sent to the send-through cluster would simply be forwarded
514 to the server at IPv4 address 10.1.0.1. If we define multiple servers,
515 all of those servers would get the same metric, thus:
516 cluster send-through forward 10.1.0.1 10.2.0.1 ;
518 The above results in a duplication of metrics send to both machines.
520 This can be useful, but most of the time it is not. The any_of cluster
521 type is like forward, but it sends each incoming metric to any of the
522 members. The same example with such cluster would be:
523 cluster send-to-any-one any_of 10.1.0.1:2010 10.1.0.1:2011;
525 This would implement a multipath scenario, where two servers are used,
527 the load between them is spread, but should any of them fail, all met‐
528 rics are sent to the remaining one. This typically works well for up‐
529 stream relays, or for balancing carbon-cache processes running on the
530 same machine. Should any member become unavailable, for instance due to
531 a rolling restart, the other members receive the traffic. If it is nec‐
532 essary to have true fail-over, where the secondary server is only used
533 if the first is down, the following would implement that:
534 cluster try-first-then-second failover 10.1.0.1:2010 10.1.0.1:2011;
536 These types are different from the two consistent hash cluster types:
538 cluster graphite carbon_ch 127.0.0.1:2006=a 127.0.0.1:2007=b
540 127.0.0.1:2008=c ;
541 If a member in this example fails, all metrics that would go to that
543 member are kept in the queue, waiting for the member to return. This is
544 useful for clusters of carbon-cache machines where it is desirable that
545 the same metric ends up on the same server always. The carbon_ch clus‐
546 ter type is compatible with carbon-relay consistent hash, and can be
547 used for existing clusters populated by carbon-relay. For new clusters,
548 however, it is better to use the fnv1a_ch cluster type, for it is
549 faster, and allows to balance over the same address but different ports
550 without an instance number, in constrast to carbon_ch.
551 Because we can use multiple clusters, we can also replicate without the
553 use of the forward cluster type, in a more intelligent way:
554 ``` cluster dc-old carbon_ch replication 2 10.1.0.1 10.1.0.2 10.1.0.3 ;
556 cluster dc-new1 fnv1a_ch replication 2 10.2.0.1 10.2.0.2 10.2.0.3 ;
557 cluster dc-new2 fnv1a_ch replication 2 10.3.0.1 10.3.0.2 10.3.0.3 ;
558 match * send to dc-old ; match * send to dc-new1 dc-new2 stop ; ```
560 In this example all incoming metrics are first sent to dc-old, then
562 dc-new1 and finally to dc-new2. Note that the cluster type of dc-old is
563 different. Each incoming metric will be send to 2 members of all three
564 clusters, thus replicating to in total 6 destinations. For each cluster
565 the destination members are computed independently. Failure of clusters
566 or members does not affect the others, since all have individual
567 queues. The above example could also be written using three match rules
568 for each dc, or one match rule for all three dcs. The difference is
569 mainly in performance, the number of times the incoming metric has to
570 be matched against an expression. The stop rule in dc-new match rule is
571 not strictly necessary in this example, because there are no more fol‐
572 lowing match rules. However, if the match would target a specific sub‐
573 set, e.g. ^sys\., and more clusters would be defined, this could be
574 necessary, as for instance in the following abbreviated example:
575 ``` cluster dc1-sys ... ; cluster dc2-sys ... ;
577 cluster dc1-misc ... ; cluster dc2-misc ... ;
579 match ^sys. send to dc1-sys; match ^sys. send to dc2-sys stop;
581 match * send to dc1-misc; match * send to dc2-misc stop; ```
583 As can be seen, without the stop in dc2-sys´ match rule, all metrics
585 starting with sys. would also be send to dc1-misc and dc2-misc. It can
586 be that this is desired, of course, but in this example there is a ded‐
587 icated cluster for the sys metrics.
588 Suppose there would be some unwanted metric that unfortunately is gen‐
590 erated, let´s assume some bad/old software. We don´t want to store this
591 metric. The blackhole cluster is suitable for that, when it is harder
592 to actually whitelist all wanted metrics. Consider the following:
593 match some_legacy1$ some_legacy2$ send to blackhole stop;
595 This would throw away all metrics that end with some_legacy, that would
597 otherwise be hard to filter out. Since the order matters, it can be
598 used in a construct like this:
599 ``` cluster old ... ; cluster new ... ;
601 match * send to old;
603 match unwanted send to blackhole stop;
605 match * send to new; ```
607 In this example the old cluster would receive the metric that´s un‐
609 wanted for the new cluster. So, the order in which the rules occur does
610 matter for the execution.
611 Validation can be used to ensure the data for metrics is as expected. A
613 global validation for just number (no floating point) values could be:
614 match * validate ^[0-9]+\ [0-9]+$ else drop ;
616 (Note the escape with backslash \ of the space, you might be able to
618 use \s or [:space:] instead, this depends on your configured regex im‐
619 plementation.)
620 The validation clause can exist on every match rule, so in principle,
622 the following is valid:
623 match ^foo validate ^[0-9]+\ [0-9]+$ else drop send to integer-cluster
625 ; match ^foo validate ^[0-9.e+-]+\ [0-9.e+-]+$ else drop send to
626 float-cluster stop;
627 Note that the behaviour is different in the previous two examples. When
629 no send to clusters are specified, a validation error makes the match
630 behave like the stop keyword is present. Likewise, when validation
631 passes, processing continues with the next rule. When destination clus‐
632 ters are present, the match respects the stop keyword as normal. When
633 specified, processing will always stop when specified so. However, if
634 validation fails, the rule does not send anything to the destination
635 clusters, the metric will be dropped or logged, but never sent.
636 The relay is capable of rewriting incoming metrics on the fly. This
638 process is done based on regular expressions with capture groups that
639 allow to substitute parts in a replacement string. Rewrite rules allow
640 to cleanup metrics from applications, or provide a migration path. In
641 it´s simplest form a rewrite rule looks like this:
642 rewrite ^server\.(.+)\.(.+)\.([a-zA-Z]+)([0-9]+) into
644 server.\_1.\2.\3.\3\4 ;
645 In this example a metric like server.DC.role.name123 would be trans‐
647 formed into server.dc.role.name.name123. For rewrite rules hold the
648 same as for matches, that their order matters. Hence to build on top of
649 the old/new cluster example done earlier, the following would store the
650 original metric name in the old cluster, and the new metric name in the
651 new cluster:
652 ``` match * send to old;
654 rewrite ... ;
656 match * send to new; ```
658 Note that after the rewrite, the original metric name is no longer
660 available, as the rewrite happens in-place.
661 Aggregations are probably the most complex part of carbon-c-relay. Two
663 ways of specifying aggregates are supported by carbon-c-relay. The
664 first, static rules, are handled by an optimiser which tries to fold
665 thousands of rules into groups to make the matching more efficient. The
666 second, dynamic rules, are very powerful compact definitions with pos‐
667 sibly thousands of internal instantiations. A typical static aggrega‐
668 tion looks like:
669 aggregate ^sys\.dc1\.somehost-[0-9]+\.somecluster\.mysql\.replica‐
671 tion_delay ^sys\.dc2\.somehost-[0-9]+\.somecluster\.mysql\.replica‐
672 tion_delay every 10 seconds expire after 35 seconds timestamp at end of
673 bucket compute sum write to mysql.somecluster.total_replication_delay
674 compute average write to mysql.somecluster.average_replication_delay
675 compute max write to mysql.somecluster.max_replication_delay compute
676 count write to mysql.somecluster.replication_delay_metric_count ;
677 In this example, four aggregations are produced from the incoming
679 matching metrics. In this example we could have written the two matches
680 as one, but for demonstration purposes we did not. Obviously they can
681 refer to different metrics, if that makes sense. The every 10 seconds
682 clause specifies in what interval the aggregator can expect new metrics
683 to arrive. This interval is used to produce the aggregations, thus each
684 10 seconds 4 new metrics are generated from the data received sofar.
685 Because data may be in transit for some reason, or generation stalled,
686 the expire after clause specifies how long the data should be kept be‐
687 fore considering a data bucket (which is aggregated) to be complete. In
688 the example, 35 was used, which means after 35 seconds the first aggre‐
689 gates are produced. It also means that metrics can arrive 35 seconds
690 late, and still be taken into account. The exact time at which the ag‐
691 gregate metrics are produced is random between 0 and interval (10 in
692 this case) seconds after the expiry time. This is done to prevent thun‐
693 dering herds of metrics for large aggregation sets. The timestamp that
694 is used for the aggregations can be specified to be the start, middle
695 or end of the bucket. Original carbon-aggregator.py uses start, while
696 carbon-c-relay´s default has always been end. The compute clauses
697 demonstrate a single aggregation rule can produce multiple aggregates,
698 as often is the case. Internally, this comes for free, since all possi‐
699 ble aggregates are always calculated, whether or not they are used. The
700 produced new metrics are resubmitted to the relay, hence matches de‐
701 fined before in the configuration can match output of the aggregator.
702 It is important to avoid loops, that can be generated this way. In gen‐
703 eral, splitting aggregations to their own carbon-c-relay instance, such
704 that it is easy to forward the produced metrics to another relay in‐
705 stance is a good practice.
706 The previous example could also be written as follows to be dynamic:
708 aggregate ^sys\.dc[0-9].(somehost-[0-9]+)\.([^.]+)\.mysql\.replica‐
710 tion_delay every 10 seconds expire after 35 seconds compute sum write
711 to mysql.host.\1.replication_delay compute sum write to
712 mysql.host.all.replication_delay compute sum write to mysql.clus‐
713 ter.\2.replication_delay compute sum write to mysql.cluster.all.repli‐
714 cation_delay ;
715 Here a single match, results in four aggregations, each of a different
717 scope. In this example aggregation based on hostname and cluster are
718 being made, as well as the more general all targets, which in this ex‐
719 ample have both identical values. Note that with this single aggrega‐
720 tion rule, both per-cluster, per-host and total aggregations are pro‐
721 duced. Obviously, the input metrics define which hosts and clusters are
722 produced.
723 With use of the send to clause, aggregations can be made more intuitive
725 and less error-prone. Consider the below example:
726 ``` cluster graphite fnv1a_ch ip1 ip2 ip3;
728 aggregate ^sys.somemetric every 60 seconds expire after 75 seconds com‐
730 pute sum write to sys.somemetric send to graphite stop ;
731 match * send to graphite; ```
733 It sends all incoming metrics to the graphite cluster, except the
735 sys.somemetric ones, which it replaces with a sum of all the incoming
736 ones. Without a stop in the aggregate, this causes a loop, and without
737 the send to, the metric name can´t be kept its original name, for the
738 output now directly goes to the cluster.
739 When configuring cluster you might want to check how the metrics will
741 be routed and hashed. That´s what the -t flag is for. For the following
742 configuration : ``` cluster graphite_swarm_odd fnv1a_ch replication 1
743 host01.dom:2003=31F7A65E315586AC198BD798B6629CE4903D089947
744 host03.dom:2003=9124E29E0C92EB63B3834C1403BD2632AA7508B740
745 host05.dom:2003=B653412CD96B13C797658D2C48D952AEC3EB667313 ;
746 cluster graphite_swarm_even fnv1a_ch replication 1
748 host02.dom:2003=31F7A65E315586AC198BD798B6629CE4903D089947
749 host04.dom:2003=9124E29E0C92EB63B3834C1403BD2632AA7508B740
750 host06.dom:2003=B653412CD96B13C797658D2C48D952AEC3EB667313
751 ;
753 match * send to graphite_swarm_odd graphite_swarm_even stop ; Running
755 the command : `echo "my.super.metric" | carbon-c-relay -f config.conf
756 -t`, will result in : [...] match * -> my.super.metric
757 fnv1a_ch(graphite_swarm_odd) host03.dom:2003
758 fnv1a_ch(graphite_swarm_even) host04.dom:2003 stop
759 ``` You now know that your metric my.super.metric will be hashed and
761 arrive on the host03 and host04 machines. Adding the -d flag will in‐
762 crease the amount of information by showing you the hashring
763
765 When carbon-c-relay is run without -d or -s arguments, statistics will
766 be produced. By default they are sent to the relay itself in the form
767 of carbon.relays.<hostname>.*. See the statistics construct to override
768 this prefix, sending interval and values produced. While many metrics
769 have a similar name to what carbon-cache.py would produce, their values
770 are likely different. By default, most values are running counters
771 which only increase over time. The use of the nonNegativeDerivative()
772 function from graphite is useful with these.
773 The following metrics are produced under the carbon.relays.<hostname>
775 namespace:
776 ○ metricsReceived
778 The number of metrics that were received by the relay. Received
780 here means that they were seen and processed by any of the dis‐
781 patchers.
782 ○ metricsSent
784 The number of metrics that were sent from the relay. This is a to‐
786 tal count for all servers combined. When incoming metrics are du‐
787 plicated by the cluster configuration, this counter will include
788 all those duplications. In other words, the amount of metrics that
789 were successfully sent to other systems. Note that metrics that are
790 processed (received) but still in the sending queue (queued) are
791 not included in this counter.
792 ○ metricsDiscarded
794 The number of input lines that were not considered to be a valid
796 metric. Such lines can be empty, only containing whitespace, or
797 hitting the limits given for max input length and/or max metric
798 length (see -m and -M options).
799 ○ metricsQueued
801 The total number of metrics that are currently in the queues for
803 all the server targets. This metric is not cumulative, for it is a
804 sample of the queue size, which can (and should) go up and down.
805 Therefore you should not use the derivative function for this met‐
806 ric.
807 ○ metricsDropped
809 The total number of metric that had to be dropped due to server
811 queues overflowing. A queue typically overflows when the server it
812 tries to send its metrics to is not reachable, or too slow in in‐
813 gesting the amount of metrics queued. This can be network or re‐
814 source related, and also greatly depends on the rate of metrics be‐
815 ing sent to the particular server.
816 ○ metricsBlackholed
818 The number of metrics that did not match any rule, or matched a
820 rule with blackhole as target. Depending on your configuration, a
821 high value might be an indication of a misconfiguration somewhere.
822 These metrics were received by the relay, but never sent anywhere,
823 thus they disappeared.
824 ○ metricStalls
826 The number of times the relay had to stall a client to indicate
828 that the downstream server cannot handle the stream of metrics. A
829 stall is only performed when the queue is full and the server is
830 actually receptive of metrics, but just too slow at the moment.
831 Stalls typically happen during micro-bursts, where the client typi‐
832 cally is unaware that it should stop sending more data, while it is
833 able to.
834 ○ connections
836 The number of connect requests handled. This is an ever increasing
838 number just counting how many connections were accepted.
839 ○ disconnects
841 The number of disconnected clients. A disconnect either happens be‐
843 cause the client goes away, or due to an idle timeout in the relay.
844 The difference between this metric and connections is the amount of
845 connections actively held by the relay. In normal situations this
846 amount remains within reasonable bounds. Many connections, but few
847 disconnections typically indicate a possible connection leak in the
848 client. The idle connections disconnect in the relay here is to
849 guard against resource drain in such scenarios.
850 ○ dispatch_wallTime_us
852 The number of microseconds spent by the dispatchers to do their
854 work. In particular on multi-core systems, this value can be con‐
855 fusing, however, it indicates how long the dispatchers were doing
856 work handling clients. It includes everything they do, from reading
857 data from a socket, cleaning up the input metric, to adding the
858 metric to the appropriate queues. The larger the configuration, and
859 more complex in terms of matches, the more time the dispatchers
860 will spend on the cpu. But also time they do /not/ spend on the cpu
861 is included in this number. It is the pure wallclock time the dis‐
862 patcher was serving a client.
863 ○ dispatch_sleepTime_us
865 The number of microseconds spent by the dispatchers sleeping wait‐
867 ing for work. When this value gets small (or even zero) the dis‐
868 patcher has so much work that it doesn´t sleep any more, and likely
869 can´t process the work in a timely fashion any more. This value
870 plus the wallTime from above sort of sums up to the total uptime
871 taken by this dispatcher. Therefore, expressing the wallTime as
872 percentage of this sum gives the busyness percentage draining all
873 the way up to 100% if sleepTime goes to 0.
874 ○ server_wallTime_us
876 The number of microseconds spent by the servers to send the metrics
878 from their queues. This value includes connection creation, reading
879 from the queue, and sending metrics over the network.
880 ○ dispatcherX
882 For each indivual dispatcher, the metrics received and blackholed
884 plus the wall clock time. The values are as described above.
885 ○ destinations.X
887 For all known destinations, the number of dropped, queued and sent
889 metrics plus the wall clock time spent. The values are as described
890 above.
891 ○ aggregators.metricsReceived
893 The number of metrics that were matched an aggregator rule and were
895 accepted by the aggregator. When a metric matches multiple aggrega‐
896 tors, this value will reflect that. A metric is not counted when it
897 is considered syntactically invalid, e.g. no value was found.
898 ○ aggregators.metricsDropped
900 The number of metrics that were sent to an aggregator, but did not
902 fit timewise. This is either because the metric was too far in the
903 past or future. The expire after clause in aggregate statements
904 controls how long in the past metric values are accepted.
905 ○ aggregators.metricsSent
907 The number of metrics that were sent from the aggregators. These
909 metrics were produced and are the actual results of aggregations.
910
914 Please report them at: https://github.com/grobian/carbon-c-relay/issues
915
917 Fabian Groffen <grobian@gentoo.org>
918
920 All other utilities from the graphite stack.
921 This project aims to be a fast replacement of the original Carbon relay
923 http://graphite.readthedocs.org/en/1.0/carbon-daemons.html#carbon-re‐
924 lay-py. carbon-c-relay aims to deliver performance and configurability.
925 Carbon is single threaded, and sending metrics to multiple consis‐
926 tent-hash clusters requires chaining of relays. This project provides a
927 multithreaded relay which can address multiple targets and clusters for
928 each and every metric based on pattern matches.
929 There are a couple more replacement projects out there, which are car‐
931 bon-relay-ng https://github.com/graphite-ng/carbon-relay-ng and
932 graphite-relay https://github.com/markchadwick/graphite-relay.
933 Compared to carbon-relay-ng, this project does provide carbon´s consis‐
935 tent-hash routing. graphite-relay, which does this, however doesn´t do
936 metric-based matches to direct the traffic, which this project does as
937 well. To date, carbon-c-relay can do aggregations, failover targets and
938 more.
939
941 This program was originally developed for Booking.com, which approved
942 that the code was published and released as Open Source on GitHub, for
943 which the author would like to express his gratitude. Development has
944 continued since with the help of many contributors suggesting features,
945 reporting bugs, adding patches and more to make carbon-c-relay into
946 what it is today.
947 November 2021 CARBON-C-RELAY(1)