1rig(1) General Commands Manual rig(1)
2
6 rig - Monitor a system for events and trigger specific actions
7
9 rig <RESOURCE OR SUBCOMMAND> [OPTIONS] <ACTIONS> [ACTION OPTIONS]
10
13 rig is a tool to assist in troubleshooting seemingly randomly occurring
14 events or events that occur at times that make active monitoring by a
15 sysadmin difficult.
16 rig sets-up detached processes, known as 'rigs', that watch a given
18 resource for a trigger condition, and once that trigger condition is
19 met takes actions defined by the user.
20
24 The following are options available to all rigs (resources).
25 --delay DELAY
28 Specify the number of seconds to wait after a rig is triggered
29 before running the configured actions. Note that the rig will
30 still trigger and stop all watcher threads immediately - this
31 delay comes after thread termination but before action execution
32 in order to avoid a possible race condition where multiple
33 watcher threads could conceivably trigger during a sufficiently
34 high delay time.
35 Default 0 seconds, meaning execute actions immediately upon rig
37 trigger condition being met.
38 --debug
41 Set logging level to debug instead of the default info level.
42 --foreground
45 Run the rig in the foreground, keeping stdout attached.
46 --interval SECONDS
48 Specify the amount of time to wait between a rig's polling
49 cycles. Most rigs monitor their resources in a flow of update ->
50 compare -> wait, where wait is simply sleeping until the next
51 needed update. Use this option to set how long a rig should
52 wait/sleep before updating their monitors again.
53 Default: 1, meaning update and compare once every second.
55 --name NAME
58 Give the rig a name, rather than generating a random one at
59 deployment.
60 By default, rigs are given a randomly generated string as a
62 name, which will appear in rig info output and in the rig's
63 socket name. Using this option will use the provided name
64 instead, and may be useful in distinguishing rigs when several
65 are deployed at one time
66 --no-archive
68 Do not create a tar archive of the collected data after a rig
69 has been triggered.
70 Normally, once all data has been collected, rig will create a
72 gzip'd tar archive under /var/tmp containing all the files cre‐
73 ated from the rig's actions - after which, the temp directory at
74 /var/tmp/rig/<id>/ is deleted.
75 Using this option skips creating the archive and preserves the
77 temp directory.
78 --repeat COUNT
81 Repeat certain actions COUNT number of times after the initial
82 execution of the action.
83 Actions will, unless otherwise specified by this option, only
85 execute once. Using this option actions that support repetition
86 will be repeated an additional COUNT number of times. For exam‐
87 ple, using --repeat 2 will result in repeatable actions being
88 executed three (3) total times.
89 Not every action supports repetition - in fact most do not. See
91 specific action's sections for information on if it can be
92 repeated or not.
93 --repeat-delay SECONDS
96 Number of seconds to wait between repetitive executions of the
97 same action.
98 This can be useful when using an action like gcore when you want
100 to get coredumps over a certain time period. For example, using
101 --repeat 1 --repeat-delay 60 will give you two (2) coredumps
102 taken one minute apart.
103 Defaults to one second.
105 --restart COUNT
108 Restart a configured rig up to COUNT number of times after being
109 triggered.
110 By default, a rig will trigger once and then terminate. Using
112 this option, an individual rig may restart itself up to COUNT
113 number of times, producing an additional archive of the
114 requested data after the triggering event happens again.
115 Note that this is the number of times to restart, not the total
117 number of times to run. Using a restart value of '2' means that
118 there will be 3 total archives generated for a rig.
119 By default, this is set to 0, meaning terminate after the first
121 trigger event. Use a value of '-1' to have a rig perpetually
122 restart itself without limit.
123
127 rig list
128 Show a list of known existing rigs and their status. Status
129 information is obtained by querying the socket created for that
130 particular rig.
131 rig destroy -i [ID or 'all']
134 Destroy a deployed rig with id ID. If ID is 'all', destroy all
135 known rigs. Note that if another entity kills the pid for the
136 running rig, destroy will fail as the socket is no longer con‐
137 nected to the (now killed) process. In this case use the --force
138 option to cleanup the lingering socket.
139 rig info -i [ID]
142 Get detailed information on a rig. This information will include
143 configuration options, the entire cmdline string given to launch
144 the rig, as well as information on each action the rig is con‐
145 figured to take and what the expected result from those
146 action(s) are.
147 Currently, this data is written to stdout in JSON format.
149 rig trigger -i [ID]
152 Manually trigger rig with id ID. This will cause the specified
153 rig to begin executing the actions configured for it, as if the
154 trigger condition had been met.
155 Note that this is only effective on a single rig basis, so using
157 a value of 'all' for the ID will not work.
158
161 These are the system resources that rig can monitor. There may be addi‐
162 tional manpages for specific resources. Where applicable this will be
163 noted below.
164 Note that 'resources', 'monitors', and referencing 'a rig' as a dis‐
166 tinct entity all refer to the same thing.
167 When creating a rig, if successful the rig's ID will be printed to con‐
169 sole.
170 logs Watch a single or multiple log files and/or journald units for a
173 specified message. When that message is matched to any watched
174 file or journal, the trigger condition is met and configured
175 actions are initiated.
176 The following options are available for the logs rig:
178 -m|--message STRING
180 Define the string that serves as the trigger condition
181 for the rig. This can be a regex string or an exact mes‐
182 sage. Be very careful in using the '*' regex character as
183 this may cause unintended behavior such as the rig imme‐
184 diately triggering on the first message seen.
185 Note that a small amount of transformation and testing is
187 done on the provided STRING. First, '*' characters are
188 converted to the python-style regex match of '.*'. After
189 which, rig performs a test on if the provided message
190 will regex-match itself, and if that fails the rig aborts
191 the creation process.
192 Aside from the conversion noted above, regexes provided
194 in this option must be python-style and not shell-style.
195 --logfile FILE
198 A comma-delimited list of files to watch. Each FILE spec‐
199 ified will be monitored from the current end of the file,
200 so old entries will not set off the rig's actions.
201 Default: /var/log/messages
203 --no-files
205 Do not monitor any log files.
206 --journal UNIT
208 A comma-delimited list of journal units to watch. The
209 journal is watched as a singular entity, and will be fil‐
210 tered to only read from the provided UNIT(s). If no UNIT
211 is specified, the whole system journal will be monitored.
212 Default: 'system'
214 --no-journal
216 Do not monitor the journal.
217 --count COUNT
219 The number of times the --message string should be
220 matched before the rig is triggered. Default 1 - meaning
221 match on the first occurence.
222 ping Perform a simple ongoing ping test against a specified host.
226 Pings are sent one at a time at a defined interval, and the
227 response is evaluated. Ping-type rigs may monitor for number of
228 lost packets and/or packets exceeding a specified RTT in mil‐
229 liseconds.
230 Packets are first evaluated for loss (including timeouts), then
232 for RTT time.
233 The following options are available for the ping rig:
235 --host ADDRESS
237 The target IP or hostname to ping. This is a required
238 option in order for a ping rig to be created.
239 During rig creation, a 'sanity check' ping is sent to the
241 ADDRESS to ensure that it is an address that is reachable
242 on the network and that it will respond to ICMP packets.
243 If this sanity check fails, rig creation is aborted.
244 --ping-timeout SECONDS
246 Specify the number of SECONDS to allow for a ping
247 response. If a ping encounters a timeout, then it is con‐
248 sidered both a lost packet and a packet exceeding the RTT
249 threshold (see --ping-ms-max and --ping-ms-count).
250 --lost-count PACKETS
252 Specify the number of PACKETS to accept being lost or
253 timed-out, before triggering the rig.
254 Default: 1 (trigger on the first lost packet)
256 --ping-interval SECONDS
258 Specify the number of SECONDS to wait between ping
259 requests sent to the target host.
260 Default: 1
262 --ping-ms-max MILLISECONDS
264 Specify the RTT threshold to allow for a returned ping
265 request. If the RTT reported by the ping command is above
266 this value in milliseconds, it is counted against the
267 threshold of packets exceeding this value specified by
268 --ping-ms-count.
269 By default, this form of checking is disabled. Any inte‐
271 ger value passed to this option will enable RTT monitor‐
272 ing.
273 --ping-ms-count PACKETS
275 Specify the number of PACKETS that may exceed the defined
276 --ping-ms-max RTT value before triggering the rig.
277 Default: 5
279 process
281 Watch a single process or list of processes for state changes or
282 resource consumption thresholds. When the process enters the
283 specified state or the specified resource consumption threshold
284 is met, the trigger condition is met.
285 The following options are available for the process rig:
287 --proc A PID or process name of processes to watch. If a process
289 name is specified, then rig will attempt to convert this
290 to a PID during rig creation. If multiple PIDs are found,
291 the default behavior is to fail creation and exit. To
292 have rig monitor all processes found for a process name,
293 use the --all option.
294 --state STATE
296 The state that a process needs to be in, in order to
297 trigger the rig. The following is a list of supported
298 states:
299 NAME DESCRIPTION SHORT‐
301 HAND
302 dead Dead - should never be seen 'X'
303 disk-sleep Uninterruptible sleep 'D' or
304 'UN'
305 running Currently running 'R' or
306 'run'
307 sleeping Interruptible sleep 'S' or
308 'sleep'
309 stopped Stopped 'T' or
310 'stop'
311 zombie Exited, still in proc table 'Z' or
312 'zomb'
313 Users can use either the full status name, or the short‐
315 hand noted in the final column of the table above. Both
316 the names and the shorthand values are case sensitive.
317 This can also be set to a "not" value by preceeding one
319 of the above state strings with a exclaimation mark (!),
320 e.g. '!sleeping' will match any non-sleep (S) state sta‐
321 tus for the process(es). Most shells will require you to
322 quote the state string when using the '!' character.
323 Note that using '!running' will cause rig to not trigger
325 against a state of 'sleeping', as generally speaking
326 'running' processes spend much of their time in S state,
327 and it is assumed that triggering against such a process
328 is not desired.
329 Process status is polled once every second.
331 --rss INTEGER
333 The amount of rss (resident set size) memory usage to use
334 as a threshold for triggering the rig. If the process'
335 RSS usage goes above this value, trigger.
336 The value provided here may be suffixed with K, M, or G
338 to denote the IEC unit. Rig will convert the provided
339 value and suffix into a value in bytes.
340 --vms INTEGER
342 The same as --rss but monitoring Virtual Memory Size
343 instead.
344 --memperc PERCENT
346 The percentage of total system memory a process is con‐
347 suming to use as a threshold for triggering the rig. If
348 the process' %mem meets or exceeds this value, trigger.
349 PERCENT may be a whole integer or a float. When using a
351 float, the process rig respects up to two (2) decimal
352 points of precision. For example, using ´--memperc 10.25´
353 is the same as using ´--memperc 10.25678´.
354 --cpuperc PERCENT
356 The percentage of CPU usage a process is consuming to use
357 as a threshold for triggering the rig. If the process'
358 %cpu meets or exceeds this value, trigger.
359 PERCENT may be a whole integer or a float. When using a
361 float and monitoring for CPU usage, rig respects one (1)
362 decimal point of precision due to how CPU usage is
363 reported.
364 PERCENT may be above 100 - as CPU usage can exceed 100
366 when a process is running on multiple CPUs.
367 system
370 Watch the system's utilization of resources as a whole, e.g.
372 total CPU or memory usage. When the utilization of a given
373 resource is either exceeded or falls below the given threshold
374 (determined as appropriate for each resource), the trigger con‐
375 dition is met.
376 The following options are available for the system rig:
378 --iowait PERCENT
380 The amount of %iowait as reported by the kernel to use as
381 a threshold value.
382 If exceeded, trigger the rig.
384 --steal PERCENT
386 The amount of %steal as reported by the kernel to use as
387 a threshold value.
388 If exceeded, trigger the rig.
390 --nice PERCENT
392 The amount of %nice as reported by the kernel to use as a
393 threshold value.
394 If exceeded, trigger the rig.
396 --guest PERCENT
398 The amount of %guest as reported by the kernel to use as
399 a threshold value.
400 If exceeded, trigger the rig.
402 --user The amount of %user as reported by the kernel to use as a
404 threshold value.
405 If exceeded, trigger the rig.
407 --available INTEGER
409 The amount of available memory in MiB as reported by the
410 kernel to use as a threshold value.
411 If the amount of available memory falls below this
413 threshold, trigger the rig.
414 --free INTEGER
416 The amount of free memory in MiB as reported by the ker‐
417 nel to use as a threshold value.
418 If the amount of free memory falls below this threshold,
420 trigger the rig.
421 --used INTEGER
423 The amount of used memory in MiB as reported by the ker‐
424 nel to use as a threshold value.
425 If the amount of used memory exceeds this threshold,
427 trigger the rig.
428 --slab INTEGER
430 The amount of slab memory in MiB as reported by the ker‐
431 nel to use as a threshold value.
432 If the amount of slab memory exceeds this threshold,
434 trigger the rig.
435 --cpuperc PERCENT
437 The amount of total CPU usage as reported by the kernel
438 as a percentage to use as a threshold value.
439 If exceeded, trigger the rig.
441 This value may be a whole integer or a float. Floats are
443 precise out to one (1) decimal point.
444 --memperc PERCENT
446 The amount of total memory usage as reported by the ker‐
447 nel as a percentage to use as a theshold value.
448 If exceeded, trigger the rig.
450 This value may be a whole integer or a float. Floats are
452 precise out to one (1) decimal point.
453 --loadavg FLOAT
455 System load average as reported by the OS to use as a
456 threshold value. If the reported loadavg exceeds this
457 value, trigger the rig. This option can accept either an
458 integer (1) or a float (1.0).
459 Linux returns loadavg data for the past 1, 5, and 15 min‐
461 utes. The system rig will monitor only one (1) of these
462 intervals at a time, as controlled by the --loadavg-
463 interval option.
464 --loadavg-interval [1, 5, 15]
466 Which time interval the rig should monitor when watching
467 the system's loadavg. Only 1, 5, and 15 are accepted
468 values for this option, as that is what the Linux kernel
469 returns loadavg data for.
470 Default: 1
472 --temp INTEGER
474 The temperature in Celsius rig should monitor the CPU for
475 meeting or exceeding.
476 This option takes an integer value, though temperature
478 data is single decimal point sensitive, so a temperature
479 of 50.9 degrees will not trigger a rig that sets this
480 option to 51.
481 By default rig will monitor the first physical CPU pack‐
483 age installed on the system. This may be changed via the
484 --cpu-id option. Note that rig will only monitor whole
485 packages and not individual cores, and that package tem‐
486 peratures reported are the highest reported temperature
487 for any core in that package.
488 --cpu-id ID
491 If specified, monitor this physical CPU package. By
492 default, rig will monitor physical CPU package 0 - mean‐
493 ing the first physically installed CPU.
494 When specifying an ID here, remember that in Linux CPU
496 IDs are zero-indexed, so the first CPU will be ID 0, the
497 second ID 1, and so forth.
498 Default: 0
500 Filesystem
502 Watch a filesystem, directory, or file for utilization changes.
504 Currently this rig is focused on space consumption, and will
505 trigger when the specified path or backing filesystem exceeds
506 the defined threshold for space utilization.
507 The following options are available for the filesystem rig:
509 --path PATH
511 Specify the filesystem, directory, or file path for the
512 rig to monitor. The location provided must exists when
513 the rig initializes for monitoring to be supported.
514 --size SIZE
516 Specify the size threshold to trigger on for the provided
517 --path. The size given must be an integer suffixed with
518 either K, M, G, or T. The provided value will be con‐
519 verted to bytes.
520 --fs-size SIZE
522 Use this option instead of --size if you want to monitor
523 the space usage of the backing filesystem for --path
524 rather than the size of the path alone.
525 Similar to --size this value must be suffixed with either
527 K, M, G, or T.
528 --fs-used PERCENT
530 Similar to --fs-size but instead provide a percentage
531 value to trigger on, when the filesystem's %used exceeds
532 this value.
533 Note that using this option is ultimately the same as
535 --fs-size as rig will convert the specified percentage
536 into a raw bytes value to use for comparisons.
537
541 The following actions are supported responses to triggered rigs. These
542 may be chained together on a single rig, so deploying multiple rigs
543 with matching trigger conditions with single, varying actions is unnec‐
544 essary.
545 Actions are executed based on a priority weighting system, where lower
547 values represent a higher priority action, and those actions with lower
548 values are executed before those with higher values. This is to allow
549 more time-sensitive actions to be taken before those that may either
550 take a long time to execute or are otherwise unaffected by allowing
551 other actions to run before them. Action priority values are set by the
552 actions directly and are currently not able to be modified by users.
553 gcore Collect a coredump of a given process or processes using GDB's
555 gcore utility.
556 Note that this does _not_ interrupt the running process(es).
558 Cores are saved to /tmp and will be named either core.$pid or
559 core.$proc_name.$pid depending on if a PID or process name was
560 provided. This action will be executed first when a rig is trig‐
561 gered and multiple actions are specified.
562 This action supports repetition via the --repeat option.
564 The gcore action supports the following options:
566 --gcore PROCESS
568 Enables this action and takes either a PID or process
569 name as a value. If a process name is given, the PID is
570 determined at rig creation. If multiple PIDs are found
571 for the same process name, the default behavior is to
572 fail rig creation. Use the --all-pids option to instead
573 use all PIDs discovered for a process name.
574 This option can be specified multiple times. E.G. --gcore
576 12345 --gcore myprocess will generate a coredump for PID
577 12345 and a process matching the name 'myprocess'.
578 --all-pids
581 Tells this action to collect a coredump for all PIDs
582 found for a provided process name.
583 --freeze
586 Freeze the process(es) that will be core dumped by send‐
587 ing a SIGSTOP prior to calling gcore on the discovered
588 pid(s).
589 If successful, then rig will send a SIGCONT after the
591 gcore execution has completed in order to thaw the
592 process.
593 kdump Generate a vmcore by triggering a kernel crash via sysrq.
596 Note that this action WILL cause node disruption by triggering a
598 kernel panic to generate the vmcore. This means your system will
599 reboot when this action is triggered.
600 The kdump action does not perform any configuration checks on
602 the system's kdump installation. It is assumed that kdump has
603 been properly configured and tested prior to using this action.
604 The kdump action supports the following options:
606 --kdump
608 Enables this action
609 --sysrq INTEGER
612 When the rig is deployed, if this option is set, rig will
613 set the system's /proc/sys/kernel/sysrq to the value pro‐
614 vided. See sysrq kernel documentation for information on
615 what values are supported.
616 sosreport
620 Run a sosreport after the rig has been triggered. Select plugin
621 enablement options as well as the --plugin-option from sosreport
622 are supported by this rig. This action should run after any
623 time-sensitive actions otherwise specified by the user for a
624 given rig.
625 The sosreport action supports the following options:
627 --sosreport
629 Enables this action
630 --enable-plugins PLUGINS
632 Specifically force the specified comma-delimited list of
633 PLUGINS to be enabled.
634 --plugin-option PLUGOPT
636 Modify a specific plugin's runtime options. This is
637 passed directly to sosreport as the same --plugin-option
638 value, which should take the form 'name.option=value'.
639 For example, to increase the podman plugin timeout use
640 ´--plugin-option podman.timeout=600´.
641 If you need to pass multiple sosreport plugin options,
643 use a comma-delimited list here instead of specifying
644 this option multiple times.
645 --skip-plugins PLUGINS
647 Do not run these specified plugins. Use a comma-delimited
648 list to skip multiple plugins.
649 --only-plugins PLUGINS
651 Only enable these specific plugins, disable all others.
652 Use a comma-delimited list to specify multiple plugins.
653 tcpdump
656 Start collecting a tcpdump when the rig is initialized, and stop
657 the collection when the rig triggers. This action will be trig‐
658 gered before most other actions, but after the gcore action.
659 Note there will be a slight delay in configuring any rig that
661 uses the tcpdump action as rig must verify that the tcpdump
662 process started successfully during the initialization process.
663 The tcpdump action supports the following options:
665 --tcpdump
667 Enables this action
668 --iface INTERFACE
670 Starts the tcpdump to monitor the provided INTERFACE. In
671 almost all situations this should likely be set to a spe‐
672 cific interface on the system, however the value of 'any'
673 is accepted by the tcpdump command in order to listen on
674 all interfaces. Be wary of using this however as use of
675 'any' means will make it impossible to determine which
676 interface a particular packet came in on in the resulting
677 packet capture.
678 Default: eth0
680 --filter FILTER
682 Provide a filter to use with tcpdump in order to reduce
683 the amount of traffic recorded in the packet capture.
684 This value is passed directly to the tcpdump utility, and
685 thus can be any valid filter accepted by tcpdump.
686 For most shells you must quote the filter string for rig
688 to pass it correctly.
689 --size SIZE
691 Limit the size of the packet capture file(s) to SIZE in
692 MB.
693 Default: 10
695 --captures CAPTURES
697 Specify the number of packet capture files to keep. If
698 more than one (1), then tcpdump will rotate the packet
699 capture file when it reaches the --size value and keep
700 CAPTURES number of files.
701 E.G. Using a CAPTURES of 2 and a SIZE of 5, then when the
703 rig terminates you will have up to 2 5MB packet captures.
704 Default: 1 (packet capture file is replaced upon reaching
706 SIZE limit).
707 noop
710 Does nothing - this action runs a no-op. This is ideally used
712 for when you need to test a rig's configuration to make sure a
713 rig's trigger condition is set properly - e.g. a regex string
714 for the logs' rig message option.
715 The noop action supports the following options:
717 --noop Enables this action
719
721 Jake Hunsaker <jhunsake@redhat.com>
722 January 2019 rig(1)