1SPAUSEDD(8)               BSD System Manager's Manual              SPAUSEDD(8)
2

NAME

4     spausedd — Utility to detect and log scheduler pause
5

SYNOPSIS

7     spausedd [-dDfhp] [-m steal_threshold] [-P mode] [-t timeout]
8

DESCRIPTION

10     The spausedd utility is used for detecting and logging scheduler pause.
11     This means, when process should have been scheduled, but it was not. It's
12     also able to use steal time (time spent in other operating systems when
13     running in a virtualized environment) so it is (to some extend) able to
14     detect if problem is on the VM or host side.  spausedd is able to read
15     information about steal time ether from kernel or (if compiled in) also
16     use VMGuestLib.  Internally spausedd works as following pseudocode:
17
18           repeat:
19               store current monotonic time
20               store current steal time
21               sleep for (timeout / 3)
22               set time_diff to (current monotonic time - stored monotonic time)
23               if time_diff > timeout:
24                   display error
25                   set steal_time_diff to (current steal time - stored steal time)
26                   if (steal_time_diff / time_diff) * 100 > steal_threshold:
27                       display steal time error
28
29     spausedd arguments are as follows:
30
31     -d      Display debug messages (specify twice to display also trace mes‐
32             sages).
33
34     -D      Run on background (daemonize).
35
36     -f      Run on foreground (do not demonize - default).
37
38     -h      Show help.
39
40     -p      Do not set RR scheduler.
41
42     -m steal_threshold
43             Set steal threshold percent. (default is 10 if kernel information
44             is used and 100 if VMGuestLib is used).
45
46     -P mode
47             Set mode of moving process to root cgroup. Default is auto which
48             first checks if setting of RR scheduler is enabled. If so, it
49             tries to set RR scheduler.  If this fails, process is moved to
50             root cgroup and set of RR scheduler is retried.  Another options
51             are on when process is always moved to root cgroup and off which
52             makes spausedd to never try move pid into root cgroup.  It's
53             worth noting that currently (May 3 2021) cgroup v2 doesn’t yet
54             support control of realtime processes and, for systems with CON‐
55             FIG_RT_GROUP_SCHED kernel option enabled, the cpu controller can
56             only be enabled when all RT processes are in the root cgroup. So
57             when moving to root cgroup is disabled and used together with
58             systemd, it may be impossible to make systemd options like
59             CPUQuota working correctly until spausedd is stopped.  Also when
60             moving to root cgroup is used together with cgroup v2 and systemd
61             it makes impossible (most of the time) for journald to add sys‐
62             temd specific metadata (most importantly _SYSTEMD_UNIT) properly,
63             because spausedd is moved out of cgroup created by systemd. This
64             means it is not possible to filter spausedd logged messages based
65             on these metadata (for example using -u or _SYSTEMD_UNIT=UNIT
66             pattern) and also running systemctl status doesn't display (all)
67             spausedd log messages.  Problem is even worse because journald
68             caches pid for some time (approx. 5 sec) so initial spausedd mes‐
69             sages have correct metadata.
70
71     -t timeout
72             Set timeout value in milliseconds (default 200).
73
74     If spausedd receives a SIGUSR1 signal, the current statistics are show.
75

EXAMPLES

77     To generate CPU load yes(1) together with chrt(1) is used in following
78     examples:
79
80           chrt -r 99 yes >/dev/null &
81
82     If chrt fails it may help to use cgexec(1) like:
83
84           cgexec -g cpu:/ chrt -r 99 yes >/dev/null &
85
86     First example is physical or virtual machine with 4 CPU threads so yes(1)
87     was executed 4 times. In a while spausedd should start logging messages
88     similar to:
89
90           Mar 20 15:01:54 spausedd: Running main poll loop with maximum
91           timeout 200 and steal threshold 10%
92           Mar 20 15:02:15 spausedd: Not scheduled for 0.2089s (threshold is
93           0.2000s), steal time is 0.0000s (0.00%)
94           Mar 20 15:02:16 spausedd: Not scheduled for 0.2258s (threshold is
95           0.2000s), steal time is 0.0000s (0.00%)
96           ...
97
98     This means that spausedd didn't got time to run for longer time than de‐
99     fault timeout. It's also visible that steal time was 0% so spausedd is
100     running ether on physical machine or VM where host machine is not over‐
101     loaded (VM was scheduled on time).
102
103     Second example is a host machine with 2 CPU threads running one VM. VM is
104     running an instance of spausedd. Two instancies of yes(1) was executed on
105     the host machine. After a while spausedd should start logging messages
106     similar to:
107
108           Mar 20 15:08:20 spausedd: Not scheduled for 0.9598s (threshold is
109           0.2000s), steal time is 0.7900s (82.31%)
110           Mar 20 15:08:20 spausedd: Steal time is > 10.0%, this is usually
111           because of overloaded host machine
112           ...
113
114     This means that spausedd didn't got the time to run for almost one sec‐
115     ond. Also because steal time is high, it means that spausedd was not
116     scheduled because VM wasn't scheduled by host machine.
117

DIAGNOSTICS

119     The spausedd utility exits 0 on success, and >0 if an error occurs.
120

AUTHORS

122     The spausedd utility was written by Jan Friesse <jfriesse@redhat.com>.
123

BUGS

125     -   OS is not updating steal time as often as monotonic clock. This means
126         that steal time difference can be (and very often is) bigger than
127         monotonic clock difference, so steal time percentage can be bigger
128         than 100%. It's happening very often for highly overloaded host ma‐
129         chine when spausedd is called with small timeout. This problem is
130         even bigger when VMGuestLib is used.
131
132     -   VMGuestLib seems to randomly block.
133
134BSD                              Jul 15, 2021                              BSD
Impressum