1SPAUSEDD(8) BSD System Manager's Manual SPAUSEDD(8)
2
4 spausedd — Utility to detect and log scheduler pause
5
7 spausedd [-dDfhp] [-m steal_threshold] [-t timeout]
8
10 The spausedd utility is used for detecting and logging scheduler pause.
11 This means, when process should have been scheduled, but it was not. It's
12 also able to use steal time (time spent in other operating systems when
13 running in a virtualized environment) so it is (to some extend) able to
14 detect if problem is on the VM or host side. spausedd is able to read
15 information about steal time ether from kernel or (if compiled in) also
16 use VMGuestLib. Internally spausedd works as following pseudocode:
17 repeat:
19 store current monotonic time
20 store current steal time
21 sleep for (timeout / 3)
22 set time_diff to (current monotonic time - stored monotonic time)
23 if time_diff > timeout:
24 display error
25 set steal_time_diff to (current steal time - stored steal time)
26 if (steal_time_diff / time_diff) * 100 > steal_threshold:
27 display steal time error
28 spausedd arguments are as follows:
30 -d Display debug messages (specify twice to display also trace mes‐
32 sages).
33 -D Run on background (daemonize).
35 -f Run on foreground (do not demonize - default).
37 -h Show help.
39 -p Do not set RR scheduler.
41 -P Do not move process to root cgroup. It's worth noting that cur‐
43 rently (May 3 2021) cgroup v2 doesn’t yet support control of re‐
44 altime processes and the cpu controller can only be enabled when
45 all RT processes are in the root cgroup. So when this option is
46 used together with systemd, it may be impossible to make systemd
47 options like CPUQuota working correctly until spausedd is
48 stopped. Also when this option is used together with cgroup v2
49 and systemd it makes impossible (most of the time) for journald
50 to add systemd specific metadata (most importantly _SYSTEMD_UNIT)
51 properly, because spausedd is moved out of cgroup created by sys‐
52 temd. This means it is not possible to filter spausedd logged
53 messages based on these metadata (for example using -u or _SYS‐
54 TEMD_UNIT=UNIT pattern) and also running systemctl status doesn't
55 display (all) spausedd log messages. Problem is even worse be‐
56 cause journald caches pid for some time (approx. 5 sec) so ini‐
57 tial spausedd messages have correct metadata.
58 -m steal_threshold
60 Set steal threshold percent. (default is 10 if kernel information
61 is used and 100 if VMGuestLib is used).
62 -t timeout
64 Set timeout value in milliseconds (default 200).
65 If spausedd receives a SIGUSR1 signal, the current statistics are show.
67
69 To generate CPU load yes(1) together with chrt(1) is used in following
70 examples:
71 chrt -r 99 yes >/dev/null &
73 If chrt fails it may help to use cgexec(1) like:
75 cgexec -g cpu:/ chrt -r 99 yes >/dev/null &
77 First example is physical or virtual machine with 4 CPU threads so yes(1)
79 was executed 4 times. In a while spausedd should start logging messages
80 similar to:
81 Mar 20 15:01:54 spausedd: Running main poll loop with maximum
83 timeout 200 and steal threshold 10%
84 Mar 20 15:02:15 spausedd: Not scheduled for 0.2089s (threshold is
85 0.2000s), steal time is 0.0000s (0.00%)
86 Mar 20 15:02:16 spausedd: Not scheduled for 0.2258s (threshold is
87 0.2000s), steal time is 0.0000s (0.00%)
88 ...
89 This means that spausedd didn't got time to run for longer time than de‐
91 fault timeout. It's also visible that steal time was 0% so spausedd is
92 running ether on physical machine or VM where host machine is not over‐
93 loaded (VM was scheduled on time).
94 Second example is a host machine with 2 CPU threads running one VM. VM is
96 running an instance of spausedd. Two instancies of yes(1) was executed on
97 the host machine. After a while spausedd should start logging messages
98 similar to:
99 Mar 20 15:08:20 spausedd: Not scheduled for 0.9598s (threshold is
101 0.2000s), steal time is 0.7900s (82.31%)
102 Mar 20 15:08:20 spausedd: Steal time is > 10.0%, this is usually
103 because of overloaded host machine
104 ...
105 This means that spausedd didn't got the time to run for almost one sec‐
107 ond. Also because steal time is high, it means that spausedd was not
108 scheduled because VM wasn't scheduled by host machine.
109
111 The spausedd utility exits 0 on success, and >0 if an error occurs.
112
114 The spausedd utility was written by Jan Friesse <jfriesse@redhat.com>.
115
117 - OS is not updating steal time as often as monotonic clock. This means
118 that steal time difference can be (and very often is) bigger than
119 monotonic clock difference, so steal time percentage can be bigger
120 than 100%. It's happening very often for highly overloaded host ma‐
121 chine when spausedd is called with small timeout. This problem is
122 even bigger when VMGuestLib is used.
123 - VMGuestLib seems to randomly block.
125BSD May 20, 2020 BSD