1STAP(1) General Commands Manual STAP(1)
2
6 stap - systemtap script translator/driver
7
11 stap [ OPTIONS ] FILENAME [ ARGUMENTS ]
12 stap [ OPTIONS ] - [ ARGUMENTS ]
13 stap [ OPTIONS ] -e SCRIPT [ ARGUMENTS ]
14 stap [ OPTIONS ] -l PROBE [ ARGUMENTS ]
15 stap [ OPTIONS ] -L PROBE [ ARGUMENTS ]
16 stap [ OPTIONS ] --dump-probe-types
17 stap [ OPTIONS ] --dump-probe-aliases
18 stap [ OPTIONS ] --dump-functions
19
24 The stap program is the front-end to the Systemtap tool. It accepts
25 probing instructions written in a simple domain-specific language,
26 translates those instructions into C code, compiles this C code, and
27 loads the resulting module into a running Linux kernel or a Dyninst
28 user-space mutator, to perform the requested system trace/probe func‐
29 tions. You can supply the script in a named file (FILENAME), from
30 standard input (use - instead of FILENAME), or from the command line
31 (using -e SCRIPT). The program runs until it is interrupted by the
32 user, or if the script voluntarily invokes the exit() function, or by
33 sufficient number of soft errors.
34 The language, which is described the SCRIPT LANGUAGE section below, is
36 strictly typed, expressive, declaration free, procedural, prototyping-
37 friendly, and inspired by awk and C. It allows source code points or
38 events in the system to be associated with handlers, which are subrou‐
39 tines that are executed synchronously. It is somewhat similar concep‐
40 tually to "breakpoint command lists" in the gdb debugger.
41
44 systemtap comes with a variety of educational, documentation and refer‐
45 ence resources. They come online and/or packaged for offline use.
46 Some systemtap diagnostic warning/error messages specially suggest
47 reading a man page by including a string like [man error::pass5]. For
48 online documentation, see the project web site,
49 https://sourceware.org/systemtap/
50 ┌──────────────────────────┬──────────────────────────────────────────────────────┐
53 │man pages │ │
54 ├──────────────────────────┼──────────────────────────────────────────────────────┤
55 │stap (this page) │ language syntax, concepts, operation, options │
56 ├──────────────────────────┼──────────────────────────────────────────────────────┤
57 │error::* │ further explanation of error conditions │
58 ├──────────────────────────┼──────────────────────────────────────────────────────┤
59 │warning::* │ further explanation of warning conditions │
60 ├──────────────────────────┼──────────────────────────────────────────────────────┤
61 │stapprobes │ probe points and their $context variables │
62 ├──────────────────────────┼──────────────────────────────────────────────────────┤
63 │stapref │ quick reference to language syntax │
64 ├──────────────────────────┼──────────────────────────────────────────────────────┤
65 │stappaths │ list of directories, including books & references │
66 ├──────────────────────────┼──────────────────────────────────────────────────────┤
67 │stap-prep │ program to install auxiliary dependencies like ker‐ │
68 │ │ nel debuginfo │
69 ├──────────────────────────┼──────────────────────────────────────────────────────┤
70 │tapset::* │ generated list of tapsets │
71 ├──────────────────────────┼──────────────────────────────────────────────────────┤
72 │probe::* │ generated list of tapset probe aliases │
73 ├──────────────────────────┼──────────────────────────────────────────────────────┤
74 │function::* │ generated list of tapset functions │
75 ├──────────────────────────┼──────────────────────────────────────────────────────┤
76 │macro::* │ generated list of tapset macros │
77 ├──────────────────────────┼──────────────────────────────────────────────────────┤
78 │stapvars │ some of the tapset global variables │
79 ├──────────────────────────┼──────────────────────────────────────────────────────┤
80 │staprun, stapdyn, stapbpf │ programs for executing compiled systemtap scripts │
81 ├──────────────────────────┼──────────────────────────────────────────────────────┤
82 │systemtap │ initscript, boot-time probing │
83 ├──────────────────────────┼──────────────────────────────────────────────────────┤
84 │stap-server │ compilation server │
85 ├──────────────────────────┼──────────────────────────────────────────────────────┤
86 │stapex │ a few very basic script examples │
87 ├──────────────────────────┼──────────────────────────────────────────────────────┤
88 │books │ │
89 ├──────────────────────────┼──────────────────────────────────────────────────────┤
90 │Beginner's Guide │ tutorial book, language essentials, examples │
91 ├──────────────────────────┼──────────────────────────────────────────────────────┤
92 │Tutorial │ shorter tutorial, exercises │
93 ├──────────────────────────┼──────────────────────────────────────────────────────┤
94 │Language Reference │ detailed language manual, covers statistics/analysis │
95 ├──────────────────────────┼──────────────────────────────────────────────────────┤
96 │Tapset Reference │ the tapset man pages, reformatted into a book │
97 ├──────────────────────────┼──────────────────────────────────────────────────────┤
98 │references │ │
99 ├──────────────────────────┼──────────────────────────────────────────────────────┤
100 │example scripts │ over a hundred directly usable sysadmin tools, toys, │
101 │ │ hacks to learn from │
102 └──────────────────────────┴──────────────────────────────────────────────────────┘
103
105 The systemtap translator supports the following options. Any other op‐
106 tion prints a list of supported options. Options may be given on the
107 command line, as usual. If the file $SYSTEMTAP_DIR/rc exist, options
108 are also loaded from there and interpreted first. ($SYSTEMTAP_DIR de‐
109 faults to $HOME/.systemtap if unset.)
110 In some cases, the default value of an option depends on particular
113 system configuration and thus can't be mentioned here directly. In
114 some of those cases running "stap --help" might display the default.
115 - Use standard input instead of a given FILENAME as probe language
118 input, unless -e SCRIPT is given.
119 -h --help
121 Show help message.
122 -V --version
124 Show version message.
125 -p NUM Stop after pass NUM. The passes are numbered 1-5: parse, elabo‐
127 rate, translate, compile, run. See the PROCESSING section for
128 details.
129 -v Increase verbosity for all passes. Produce a larger volume of
131 informative (?) output each time option repeated.
132 --vp ABCDE
134 Increase verbosity on a per-pass basis. For example, "--vp 002"
135 adds 2 units of verbosity to pass 3 only. The combination
136 "-v --vp 00004" adds 1 unit of verbosity for all passes, and 4
137 more for pass 5.
138 -k Keep the temporary directory after all processing. This may be
140 useful in order to examine the generated C code, or to reuse the
141 compiled kernel object.
142 -g Guru mode. Enable parsing of unsafe expert-level constructs
144 like embedded C.
145 -P Prologue-searching mode. This is equivalent to --pro‐
147 logue-searching=always. Activate heuristics to work around in‐
148 correct debugging information for function parameter $context
149 variables.
150 -u Unoptimized mode. Disable unused code elision and many other
152 optimizations during elaboration / translation.
153 -w Suppressed warnings mode. Disables all warning messages.
155 -W Treat all warnings as errors.
157 -b Use bulk mode (percpu files) for kernel-to-user data transfer.
159 Use the stap-merge program to multiplex them back together lat‐
160 er.
161 -i --interactive
163 Interactive mode. Enable an interface to build the systemtap
164 script incrementally and interactively.
165 -t Collect timing information on the number of times probe executes
167 and average amount of time spent in each probe-point. Also shows
168 the derivation for each probe-point.
169 -s NUM Use NUM megabyte buffers for kernel-to-user data transfer per
171 processor. The default is 16MB, or less on smaller memory ma‐
172 chines.
173 -I DIR Add the given directory to the tapset search directory. See the
175 description of pass 2 for details.
176 -D NAME=VALUE
178 Add the given C preprocessor directive to the module Makefile.
179 These can be used to override limit parameters described below.
180 -B NAME=VALUE
182 In kernel-runtime mode, add the given make directive to the ker‐
183 nel module build's make invocation. These can be used to add or
184 override kconfig options. For example, use
185 -B CONFIG_DEBUG_INFO=y
187 to add debugging information.
189 -B FLAG
191 In dyninst-runtime mode, add the given parameter to the compiler
192 CFLAGS used for building the dyninst shared library. For exam‐
193 ple, use
194 -B -g
196 to add debugging information.
198 -a ARCH
200 Use a cross-compilation mode for the given target architecture.
201 This requires access to the cross-compiler and the kernel build
202 tree, and goes along with the
203 -B CROSS_COMPILE=arch-tool-prefix-
205 and
206 -r /build/tree
207 options.
209 --modinfo NAME=VALUE
211 Add the name/value pair as a MODULE_INFO macro call to the gen‐
212 erated module. This may be useful to inform or override various
213 module-related checks in the kernel.
214 -G NAME=VALUE
216 Sets the value of global variable NAME to VALUE when staprun is
217 invoked. This applies to scalar variables declared global in
218 the script/tapset.
219 -R DIR Look for the systemtap runtime sources in the given directory.
221 Your DIR default can be seen using "stap --help".
222 -r /DIR
224 Build for kernel in given build tree. Can also be set with the
225 SYSTEMTAP_RELEASE environment variable.
226 -r RELEASE
228 Build for kernel in build tree /lib/modules/RELEASE/build. Can
229 also be set with the SYSTEMTAP_RELEASE environment variable.
230 -m MODULE
232 Use the given name for the generated kernel object module, in‐
233 stead of a unique randomized name. The generated kernel object
234 module is copied to the current directory.
235 -d MODULE
237 Add symbol/unwind information for the given module into the ker‐
238 nel object module. This may enable symbolic tracebacks from
239 those modules/programs, even if they do not have an explicit
240 probe placed into them.
241 --ldd Add symbol/unwind information for all user-space shared li‐
243 braries suspected by ldd to be necessary for user-space binaries
244 being probed or listed with the -d option. Caution: this can
245 make the probe modules considerably larger. Note that this op‐
246 tion does not deal with kernel-space modules: see instead
247 --all-modules below.
248 --all-modules
250 Equivalent to specifying "-dkernel" and a "-d" for each kernel
251 module that is currently loaded. Caution: this can make the
252 probe modules considerably larger.
253 -o FILE
255 Send standard output to named file. In bulk mode, percpu files
256 will start with FILE_ (FILE_cpu with -F) followed by the cpu
257 number. This supports strftime(3) formats for FILE.
258 -c CMD Start the probes, run CMD, and exit when CMD finishes. This al‐
260 so has the effect of setting target() to the pid of the command
261 ran.
262 -x PID Sets target() to PID. This allows scripts to be written that
264 filter on a specific process. Scripts run independent of the
265 PID's lifespan.
266 -e SCRIPT
268 Run the given SCRIPT specified on the command line.
269 -E SCRIPT
271 Run the given SCRIPT specified. This SCRIPT is run in addition
272 to the main script specified, through -e, or as a script file.
273 This option can be repeated to run multiple scripts, and can be
274 used in listing mode (-l/-L).
275 -l PROBE
277 Instead of running a probe script, just list all available probe
278 points matching the given single probe point. The pattern may
279 include wildcards and aliases, but not comma-separated multiple
280 probe points. The process result code will indicate failure if
281 there are no matches.
282 % stap -e 'probe syscall.* { }'
284 [...]
285 % stap -l 'syscall.*'
286 syscall.accept
287 [...]
288 syscall.writev
289 -L PROBE
292 Similar to "-l", but list matching probe points plus their
293 available context variables. When -v is set with -L, the output
294 includes duplicate probe points which are distinguished by their
295 PC address.
296 % stap -L 'process("/lib64/libpython*.so.*").mark("*")'
298 process("/usr/lib64/libpython2.7.so.1.0").mark("function__entry") $arg1:long $arg2:long $arg3:long
299 process("/usr/lib64/libpython2.7.so.1.0").mark("function__return") $arg1:long $arg2:long $arg3:long
300 process("/usr/lib64/libpython3.6m.so.1.0").mark("function__entry") $arg1:long $arg2:long $arg3:long
301 process("/usr/lib64/libpython3.6m.so.1.0").mark("function__return") $arg1:long $arg2:long $arg3:long
302 process("/usr/lib64/libpython3.6m.so.1.0").mark("gc__done") $arg1:long
303 process("/usr/lib64/libpython3.6m.so.1.0").mark("gc__start") $arg1:long
304 process("/usr/lib64/libpython3.6m.so.1.0").mark("line") $arg1:long $arg2:long $arg3:long
305 -F Without -o option, load module and start probes, then detach
308 from the module leaving the probes running. With -o option, run
309 staprun in background as a daemon and show its pid.
310 -S size[,N]
312 Sets the maximum size of output file and the maximum number of
313 output files. If the size of output file will exceed size ,
314 systemtap switches output file to the next file. And if the num‐
315 ber of output files exceed N , systemtap removes the oldest out‐
316 put file. You can omit the second argument.
317 -T TIMEOUT
319 Exit the script after TIMEOUT seconds.
320 --skip-badvars
322 Ignore unresolvable or run-time-inaccessible context variables
323 and substitute with 0, without errors.
324 --prologue-searching[=WHEN]
327 Prologue-searching mode. Activate heuristics to work around in‐
328 correct debugging information for function parameter $context
329 variables. WHEN can be either "never", "always", or "auto" (i.e.
330 enabled by heuristic). If WHEN is missing, then "always" is as‐
331 sumed. If the option is missing, then "auto" is assumed.
332 --suppress-handler-errors
335 Wrap all probe handlers into something like this
336 try { ... } catch { next }
338 block, which causes any runtime errors to be quietly suppressed.
340 Suppressed errors do not count against MAXERRORS limits. In
341 this mode, the MAXSKIPPED limits are also suppressed, so that
342 many errors and skipped probes may be accumulated during a
343 script's runtime. Any overall counts will still be reported at
344 shutdown.
345 --compatible VERSION
348 Suppress recent script language or tapset changes which are in‐
349 compatible with given older version of systemtap. This may be
350 useful if a much older systemtap script fails to run. See the
351 DEPRECATION section for more details.
352 --check-version
355 This option is used to check if the active script has any con‐
356 structs that may be systemtap version specific. See the DEPRE‐
357 CATION section for more details.
358 --clean-cache
361 This option prunes stale entries from the cache directory. This
362 is normally done automatically after successful runs, but this
363 option will trigger the cleanup manually and then exit. See the
364 CACHING section for more details about cache limits.
365 --color[=WHEN], --colour[=WHEN]
368 This option controls coloring of error messages. WHEN can be ei‐
369 ther "never", "always", or "auto" (i.e. enable only if at a ter‐
370 minal). If WHEN is missing, then "always" is assumed. If the op‐
371 tion is missing, then "auto" is assumed.
372 Colors can be modified using the SYSTEMTAP_COLORS environment
374 variable. The format must be of the form
375 key1=val1:key2=val2:key3=val3 ...etc. Valid keys are "error",
376 "warning", "source", "caret", and "token". Values constitute
377 Select Graphic Rendition (SGR) parameter(s). Consult the docu‐
378 mentation of your terminal for the SGRs it supports. As an exam‐
379 ple, the default colors would be expressed as
380 error=01;31:warning=00;33:source=00;34:caret=01:token=01. If
381 SYSTEMTAP_COLORS is absent, the default colors will be used. If
382 it is empty or invalid, coloring is turned off.
383 --disable-cache
386 This option disables all use of the cache directory. No files
387 will be either read from or written to the cache.
388 --poison-cache
391 This option treats files in the cache directory as invalid. No
392 files will be read from the cache, but resulting files from this
393 run will still be written to the cache. This is meant as a
394 troubleshooting aid when stap's cached behavior seems to be mis‐
395 behaving. If it helped, there is a probably a bug in systemtap
396 that the developers would like you to report.
397 --privilege[=stapusr | =stapsys | =stapdev]
400 This option instructs stap to examine the script looking for
401 constructs which are not allowed for the specified privilege
402 level (see UNPRIVILEGED USERS). Compilation fails if any such
403 constructs are used. If stapusr or stapsys are specified when
404 using a compile server (see --use-server), the server will exam‐
405 ine the script and, if compilation succeeds, the server will
406 cryptographically sign the resulting kernel module, certifying
407 that is it safe for use by users at the specified privilege lev‐
408 el.
409 If --privilege has not been specified, -pN has not been speci‐
411 fied with N < 5, and the invoking user is not root, and is not a
412 member of the group stapdev, then stap will automatically add
413 the appropriate --privilege option to the options already speci‐
414 fied.
415 --unprivileged
418 This option is equivalent to --privilege=stapusr.
419 --use-server[=HOSTNAME[:PORT] | =IP_ADDRESS[:PORT] | =CERT_SERIAL]
422 Specify compile-server(s) to be used for compilation and/or in
423 conjunction with --list-servers and --trust-servers (see below)
424 for listing. If no argument is supplied, then the default in un‐
425 privileged mode (see --privilege) is to select compatible
426 servers which are trusted as SSL peers and as module signers and
427 currently online. Otherwise the default is to select compatible
428 servers which are trusted as SSL peers and currently online.
429 --use-server may be specified more than once, in which case a
430 list of servers is accumulated in the order specified. Servers
431 may be specified by host name, ip address, or by certificate se‐
432 rial number (obtained using --list-servers). The latter is most
433 commonly used when adding or revoking trust in a server (see
434 --trust-servers below). If a server is specified by host name or
435 ip address, then an optional port number may be specified. This
436 is useful for accessing servers which are not on the local net‐
437 work or to specify a particular server.
438 IP addresses may be IPv4 or IPv6 addresses.
440 If a particular IPv6 address is link local and exists on more
442 than one interface, the intended interface may be specified by
443 appending the address with a percent sign (%) followed by the
444 intended interface name. For example,
445 "fe80::5eff:35ff:fe07:55ca%eth0".
446 In order to specify a port number with an IPv6 address, it is
448 necessary to enclose the IPv6 address in square brackets ([]) in
449 order to separate the port number from the rest of the address.
450 For example, "[fe80::5eff:35ff:fe07:55ca]:5000" or
451 "[fe80::5eff:35ff:fe07:55ca%eth0]:5000".
452 If --use-server has not been specified, -pN has not been speci‐
454 fied with N < 5, and the invoking user not root, is not a member
455 of the group stapdev, but is a member of the group stapusr, then
456 stap will automatically add --use-server to the options already
457 specified.
458 --use-server-on-error[=yes|=no]
461 Instructs stap to retry compilation of a script using a compile
462 server if compilation on the local host fails in a manner which
463 suggests that it might succeed using a server. If this option
464 is not specified, the default is no. If no argument is provid‐
465 ed, then the default is yes. Compilation will be retried for
466 certain types of errors (e.g. insufficient data or resources)
467 which may not occur during re-compilation by a compile server.
468 Compile servers will be selected automatically for the re-compi‐
469 lation attempt as if --use-server was specified with no argu‐
470 ments.
471 --list-servers[=SERVERS]
474 Display the status of the requested SERVERS, where SERVERS is a
475 comma-separated list of server attributes. The list of at‐
476 tributes is combined to filter the list of servers displayed.
477 Supported attributes are:
478 all specifies all known servers (trusted SSL peers, trusted
480 module signers, online servers).
481 specified
483 specifies servers specified using --use-server.
484 online filters the output by retaining information about servers
486 which are currently online.
487 trusted
489 filters the output by retaining information about servers
490 which are trusted as SSL peers.
491 signer filters the output by retaining information about servers
493 which are trusted as module signers (see --privilege).
494 compatible
496 filters the output by retaining information about servers
497 which are compatible with the current kernel release and
498 architecture.
499 If no argument is provided, then the default is specified. If
501 no servers were specified using --use-server, then the default
502 servers for --use-server are listed.
503 Note that --list-servers uses the avahi-daemon service to detect
505 online servers. If this service is not available, then
506 --list-servers will fail to detect any online servers. In order
507 for --list-servers to detect servers listening on IPv6 address‐
508 es, the avahi-daemon configuration file /etc/avahi/avahi-dae‐
509 mon.conf must contain an active "use-ipv6=yes" line. The service
510 must be restarted after adding this line in order for IPv6 to be
511 enabled.
512 --trust-servers[=TRUST_SPEC]
515 Grant or revoke trust in compile-servers, specified using
516 --use-server as specified by TRUST_SPEC, where TRUST_SPEC is a
517 comma-separated list specifying the trust which is to be granted
518 or revoked. Supported elements are:
519 ssl trust the specified servers as SSL peers.
521 signer trust the specified servers as module signers (see
523 --privilege). Only root can specify signer.
524 all-users
526 grant trust as an ssl peer for all users on the local
527 host. The default is to grant trust as an ssl peer for
528 the current user only. Trust as a module signer is always
529 granted for all users. Only root can specify all-users.
530 revoke revoke the specified trust. The default is to grant it.
532 no-prompt
534 do not prompt the user for confirmation before carrying
535 out the requested action. The default is to prompt the
536 user for confirmation.
537 If no argument is provided, then the default is ssl. If no
539 servers were specified using --use-server, then no trust will be
540 granted or revoked.
541 Unless no-prompt has been specified, the user will be prompted
543 to confirm the trust to be granted or revoked before the opera‐
544 tion is performed.
545 --sign-module
548 Sign the module with a MOK (Machine Owner Key) on UEFI/Secure‐
549 Boot systems. See the SECUREBOOT section for more details.
550 --dump-probe-types
553 Dumps a list of supported probe types and exits. If --privi‐
554 lege=stapusr is also specified, the list will be limited to
555 probe types available to unprivileged users.
556 --dump-probe-aliases
559 Dumps a list of all probe aliases found in library files and ex‐
560 its.
561 --dump-functions
564 Dumps a list of all the public functions found in library files
565 and exits. Also includes their parameters and types. A function
566 of type 'unknown' indicates a function that does not return a
567 value. Note that not all function/parameter types may be re‐
568 solved (these are also shown by 'unknown'). This features is
569 very memory-intensive and thus may not work properly with --use-
570 server if the target server imposes an rlimit on process memory
571 (i.e. through the ~stap-server/.systemtap/rc configuration file,
572 see stap-server(8)).
573 --remote URL
576 Set the execution target to the given host. This option may be
577 repeated to target multiple execution targets. Passes 1-4 are
578 completed locally as normal to build the script, and then pass 5
579 will copy the module to the target and run it. Acceptable URL
580 forms include:
581 [USER@]HOSTNAME, ssh://[USER@]HOSTNAME
583 This mode uses ssh, optionally using a username not
584 matching your own. If a custom ssh_config file is in use,
585 add SendEnv LANG to retain internationalization function‐
586 ality.
587 libvirt://DOMAIN, libvirt://DOMAIN/LIBVIRT_URI
589 This mode uses stapvirt to execute the script on a domain
590 managed by libvirt. Optionally, LIBVIRT_URI may be speci‐
591 fied to connect to a specific driver and/or a remote
592 host. For example, to connect to the local privileged QE‐
593 MU driver, use:
594 --remote libvirt://MyDomain/qemu:///system
596 See the page at <http://libvirt.org/uri.html> for sup‐
598 ported URIs. Also see stapvirt(1) for more information on
599 how to prepare the domain for stap probing.
600 unix:PATH
602 This mode connects to a UNIX socket. This can be used
603 with a QEMU virtio-serial port for executing scripts in‐
604 side a running virtual machine.
605 direct://
607 Special loopback mode to run on the local host.
608 --remote-prefix
610 Prefix each line of remote output with "N: ", where N is the in‐
611 dex of the remote execution target from which the given line
612 originated.
613 --download-debuginfo[=OPTION]
616 Enable, disable or set a timeout for the automatic debuginfo
617 downloading feature offered by abrt as specified by OPTION,
618 where OPTION is one of the following:
619 yes enable automatic downloading of debuginfo with no time‐
621 out. This is the same as not providing an OPTION value to
622 --download-debuginfo
623 no explicitly disable automatic downloading of debuginfo.
625 This is the same as not using the option at all.
626 ask show abrt output, and ask before continuing download. No
628 timeout will be set.
629 <timeout>
631 specify a timeout as a positive number to stop the down‐
632 load if it is taking longer than <timeout> seconds.
633 --rlimit-as=NUM
635 Specify the maximum size of the process's virtual memory (ad‐
636 dress space), in bytes.
637 --rlimit-cpu=NUM
640 Specify the CPU time limit, in seconds.
641 --rlimit-nproc=NUM
644 Specify the maximum number of processes that can be created.
645 --rlimit-stack=NUM
648 Specify the maximum size of the process stack, in bytes.
649 --rlimit-fsize=NUM
652 Specify the maximum size of files that the process may create,
653 in bytes.
654 --sysroot=DIR
657 Specify sysroot directory where target files (executables, li‐
658 braries, etc.) are located. With -r RELEASE, the sysroot will
659 be searched for the appropriate kernel build directory. With -r
660 /DIR, however, the sysroot will not be used to find the kernel
661 build.
662 --sysenv=VAR=VALUE
665 Provide an alternate value for an environment variable where the
666 value on a remote system differs. Path variables (e.g. PATH,
667 LD_LIBRARY_PATH) are assumed to be relative to the directory
668 provided by --sysroot, if provided.
669 --suppress-time-limits
672 Disable -DSTP_OVERLOAD related options as well as -DMAXACTION
673 and -DMAXTRYLOCK. This option requires guru mode.
674 --runtime=MODE
677 Set the pass-5 runtime mode. Valid options are kernel (de‐
678 fault), dyninst and bpf. See ALTERNATE RUNTIMES below for more
679 information.
680 --dyninst
683 Shorthand for --runtime=dyninst.
684 --bpf Shorthand for --runtime=bpf.
687 --save-uprobes
690 On machines that require SystemTap to build its own uprobes mod‐
691 ule (kernels prior to version 3.5), this option instructs Sys‐
692 temTap to also save a copy of the module in the current directo‐
693 ry (creating a new "uprobes" directory first).
694 --target-namespaces=PID
697 Allow for a set of target namespaces to be set based on the
698 namespaces the given PID is in. This is for namespace-aware
699 tapset functions. If the target namespaces was not set, the tar‐
700 get defaults to the stap process' namespaces.
701 --monitor=INTERVAL
704 Enables an interface to display status information about the
705 module(uptime, module name, invoker uid, memory sizes, global
706 variables, list of probes with their statistics). An optional
707 argument INTERVAL can be supplied to set the refresh rate in
708 seconds of the status window. The module can also be controlled
709 by a list of commands using the following keys:
710 c Resets all global variables to their initial values or
712 zeroes them if they did not have an initial value.
713 s Rotates the attribute used to sort the list of probes.
715 t Brings up a prompt to allow toggling(on/off) of probes by
717 index. Probe points are still affected by their condi‐
718 tions.
719 r Resumes the script by toggling on all probes.
721 p Pauses the script by toggling off all probes.
723 x Hides/shows the status window. This allows for more out‐
725 put to be seen.
726 navigation-keys
728 The navigation keys can be used to scroll up and down the
729 windows.
730 Tab Toggle scrolling between status and output windows.
732 --example
735 This option is used to run example scripts without having to en‐
736 ter the entire path to the script. Example scripts can be found
737 in the directory specified in the stappaths(7) manual page.
738 --no-global-var-display
741 This option is used to disable the automatic logging of unused
742 global variables at the end of a stap session.
743
746 Any additional arguments on the command line are passed to the script
747 parser for substitution. See below.
748
751 The systemtap script language resembles awk and C. There are two main
752 outermost constructs: probes and functions. Within these, statements
753 and expressions use C-like operator syntax and precedence.
754 GENERAL SYNTAX
757 Whitespace is ignored. Three forms of comments are supported:
758 # ... shell style, to the end of line, except for $# and @#
759 // ... C++ style, to the end of line
760 /* ... C style ... */
761 Literals are either strings enclosed in double-quotes (passing through
762 the usual C escape codes with backslashes, and with adjacent string
763 literals glued together, also as in C), or integers (in decimal, hexa‐
764 decimal, or octal, using the same notation as in C). All strings are
765 limited in length to some reasonable value (a few hundred bytes). In‐
766 tegers are 64-bit signed quantities, although the parser also accepts
767 (and wraps around) values above positive 2**63.
768 In addition, script arguments given at the end of the command line may
770 be inserted. Use $1 ... $<NN> for insertion unquoted, @1 ... @<NN> for
771 insertion as a string literal. The number of arguments may be accessed
772 through $# (as an unquoted number) or through @# (as a quoted number).
773 These may be used at any place a token may begin, including within the
774 preprocessing stage. Reference to an argument number beyond what was
775 actually given is an error.
776 PREPROCESSING
779 A simple conditional preprocessing stage is run as a part of parsing.
780 The general form is similar to the cond ? exp1 : exp2 ternary operator:
781 %( CONDITION %? TRUE-TOKENS %)
783 %( CONDITION %? TRUE-TOKENS %: FALSE-TOKENS %)
784 The CONDITION is either an expression whose format is determined by its
786 first keyword, or a string literals comparison or a numeric literals
787 comparison. It can be also composed of many alternatives and conjunc‐
788 tions of CONDITIONs (meant as in previous sentence) using || and && re‐
789 spectively. However, parentheses are not supported yet, so remembering
790 that conjunction takes precedence over alternative is important.
791 If the first part is the identifier kernel_vr or kernel_v to refer to
793 the kernel version number, with ("2.6.13-1.322FC3smp") or without
794 ("2.6.13") the release code suffix, then the second part is one of the
795 six standard numeric comparison operators <, <=, ==, !=, >, and >=, and
796 the third part is a string literal that contains an RPM-style version-
797 release value. The condition is deemed satisfied if the version of the
798 target kernel (as optionally overridden by the -r option) compares to
799 the given version string. The comparison is performed by the glibc
800 function strverscmp. As a special case, if the operator is for simple
801 equality (==), or inequality (!=), and the third part contains any
802 wildcard characters (* or ? or [), then the expression is treated as a
803 wildcard (mis)match as evaluated by fnmatch.
804 If, on the other hand, the first part is the identifier arch to refer
806 to the processor architecture (as named by the kernel build system
807 ARCH/SUBARCH), then the second part is one of the two string comparison
808 operators == or !=, and the third part is a string literal for matching
809 it. This comparison is a wildcard (mis)match.
810 Similarly, if the first part is an identifier like CONFIG_something to
812 refer to a kernel configuration option, then the second part is == or
813 !=, and the third part is a string literal for matching the value (com‐
814 monly "y" or "m"). Nonexistent or unset kernel configuration options
815 are represented by the empty string. This comparison is also a wild‐
816 card (mis)match.
817 If the first part is the identifier systemtap_v, the test refers to the
819 systemtap compatibility version, which may be overridden for old
820 scripts with the --compatible flag. The comparison operator is as is
821 for kernel_v and the right operand is a version string. See also the
822 DEPRECATION section below.
823 If the first part is the identifier systemtap_privilege, the test
825 refers to the privilege level that the systemtap script is compiled
826 with. Here the second part is == or !=, and the third part is a string
827 literal, either "stapusr" or "stapsys" or "stapdev".
828 If the first part is the identifier guru_mode, the test refers to if
830 the systemtap script is compiled with guru_mode. Here the second part
831 is == or !=, and the third part is a number, either 1 or 0.
832 If the first part is the identifier runtime, the test refers to the
834 systemtap runtime mode. See ALTERNATE RUNTIMES below for more informa‐
835 tion on runtimes. The second part is one of the two string comparison
836 operators == or !=, and the third part is a string literal for matching
837 it. This comparison is a wildcard (mis)match.
838 Otherwise, the CONDITION is expected to be a comparison between two
840 string literals or two numeric literals. In this case, the arguments
841 are the only variables usable.
842 The TRUE-TOKENS and FALSE-TOKENS are zero or more general parser tokens
844 (possibly including nested preprocessor conditionals), and are passed
845 into the input stream if the condition is true or false. For example,
846 the following code induces a parse error unless the target kernel ver‐
847 sion is newer than 2.6.5:
848 %( kernel_v <= "2.6.5" %? **ERROR** %) # invalid token sequence
850 The following code might adapt to hypothetical kernel version drift:
852 probe kernel.function (
854 %( kernel_v <= "2.6.12" %? "__mm_do_fault" %:
855 %( kernel_vr == "2.6.13*smp" %? "do_page_fault" %:
856 UNSUPPORTED %) %)
857 ) { /* ... */ }
858 %( arch == "ia64" %?
860 probe syscall.vliw = kernel.function("vliw_widget") {}
861 %)
862 PREPROCESSOR MACROS
866 The preprocessor also supports a simple macro facility, run as a sepa‐
867 rate pass before conditional preprocessing.
868 Macros are defined using the following construct:
870 @define NAME %( BODY %)
872 @define NAME(PARAM_1, PARAM_2, ...) %( BODY %)
873 Macros, and parameters inside a macro body, are both invoked by prefix‐
875 ing the macro name with an @ symbol:
876 @define foo %( x %)
878 @define add(a,b) %( ((@a)+(@b)) %)
879 @foo = @add(2,2)
881 Macro expansion is currently performed in a separate pass before condi‐
884 tional compilation. Therefore, both TRUE- and FALSE-tokens in condi‐
885 tional expressions will be macroexpanded regardless of how the condi‐
886 tion is evaluated. This can sometimes lead to errors:
887 // The following results in a conflict:
889 %( CONFIG_UTRACE == "y" %?
890 @define foo %( process.syscall %)
891 %:
892 @define foo %( **ERROR** %)
893 %)
894 // The following works properly as expected:
896 @define foo %(
897 %( CONFIG_UTRACE == "y" %? process.syscall %: **ERROR** %)
898 %)
899 The first example is incorrect because both @defines are evaluated in a
901 pass prior to the conditional being evaluated.
902 Normally, a macro definition is local to the file it occurs in. Thus,
904 defining a macro in a tapset does not make it available to the user of
905 the tapset. Publically available library macros can be defined by in‐
906 cluding .stpm files on the tapset search path. These files may only
907 contain @define constructs, which become visible across all tapsets and
908 user scripts. Optionally, within the .stpm files, a public macro defi‐
909 nition can be surrounded by a preprocessor conditional as described
910 above.
911 CONSTANTS
914 Tapsets or guru-mode user scripts can access header file constant to‐
915 kens, typically macros, using built-in @const() operator. The respec‐
916 tive header file inclusion is possible either via the tapset library,
917 or using a top-level guru mode embedded-C construct. This results in
918 appropriate embedded C pragma comments setting.
919 @const("STP_SKIP_BADVARS")
921 VARIABLES
925 Identifiers for variables and functions are an alphanumeric sequence,
926 and may include _ and $ characters. They may not start with a plain
927 digit, as in C. Each variable is by default local to the probe or
928 function statement block within which it is mentioned, and therefore
929 its scope and lifetime is limited to a particular probe or function in‐
930 vocation.
931 Scalar variables are implicitly typed as either string or integer. As‐
933 sociative arrays also have a string or integer value, and a tuple of
934 strings and/or integers serving as a key. Here are a few basic expres‐
935 sions.
936 var1 = 5
938 var2 = "bar"
939 array1 [pid()] = "name" # single numeric key
940 array2 ["foo",4,i++] += 5 # vector of string/num/num keys
941 if (["hello",5,4] in array2) println ("yes") # membership test
942 The translator performs type inference on all identifiers, including
945 array indexes and function parameters. Inconsistent type-related use
946 of identifiers signals an error.
947 Variables may be declared global, so that they are shared amongst all
949 probes and functions and live as long as the entire systemtap session.
950 There is one namespace for all global variables, regardless of which
951 script file they are found within. Concurrent access to global vari‐
952 ables is automatically protected with locks, see the SAFETY AND SECURI‐
953 TY section for more details. A global declaration may be written at
954 the outermost level anywhere, not within a block of code. Global vari‐
955 ables which are written but never read will be displayed automatically
956 at session shutdown. The translator will infer for each its value
957 type, and if it is used as an array, its key types. Optionally, scalar
958 globals may be initialized with a string or number literal. The fol‐
959 lowing declaration marks variables as global.
960 global var1, var2, var3=4
962 Global variables can also be set as module options. One can do this by
965 either using the -G option, or the module must first be compiled using
966 stap -p4. Global variables can then be set on the command line when
967 calling staprun on the module generated by stap -p4. See staprun(8) for
968 more information.
969 The scope of a global variable may be limited to a tapset or user
971 script file using private keyword. The global keyword is optional when
972 defining a private global variable. Following declaration marks var1
973 and var2 private globals.
974 private global var1=2
976 private var2
977 Arrays are limited in size by the MAXMAPENTRIES variable -- see the
980 SAFETY AND SECURITY section for details. Optionally, global arrays may
981 be declared with a maximum size in brackets, overriding MAXMAPENTRIES
982 for that array only. Note that this doesn't indicate the type of keys
983 for the array, just the size.
984 global tiny_array[10], normal_array, big_array[50000]
986 Arrays may be configured for wrapping using the '%' suffix. This caus‐
989 es older elements to be overwritten if more elements are inserted than
990 the array can hold. This works for both associative and statistics
991 typed arrays.
992 global wrapped_array1%[10], wrapped_array2%
994 Many types of probe points provide context variables, which are run-
998 time values, safely extracted from the kernel or userspace program be‐
999 ing probed. These are prefixed with the $ character. The CONTEXT
1000 VARIABLES section in stapprobes(3stap) lists what is available for each
1001 type of probe point. These context variables become normal string or
1002 numeric scalars once they are stored in normal script variables. See
1003 the TYPECASTING section below on how to to turn them back into typed
1004 pointers for further processing as context variables. There is some
1005 automation to help!
1006 STATEMENTS
1009 Statements enable procedural control flow. They may occur within func‐
1010 tions and probe handlers. The total number of statements executed in
1011 response to any single probe event is limited to some number defined by
1012 the MAXACTION macro in the translated C code, and is in the neighbour‐
1013 hood of 1000.
1014 EXP Execute the string- or integer-valued expression and throw away
1016 the value.
1017 { STMT1 STMT2 ... }
1019 Execute each statement in sequence in this block. Note that
1020 separators or terminators are generally not necessary between
1021 statements.
1022 ; Null statement, do nothing. It is useful as an optional separa‐
1024 tor between statements to improve syntax-error detection and to
1025 handle certain grammar ambiguities.
1026 if (EXP) STMT1 [ else STMT2 ]
1028 Compare integer-valued EXP to zero. Execute the first (non-ze‐
1029 ro) or second STMT (zero).
1030 while (EXP) STMT
1032 While integer-valued EXP evaluates to non-zero, execute STMT.
1033 for (EXP1; EXP2; EXP3) STMT
1035 Execute EXP1 as initialization. While EXP2 is non-zero, execute
1036 STMT, then the iteration expression EXP3.
1037 foreach (VAR in ARRAY [ limit EXP ]) STMT
1039 Loop over each element of the named global array, assigning cur‐
1040 rent key to VAR. The array may not be modified within the
1041 statement. By adding a single + or - operator after the VAR or
1042 the ARRAY identifier, the iteration will proceed in a sorted or‐
1043 der, by ascending or descending index or value. If the array
1044 contains statistics aggregates, adding the desired @operator be‐
1045 tween the ARRAY identifier and the + or - will specify the sort‐
1046 ing aggregate function. See the STATISTICS section below for
1047 the ones available. Default is @count. Using the optional lim‐
1048 it keyword limits the number of loop iterations to EXP times.
1049 EXP is evaluated once at the beginning of the loop.
1050 foreach ([VAR1, VAR2, ...] in ARRAY [ limit EXP ]) STMT
1052 Same as above, used when the array is indexed with a tuple of
1053 keys. A sorting suffix may be used on at most one VAR or ARRAY
1054 identifier.
1055 foreach ([VAR1, VAR2, ...] in ARRAY [INDEX1, INDEX2, ...] [ limit EXP
1057 ]) STMT
1058 Same as above, where iterations are limited to elements in the
1059 array where the keys match the index values specified. The sym‐
1060 bol * can be used to specify an index and will be treated as a
1061 wildcard.
1062 foreach (VAR0 = VAR in ARRAY [ limit EXP ]) STMT
1064 This variant of foreach saves current value into VAR0 on each
1065 iteration, so it is the same as ARRAY[VAR]. This also works
1066 with a tuple of keys. Sorting suffixes on VAR0 have the same
1067 effect as on ARRAY.
1068 foreach (VAR0 = VAR in ARRAY [INDEX1, INDEX2, ...] [ limit EXP ]) STMT
1070 Same as above, where iterations are limited to elements in the
1071 array where the keys match the index values specified. The sym‐
1072 bol * can be used to specify an index and will be treated as a
1073 wildcard.
1074 break, continue
1076 Exit or iterate the innermost nesting loop (while or for or
1077 foreach) statement.
1078 return EXP
1080 Return EXP value from enclosing function. If the function's
1081 value is not taken anywhere, then a return statement is not
1082 needed, and the function will have a special "unknown" type with
1083 no return value.
1084 next Return now from enclosing probe handler. This is especially
1086 useful in probe aliases that apply event filtering predicates.
1087 When used in functions, the execution will be immediately trans‐
1088 ferred to the next overloaded function.
1089 try { STMT1 } catch { STMT2 }
1091 Run the statements in the first block. Upon any run-time er‐
1092 rors, abort STMT1 and start executing STMT2. Any errors in
1093 STMT2 will propagate to outer try/catch blocks, if any.
1094 try { STMT1 } catch(VAR) { STMT2 }
1096 Same as above, plus assign the error message to the string
1097 scalar variable VAR.
1098 delete ARRAY[INDEX1, INDEX2, ...]
1100 Remove from ARRAY the element specified by the index tuple. If
1101 the index tuple contains a * in place of an index, the * is
1102 treated as a wildcard and all elements with keys that match the
1103 index tuple will be removed from ARRAY. The value will no
1104 longer be available, and subsequent iterations will not report
1105 the element. It is not an error to delete an element that does
1106 not exist.
1107 delete ARRAY
1109 Remove all elements from ARRAY.
1110 delete SCALAR
1112 Removes the value of SCALAR. Integers and strings are cleared
1113 to 0 and "" respectively, while statistics are reset to the ini‐
1114 tial empty state.
1115 EXPRESSIONS
1118 Systemtap supports a number of operators that have the same general
1119 syntax, semantics, and precedence as in C and awk. Arithmetic is per‐
1120 formed as per typical C rules for signed integers. Division by zero or
1121 overflow is detected and results in an error.
1122 binary numeric operators
1124 * / % + - >> << & ^ | && ||
1125 binary string operators
1127 . (string concatenation)
1128 numeric assignment operators
1130 = *= /= %= += -= >>= <<= &= ^= |=
1131 string assignment operators
1133 = .=
1134 unary numeric operators
1136 + - ! ~ ++ --
1137 binary numeric, string comparison or regex matching operators
1139 < > <= >= == != =~ !~
1140 ternary operator
1142 cond ? exp1 : exp2
1143 grouping operator
1145 ( exp )
1146 function call
1148 fn ([ arg1, arg2, ... ])
1149 array membership check
1151 exp in array
1152 [exp1, exp2, ... ] in array
1153 [*, *, ... ] in array
1154 REGULAR EXPRESSION MATCHING
1157 The scripting language supports regular expression matching. The basic
1158 syntax is as follows:
1159 exp =~ regex
1161 exp !~ regex
1162 (The first operand must be an expression evaluating to a string; the
1164 second operand must be a string literal containing a syntactically
1165 valid regular expression.)
1166 The regular expression syntax supports POSIX Extended Regular Expres‐
1168 sion features as documented in egrep(1) except for subexpression reuse
1169 ("\1") functionality.
1170 After a successful match, the contents of the matched string and subex‐
1172 pressions can be extracted using the matched() and ngroups() tapset
1173 functions as follows:
1174 if ("an example string" =~ "str(ing)") {
1176 matched(0) // -> returns "string", the matched substring
1177 matched(1) // -> returns "ing", the 1st matched subexpression
1178 ngroups() // -> returns 2, the number of matched groups
1179 }
1180 PROBES
1183 The main construct in the scripting language identifies probes. Probes
1184 associate abstract events with a statement block ("probe handler") that
1185 is to be executed when any of those events occur. The general syntax
1186 is as follows:
1187 probe PROBEPOINT [, PROBEPOINT] { [STMT ...] }
1189 probe PROBEPOINT [, PROBEPOINT] if (CONDITION) { [STMT ...] }
1190 Events are specified in a special syntax called "probe points". There
1193 are several varieties of probe points defined by the translator, and
1194 tapset scripts may define further ones using aliases. Probe points may
1195 be wildcarded, grouped, or listed in preference sequences, or declared
1196 optional. More details on probe point syntax and semantics are listed
1197 on the stapprobes(3stap) manual page.
1198 The probe handler is interpreted relative to the context of each event.
1200 For events associated with kernel code, this context may include vari‐
1201 ables defined in the source code at that spot. These "context vari‐
1202 ables" are presented to the script as variables whose names are pre‐
1203 fixed with "$". They may be accessed only if the kernel's compiler
1204 preserved them despite optimization. This is the same constraint that
1205 a debugger user faces when working with optimized code. In addition,
1206 the objects must exist in paged-in memory at the moment of the system‐
1207 tap probe handler's execution, because systemtap must not cause (sup‐
1208 presses) any additional paging. Some probe types have very little con‐
1209 text. See the stapprobes(3stap) man pages to see the kinds of context
1210 variables available at each kind of probe point. As of systemtap ver‐
1211 sion 4.3, functions called from the handlers of some probe point types
1212 may also refer to context variables. These are treated as if a clone
1213 of that function was inlined into the calling probe handler and $vari‐
1214 ables evaluated in its context.
1215 Probes may be decorated with an arming condition, consisting of a sim‐
1217 ple boolean expression on read-only global script variables. While
1218 disarmed (inactive, condition evaluates to false), some probe types re‐
1219 duce or eliminate their run-time overheads. When an arming condition
1220 evaluates to true, probes will be soon re-armed, and their probe han‐
1221 dlers will start getting called as the events fire. (Some events may
1222 be lost during the arming interval. If this is unacceptable, do not
1223 use arming conditions for those probes.) Example of the syntax:
1224 probe timer.us(TIMER) if (enabled) {
1226 }
1227 New probe points may be defined using "aliases". Probe point aliases
1230 look similar to probe definitions, but instead of activating a probe at
1231 the given point, it just defines a new probe point name as an alias to
1232 an existing one. There are two types of alias, i.e. the prologue style
1233 and the epilogue style which are identified by "=" and "+=" respective‐
1234 ly.
1235 For prologue style alias, the statement block that follows an alias
1237 definition is implicitly added as a prologue to any probe that refers
1238 to the alias. While for the epilogue style alias, the statement block
1239 that follows an alias definition is implicitly added as an epilogue to
1240 any probe that refers to the alias. For example:
1241 probe syscall.read = kernel.function("sys_read") {
1243 fildes = $fd
1244 if (execname() == "init") next # skip rest of probe
1245 }
1246 defines a new probe point syscall.read, which expands to
1248 kernel.function("sys_read"), with the given statement as a prologue,
1249 which is useful to predefine some variables for the alias user and/or
1250 to skip probe processing entirely based on some conditions. And
1251 probe syscall.read += kernel.function("sys_read") {
1253 if (tracethis) println ($fd)
1254 }
1255 defines a new probe point with the given statement as an epilogue,
1257 which is useful to take actions based upon variables set or left over
1258 by the the alias user. Please note that in each case, the statements
1259 in the alias handler block are treated ordinarily, so that variables
1260 assigned there constitute mere initialization, not a macro substitu‐
1261 tion.
1262 Aliases can also be defined to include both a prologue and an epilogue.
1264 probe syscall.read = kernel.function("sys_read") {
1266 fildes = $fd
1267 if (execname() == "init") next
1268 },{
1269 if (tracethis) println ($fd)
1270 }
1271 An alias is used just like a built-in probe type.
1274 probe syscall.read {
1276 printf("reading fd=%d\n", fildes)
1277 if (fildes > 10) tracethis = 1
1278 }
1279 Probes with an alias can make use of the @probewrite predicate. This
1282 check is used to detect whether a script variable or target variable
1283 has been written to in the probe handler body.
1284 @probewrite(var)
1286 expands to 1 iff var has been written to in the probe handler
1287 body, otherwise it expands to 0.
1288 In the following example, @probewrite(var) expands to 1 because var has
1290 been written to in the probe handler body and consequently, the condi‐
1291 tional statement will run.
1292 probe foo = begin { var = 0 }, { if (@probewrite(var)) println(var) }
1294 probe foo {
1296 var = 1
1297 }
1298 FUNCTIONS
1302 Systemtap scripts may define subroutines to factor out common work.
1303 Functions take any number of scalar (integer or string) arguments, and
1304 must return a single scalar (integer or string). An example function
1305 declaration looks like this:
1306 function thisfn (arg1, arg2) {
1308 return arg1 + arg2
1309 }
1310 Note the general absence of type declarations, which are instead in‐
1312 ferred by the translator. However, if desired, a function definition
1313 may include explicit type declarations for its return value and/or its
1314 arguments. This is especially helpful for embedded-C functions. In
1315 the following example, the type inference engine need only infer type
1316 type of arg2 (a string).
1317 function thatfn:string (arg1:long, arg2) {
1319 return sprint(arg1) . arg2
1320 }
1321 Functions may call others or themselves recursively, up to a fixed
1323 nesting limit. This limit is defined by the MAXNESTING macro in the
1324 translated C code and is in the neighbourhood of 10.
1325 Functions may be marked private using the private keyword to limit
1327 their scope to the tapset or user script file they are defined in. An
1328 example definition of a private function follows:
1329 private function three:long () { return 3 }
1331 Functions terminating without reaching an explicit return statement
1334 will return an implicit 0 or "", determined by type inference.
1335 Functions may be overloaded during both runtime and compile time.
1337 Runtime overloading allows the executed function to be selected while
1339 the module is running based on runtime conditions and is achieved using
1340 the "next" statement in script functions and STAP_NEXT macro for embed‐
1341 ded-C functions. For example,
1342 function f() { if (condition) next; print("first function") }
1345 function f() %{ STAP_NEXT; print("second function") %}
1346 function f() { print("third function") }
1347 During a functioncall f(), the execution will transfer to the third
1350 function if condition evaluates to true and print "third function".
1351 Note that the second function is unconditionally nexted.
1352 Parameter overloading allows the function to be executed to be selected
1354 at compile time based on the number of arguments provided to the func‐
1355 tioncall. For example,
1356 function g() { print("first function") }
1359 function g(x) { print("second function") }
1360 g() -> "first function"
1361 g(1) -> "second function"
1362 Note that runtime overloading does not occur in the above example, as
1365 exactly one function will be resolved for the functioncall. The use of
1366 a next statement inside a function while no more overloads remain will
1367 trigger a runtime exception Runtime overloading will only occur if the
1368 functions have the same arity, functions with the same name but differ‐
1369 ent number of parameters are completely unrelated.
1370 Execution order is determined by a priority value which may be speci‐
1372 fied. If no explicit priority is specified, user script functions are
1373 given a higher priority than library functions. User script functions
1374 and library functions are assigned a default priority value of 0 and 1
1375 respectively. Functions with the same priority are executed in decla‐
1376 ration order. For example,
1377 function f():3 { if (condition) next; print("first function") }
1380 function f():1 { if (condition) next; print("second function") }
1381 function f():2 { print("third function") }
1382 Since the second function has highest priority, it is executed first.
1385 The first function is never executed as there no "next" statements in
1386 the third function to transfer execution.
1387 PRINTING
1390 There are a set of function names that are specially treated by the
1391 translator. They format values for printing to the standard systemtap
1392 output stream in a more convenient way (note that data generated in the
1393 kernel module need to get transferred to user-space in order to get
1394 printed).
1395 The sprint* variants return the formatted string instead of printing
1397 it.
1398 print, sprint
1400 Print one or more values of any type, concatenated directly to‐
1401 gether.
1402 println, sprintln
1404 Print values like print and sprint, but also append a newline.
1405 printd, sprintd
1407 Take a string delimiter and two or more values of any type, and
1408 print the values with the delimiter interposed. The delimiter
1409 must be a literal string constant.
1410 printdln, sprintdln
1412 Print values with a delimiter like printd and sprintd, but also
1413 append a newline.
1414 printf, sprintf
1416 Take a formatting string and a number of values of corresponding
1417 types, and print them all. The format must be a literal string
1418 constant.
1419 The printf formatting directives similar to those of C, except that
1421 they are fully type-checked by the translator:
1422 %b Writes a binary blob of the value given, instead of ASCII
1424 text. The width specifier determines the number of bytes
1425 to write; valid specifiers are %b %1b %2b %4b %8b. De‐
1426 fault (%b) is 8 bytes.
1427 %c Character.
1429 %d,%i Signed decimal.
1431 %m Safely reads kernel (without #) or user (with #) memory
1433 at the given address, outputs its content. The optional
1434 precision specifier (not field width) determines the num‐
1435 ber of bytes to read - default is 1 byte. %10.4m prints
1436 4 bytes of the memory in a 10-character-wide field.
1437 Note, on some architectures user memory can still be read
1438 without #.
1439 %M Same as %m, but outputs in hexadecimal. The minimal size
1441 of output is double the optional precision specifier -
1442 default is 1 byte (2 hex chars). %10.4M prints 4 bytes
1443 of the memory as 8 hexadecimal characters in a 10-charac‐
1444 ter-wide field. %.*M hex-dumps a given number of bytes
1445 from a given buffer.
1446 %o Unsigned octal.
1448 %p Unsigned pointer address.
1450 %s String.
1452 %u Unsigned decimal.
1454 %x Unsigned hex value, in all lower-case.
1456 %X Unsigned hex value, in all upper-case.
1458 %% Writes a %.
1460 The # flag selects the alternate forms. For octal, this prefixes a 0.
1462 For hex, this prefixes 0x or 0X, depending on case. For characters,
1463 this escapes non-printing values with either C-like escapes or raw oc‐
1464 tal. In the case of %#m/%#M, this safely accesses user space memory
1465 rather than kernel space memory.
1466 Examples:
1468 a = "alice", b = "bob", p = 0x1234abcd, i = 123, j = -1, id[a] = 1234, id[b] = 4567
1470 print("hello")
1471 Prints: hello
1472 println(b)
1473 Prints: bob\n
1474 println(a . " is " . sprint(16))
1475 Prints: alice is 16
1476 foreach (name in id) printdln("|", strlen(name), name, id[name])
1477 Prints: 5|alice|1234\n3|bob|4567
1478 printf("%c is %s; %x or %X or %p; %d or %u\n",97,a,p,p,p,j,j)
1479 Prints: a is alice; 1234abcd or 1234ABCD or 0x1234abcd; -1 or 18446744073709551615\n
1480 printf("2 bytes of kernel buffer at address %p: %2m", p, p)
1481 Prints: 2 byte of kernel buffer at address 0x1234abcd: <binary data>
1482 printf("%4b", p)
1483 Prints (these values as binary data): 0x1234abcd
1484 printf("%#o %#x %#X\n", 1, 2, 3)
1485 Prints: 01 0x2 0X3
1486 printf("%#c %#c %#c\n", 0, 9, 42)
1487 Prints: \000 \t *
1488 STATISTICS
1492 It is often desirable to collect statistics in a way that avoids the
1493 penalties of repeatedly exclusive locking the global variables those
1494 numbers are being put into. Systemtap provides a solution using a spe‐
1495 cial operator to accumulate values, and several pseudo-functions to ex‐
1496 tract the statistical aggregates.
1497 The aggregation operator is <<<, and resembles an assignment, or a C++
1499 output-streaming operation. The left operand specifies a scalar or ar‐
1500 ray-index lvalue, which must be declared global. The right operand is
1501 a numeric expression. The meaning is intuitive: add the given number
1502 to the pile of numbers to compute statistics of. (The specific list of
1503 statistics to gather is given separately, by the extraction functions.)
1504 foo <<< 1
1506 stats[pid()] <<< memsize
1507 The extraction functions are also special. For each appearance of a
1510 distinct extraction function operating on a given identifier, the
1511 translator arranges to compute a set of statistics that satisfy it.
1512 The statistics system is thereby "on-demand". Each execution of an ex‐
1513 traction function causes the aggregation to be computed for that moment
1514 across all processors.
1515 Here is the set of extractor functions. The first argument of each is
1517 the same style of lvalue used on the left hand side of the accumulate
1518 operation. The @count(v), @sum(v), @min(v), @max(v), @avg(v), @vari‐
1519 ance(v[, b]) extractor functions compute the number/total/minimum/maxi‐
1520 mum/average/variance of all accumulated values. The resulting values
1521 are all simple integers. Arrays containing aggregates may be sorted
1522 and iterated. See the foreach construct above.
1523 Variance uses Welford's online algorithm. The calculations are based
1525 on integer arithmetic, and so may suffer from low precision and over‐
1526 flow. To improve this, @variance(v[, b]) accepts an optional parameter
1527 b, the bit-shift, ranging from 0 (default) to 62, for internal scaling.
1528 Only one value of bit-shift may be used with given global variable. A
1529 larger bitshift value increases precision, but increases the likelihood
1530 of overflow.
1531 $ stap -e \
1534 > 'global x probe oneshot { for(i=1;i<=5;i++) x<<<i println(@variance(x)) }'
1535 12
1536 $ stap -e \
1537 > 'global x probe oneshot { for(i=1;i<=5;i++) x<<<i println(@variance(x,1)) }'
1538 2
1539 $ python3 -c 'import statistics; print(statistics.variance([1, 2, 3, 4, 5]))'
1540 2.5
1541 $
1542 Overflow (from internal multiplication of large numbers) may occur and
1545 may cause a negative variance result. Consider normalizing your input
1546 data. Adding or subtracting a fixed value from all variance inputs
1547 preserves the original variance. Dividing the variance inputs by a
1548 fixed value shrinks the original variance by that value squared.
1549 Histograms are also available, but are more complicated because they
1553 have a vector rather than scalar value. @hist_linear(v,start,stop,in‐
1554 terval) represents a linear histogram from "start" to "stop" (inclu‐
1555 sive) by increments of "interval". The interval must be positive. Sim‐
1556 ilarly, @hist_log(v) represents a base-2 logarithmic histogram. Print‐
1557 ing a histogram with the print family of functions renders a histogram
1558 object as a tabular "ASCII art" bar chart.
1559 probe timer.profile {
1562 x[1] <<< pid()
1563 x[2] <<< uid()
1564 y <<< tid()
1565 }
1566 global x // an array containing aggregates
1567 global y // a scalar
1568 probe end {
1569 foreach ([i] in x @count+) {
1570 printf ("x[%d]: avg %d = sum %d / count %d\n",
1571 i, @avg(x[i]), @sum(x[i]), @count(x[i]))
1572 println (@hist_log(x[i]))
1573 }
1574 println ("y:")
1575 println (@hist_log(y))
1576 }
1577 The counts of each histogram bucket may be individually accessed via
1580 the [index] operator. Each bucket is addressed from 1 through N (for
1581 each natural bucket). In addition bucket #0 counts all the samples be‐
1582 neath the start value, and bucket #N+1 counts all the samples above the
1583 stop value. Histogram buckets (including the two out-of-range buckets)
1584 may also be iterated with foreach.
1585 global x
1588 probe oneshot {
1589 x <<< -100
1590 x <<< 1
1591 x <<< 2
1592 x <<< 3
1593 x <<< 100
1594 foreach (bucket in @hist_linear(x,1,3,1))
1595 // expecting 1 out-of-range-low bucket
1596 // 3 payload buckets
1597 // 1 out-of-range-high bucket
1598 printf("bucket %d count %d\n",
1599 bucket, @hist_linear(x,1,3,1)[bucket])
1600 }
1601 TYPECASTING
1605 Once a pointer (see the CONTEXT VARIABLES section of stapprobes(3stap))
1606 has been saved into a script integer variable, the translator attempts
1607 to keep the type information necessary to access members from that
1608 pointer.
1609 The translator attempts to track DWARF typing associated with script
1611 variables assigned from addresses of context $variables, @cast or @var
1612 operators. Depending on the complexity of the script code, this asso‐
1613 ciation may pass to related variables, so that -> and [] operators may
1614 be used on them, just as on the original context variable. For exam‐
1615 ple:
1616 foo = $param->foo; printf("x:%d y:%d\n", foo->x, foo->y)
1619 printf("my value is %d\n", ($type == 42 ? $foo : $bar)->value)
1620 printf("my parent pid is %d\n", task_parent(task_current())->tgid)
1621 However, if this association heuristic doesn't work for a script, using
1624 the @cast() operator tells the translator how to interpret the number
1625 as a typed pointer.
1626 @cast(p, "type_name"[, "module"])->member
1628 This will interpret p as a pointer to a struct/union named type_name
1631 and dereference the member value. Further ->subfield expressions may
1632 be appended to dereference more levels. Note that for direct derefer‐
1633 encing of a pointer {kernel,user}_{char,int,...}($p) should be used.
1634 (Refer to stapfuncs(5) for more details.) NOTE: the same dereferenc‐
1635 ing operator -> is used to refer to both direct containment or pointer
1636 indirection. Systemtap automatically determines which. The optional
1637 module tells the translator where to look for information about that
1638 type. Multiple modules may be specified as a list with : separators.
1639 If the module is not specified, it will default either to the probe
1640 module for dwarf probes, or to "kernel" for functions and all other
1641 probes types.
1642 The translator can create its own module with type information from a
1644 header surrounded by angle brackets, in case normal debuginfo is not
1645 available. For kernel headers, prefix it with "kernel" to use the ap‐
1646 propriate build system. All other headers are built with default GCC
1647 parameters into a user module. Multiple headers may be specified in
1648 sequence to resolve a codependency.
1649 @cast(tv, "timeval", "<sys/time.h>")->tv_sec
1651 @cast(task, "task_struct", "kernel<linux/sched.h>")->tgid
1652 @cast(task, "task_struct",
1653 "kernel<linux/sched.h><linux/fs_struct.h>")->fs->umask
1654 Values acquired by @cast may be pretty-printed by the $ and $$ suffix
1656 operators, the same way as described in the CONTEXT VARIABLES section
1657 of the stapprobes(3stap) manual page.
1658 When in guru mode, the translator will also allow scripts to assign new
1661 values to members of typecasted pointers.
1662 Typecasting is also useful in the case of void* members whose type may
1664 be determinable at runtime.
1665 probe foo {
1667 if ($var->type == 1) {
1668 value = @cast($var->data, "type1")->bar
1669 } else {
1670 value = @cast($var->data, "type2")->baz
1671 }
1672 print(value)
1673 }
1674 EMBEDDED C
1678 When in guru mode, the translator accepts embedded C code in the top
1679 level of the script. Such code is enclosed between %{ and %} markers,
1680 and is transcribed verbatim, without analysis, in some sequence, into
1681 the top level of the generated C code. At the outermost level, this
1682 may be useful to add #include instructions, and any auxiliary defini‐
1683 tions for use by other embedded code.
1684 Another place where embedded code is permitted is as a function body.
1686 In this case, the script language body is replaced entirely by a piece
1687 of C code enclosed again between %{ and %} markers. This C code may do
1688 anything reasonable and safe. There are a number of undocumented but
1689 complex safety constraints on atomicity, concurrency, resource consump‐
1690 tion, and run time limits, so this is an advanced technique.
1691 The memory locations set aside for input and output values are made
1693 available to it using macros STAP_ARG_* and STAP_RETVALUE. Errors may
1694 be signalled with STAP_ERROR. Output may be written with STAP_PRINTF.
1695 The function may return early with STAP_RETURN. Here are some exam‐
1696 ples:
1697 function integer_ops (val) %{
1699 STAP_PRINTF("%d\n", STAP_ARG_val);
1700 STAP_RETVALUE = STAP_ARG_val + 1;
1701 if (STAP_RETVALUE == 4)
1702 STAP_ERROR("wrong guess: %d", (int) STAP_RETVALUE);
1703 if (STAP_RETVALUE == 3)
1704 STAP_RETURN(0);
1705 STAP_RETVALUE ++;
1706 %}
1707 function string_ops (val) %{
1708 strlcpy (STAP_RETVALUE, STAP_ARG_val, MAXSTRINGLEN);
1709 strlcat (STAP_RETVALUE, "one", MAXSTRINGLEN);
1710 if (strcmp (STAP_RETVALUE, "three-two-one"))
1711 STAP_RETURN("parameter should be three-two-");
1712 %}
1713 function no_ops () %{
1714 STAP_RETURN(); /* function inferred with no return value */
1715 %}
1716 The function argument and return value types have to be inferred by the
1718 translator from the call sites in order for this to work. The user
1719 should examine C code generated for ordinary script-language functions
1720 in order to write compatible embedded-C ones.
1721 The last place where embedded code is permitted is as an expression
1723 rvalue. In this case, the C code enclosed between %{ and %} markers is
1724 interpreted as an ordinary expression value. It is assumed to be a
1725 normal 64-bit signed number, unless the marker /* string */ is includ‐
1726 ed, in which case it's treated as a string.
1727 function add_one (val) {
1729 return val + %{ 1 %}
1730 }
1731 function add_string_two (val) {
1732 return val . %{ /* string */ "two" %}
1733 }
1734 @define SOME_STAP_MACRO %( %{ SOME_C_MACRO %} %)
1735 probe begin {
1736 printf("SOME_C_MACRO has value: %d\n", @SOME_STAP_MACRO);
1737 }
1738 The embedded-C code may contain markers to assert optimization and
1741 safety properties.
1742 /* pure */
1744 means that the C code has no side effects and may be elided en‐
1745 tirely if its value is not used by script code.
1746 /* stable */
1748 means that the C code always has the same value (in any given
1749 probe handler invocation), so repeated calls may be automatical‐
1750 ly replaced by memoized values. Such functions must take no pa‐
1751 rameters, and also be pure.
1752 /* unprivileged */
1754 means that the C code is so safe that even unprivileged users
1755 are permitted to use it.
1756 /* myproc-unprivileged */
1758 means that the C code is so safe that even unprivileged users
1759 are permitted to use it, provided that the target of the current
1760 probe is within the user's own process.
1761 /* guru */
1763 means that the C code is so unsafe that a systemtap user must
1764 specify -g (guru mode) to use this. (Tapsets are permitted and
1765 presumed to call them safely.)
1766 /* unmangled */
1768 in an embedded-C function, means that the legacy (pre-1.8) argu‐
1769 ment access syntax should be made available inside the function.
1770 Hence, in addition to STAP_ARG_foo and STAP_RETVALUE one can use
1771 THIS->foo and THIS->__retvalue respectively inside the function.
1772 This is useful for quickly migrating code written for SystemTap
1773 version 1.7 and earlier.
1774 /* unmodified-fnargs */
1776 in an embedded-C function, means that the function arguments are
1777 not modified inside the function body.
1778 /* string */
1780 in embedded-C expressions only, means that the expression has
1781 const char * type and should be treated as a string value, in‐
1782 stead of the default long numeric.
1783 Script level global variables may be accessed in embedded-C functions
1785 and blocks. To read or write the global variable var , the /* prag‐
1786 ma:read:var */ or /* pragma:write:var */ marker must be first placed in
1787 the embedded-C function or block. This provides the macros STAP_GLOB‐
1788 AL_GET_* and STAP_GLOBAL_SET_* macros to allow reading and writing, re‐
1789 spectively. For example:
1790 global var
1792 global var2[100]
1793 function increment() %{
1794 /* pragma:read:var */ /* pragma:write:var */
1795 /* pragma:read:var2 */ /* pragma:write:var2 */
1796 STAP_GLOBAL_SET_var(STAP_GLOBAL_GET_var()+1); //var++
1797 STAP_GLOBAL_SET_var2(1, 1, STAP_GLOBAL_GET_var2(1, 1)+1); //var2[1,1]++
1798 %}
1799 Variables may be read and set in both embedded-C functions and expres‐
1801 sions. Strings returned from embedded-C code are decayed to pointers.
1802 Variables must also be assigned at script level to allow for type in‐
1803 ference. Map assignment does not return the value written, so chaining
1804 does not work.
1805 BUILT-INS
1808 A set of builtin probe point aliases are provided by the scripts in‐
1809 stalled in the directory specified in the stappaths(7) manual page.
1810 The functions are described in the stapprobes(3stap) manual page.
1811 DEREFERENCING
1814 Integers can be dereferenced from pointers saved as a script integer
1815 variables using the @kderef() or @uderef() operators. @kderef() is
1816 used for kernel space addresses and @uderef() is used for user space
1817 addresses.
1818 @kderef(SIZE, addr)
1820 @uderef(SIZE, addr)
1821 This will interpret addr as a kernel/user address and read SIZE bytes
1823 starting at that address. SIZE should be either 1, 2, 4 or 8 bytes.
1824 REGISTERS
1827 The value stored within a register can be accessed using the @kregis‐
1828 ter() or @uregister() operators. @kregister() is used for kernel space
1829 registers and @uregister() is used for user space registers. The regis‐
1830 ter of interest is specified using its DWARF number.
1831 @kregister(0)
1833 @uregister(5)
1834
1837 The translator begins pass 1 by parsing the given input script, and all
1838 scripts (files named *.stp) found in a tapset directory. The
1839 directories listed with -I are processed in sequence, each processed in
1840 "guru mode". For each directory, a number of subdirectories are also
1841 searched. These subdirectories are derived from the selected kernel
1842 version (the -R option), in order to allow more kernel-version-specific
1843 scripts to override less specific ones. For example, for a kernel
1844 version 2.6.12-23.FC3 the following patterns would be searched, in
1845 sequence: 2.6.12-23.FC3/*.stp, 2.6.12/*.stp, 2.6/*.stp, and finally
1846 *.stp. Stopping the translator after pass 1 causes it to print the
1847 parse trees.
1848 In pass 2, the translator analyzes the input script to resolve symbols
1851 and types. References to variables, functions, and probe aliases that
1852 are unresolved internally are satisfied by searching through the parsed
1853 tapset script files. If any tapset script file is selected because it
1854 defines an unresolved symbol, then the entirety of that file is added
1855 to the translator's resolution queue. This process iterates until all
1856 symbols are resolved and a subset of tapset script files is selected.
1857 Next, all probe point descriptions are validated against the wide
1859 variety supported by the translator. Probe points that refer to code
1860 locations ("synchronous probe points") require the appropriate kernel
1861 debugging information to be installed. In the associated probe
1862 handlers, target-side variables (whose names begin with "$") are found
1863 and have their run-time locations decoded.
1864 Next, all probes and functions are analyzed for optimization
1866 opportunities, in order to remove variables, expressions, and functions
1867 that have no useful value and no side-effect. Embedded-C functions are
1868 assumed to have side-effects unless they include the magic string
1869 /* pure */. Since this optimization can hide latent code errors such
1870 as type mismatches or invalid $context variables, it sometimes may be
1871 useful to disable the optimizations with the -u option.
1872 Finally, all variable, function, parameter, array, and index types are
1874 inferred from context (literals and operators). Stopping the
1875 translator after pass 2 causes it to list all the probes, functions,
1876 and variables, along with all inferred types. Any inconsistent or
1877 unresolved types cause an error.
1878 In pass 3, the translator writes C code that represents the actions of
1881 all selected script files, and creates a Makefile to build that into a
1882 kernel object. These files are placed into a temporary directory.
1883 Stopping the translator at this point causes it to print the contents
1884 of the C file.
1885 In pass 4, the translator invokes the Linux kernel build system to
1888 create the actual kernel object file. This involves running make in
1889 the temporary directory, and requires a kernel module build system
1890 (headers, config and Makefiles) to be installed in the usual spot
1891 /lib/modules/VERSION/build. Stopping the translator after pass 4 is
1892 the last chance before running the kernel object. This may be useful
1893 if you want to archive the file.
1894 In pass 5, the translator invokes the systemtap auxiliary program
1897 staprun program for the given kernel object. This program arranges to
1898 load the module then communicates with it, copying trace data from the
1899 kernel into temporary files, until the user sends an interrupt signal.
1900 Any run-time error encountered by the probe handlers, such as running
1901 out of memory, division by zero, exceeding nesting or runtime limits,
1902 results in a soft error indication. Soft errors in excess of MAXERRORS
1903 block of all subsequent probes (except error-handling probes), and
1904 terminate the session. Finally, staprun unloads the module, and cleans
1905 up.
1906 ABNORMAL TERMINATION
1909 One should avoid killing the stap process forcibly, for example with
1910 SIGKILL, because the stapio process (a child process of the stap
1911 process) and the loaded module may be left running on the system. If
1912 this happens, send SIGTERM or SIGINT to any remaining stapio processes,
1913 then use rmmod to unload the systemtap module.
1914
1918 See the stapex(3stap) manual page for a brief collection of samples, or
1919 a large set of installed samples under the systemtap
1920 documentation/testsuite directories. See stappaths(7stap) for the
1921 likely location of these on the system.
1922
1925 The systemtap translator caches the pass 3 output (the generated C
1926 code) and the pass 4 output (the compiled kernel module) if pass 4
1927 completes successfully. This cached output is reused if the same
1928 script is translated again assuming the same conditions exist (same
1929 kernel version, same systemtap version, etc.). Cached files are stored
1930 in the $SYSTEMTAP_DIR/cache directory. The cache can be limited by
1931 having the file cache_mb_limit placed in the cache directory (shown
1932 above) containing only an ASCII integer representing how many MiB the
1933 cache should not exceed. In the absence of this file, a default will be
1934 created with the limit set to 256MiB. This is a 'soft' limit in that
1935 the cache will be cleaned after a new entry is added if the cache clean
1936 interval is exceeded, so the total cache size may temporarily exceed
1937 this limit. This interval can be specified by having the file
1938 cache_clean_interval_s placed in the cache directory (shown above)
1939 containing only an ASCII integer representing the interval in seconds.
1940 In the absence of this file, a default will be created with the
1941 interval set to 300 s.
1942
1945 Systemtap may be used as a powerful administrative tool. It can expose
1946 kernel internal data structures and potentially private user
1947 information. (In dyninst runtime mode, this is not the case, see the
1948 ALTERNATE RUNTIMES section below.)
1949 The translator asserts many safety constraints during compilation and
1951 more during run-time. It aims to ensure that no handler routine can
1952 run for very long, allocate boundless memory, perform unsafe
1953 operations, or in unintentionally interfere with the system. Uses of
1954 script global variables are automatically read/write locked as
1955 appropriate, to protect against manipulation by concurrent probe
1956 handlers. Locks are taken so as to run the global-variable
1957 manipulation portion of probe handlers atomically (locks are taken all-
1958 or-none). Deadlocks are detected with timeouts. Use the -t flag to
1959 receive reports of excessive lock contention. Experimenting with
1960 scripts is therefore generally safe. The guru-mode -g option allows
1961 administrators to bypass most safety measures, which permits invasive
1962 or state-changing operations, embedded-C code, and increases the risk
1963 of upset. By default, overload prevention is turned on for all
1964 modules. If you would like to disable overload processing, use the
1965 --suppress-time-limits option.
1966 Errors that are caught at run time normally result in a clean script
1968 shutdown and a pass-5 error message. The --suppress-handler-errors
1969 option lets scripts tolerate soft errors without shutting down.
1970 PERMISSIONS
1974 For the normal linux-kernel-module runtime, to run the kernel objects
1975 systemtap builds, a user must be one of the following:
1976 • the root user;
1978 • a member of the stapdev and stapusr groups;
1980 • a member of the stapsys and stapusr groups; or
1982 • a member of the stapusr group.
1984 The root user or a user who is a member of both the stapdev and stapusr
1986 groups can build and run any systemtap script.
1987 A user who is a member of both the stapsys and stapusr groups can only
1989 use pre-built modules under the following conditions:
1990 • The module has been signed by a trusted signer. Trusted signers are
1992 normally systemtap compile-servers which sign modules when the
1993 --privilege option is specified by the client. See the
1994 stap-server(8) manual page for more information.
1995 • The module was built using the --privilege=stapsys or the
1997 --privilege=stapusr options.
1998 Members of only the stapusr group can only use pre-built modules under
2000 the following conditions:
2001 • The module is located in the /lib/modules/VERSION/systemtap
2003 directory. This directory must be owned by root and not be world
2004 writable.
2005 or
2007 • The module has been signed by a trusted signer. Trusted signers are
2009 normally systemtap compile-servers which sign modules when the
2010 --privilege option is specified by the client. See the
2011 stap-server(8) manual page for more information.
2012 • The module was built using the --privilege=stapusr option.
2014 The kernel modules generated by stap program are run by the staprun
2016 program. The latter is a part of the Systemtap package, dedicated to
2017 module loading and unloading (but only in the white zone), and kernel-
2018 to-user data transfer. Since staprun does not perform any additional
2019 security checks on the kernel objects it is given, it would be unwise
2020 for a system administrator to add untrusted users to the stapdev or
2021 stapusr groups.
2022 SECUREBOOT
2025 If the current system has SecureBoot turned on in the UEFI firmware,
2026 all kernel modules must be signed. (Some kernels may allow disabling
2027 SecureBoot long after booting with a key sequence such as SysRq-X,
2028 making it unnecessary to sign modules.) There are two ways to sign a
2029 systemtap module. The systemtap compile server can sign modules with a
2030 MOK (Machine Owner Key) that it has in common with a client system.
2031 For example:
2032 stap --use-server=HOSTNAME:PORT -e 'SCRIPT'
2034 # If there is no mok key in common with the server's systemtap mok key
2035 # list and the client's mok database then the user is directed by stap
2036 # to invoke:
2037 sudo mokutil --import signing_key.x509
2038 # then after rebooting the system:
2039 stap --use-server=HOSTNAME:PORT -e 'SCRIPT'
2040 # will use the server to build and sign the module and the module will run
2041 # on the client
2042 Another way to sign modules is to use the stap --sign-module option,
2044 which uses a MOK on the client system without using a server. For ex‐
2045 ample:
2046 stap --sign-module -e 'SCRIPT'
2048 # If there is no systemtap mok key in the system mok database
2049 # then the user is directed by stap to invoke:
2050 sudo mokutil --import /home/USER/.systemtap/ssl/server/moks/FINGERPRINT/signing_key.x509
2051 # then after rebooting the system:
2052 stap --sign-module -e 'SCRIPT'
2053 # will sign and run the module
2054 See the following wiki page for more details:
2057 https://sourceware.org/systemtap/wiki/SecureBoot
2059 Some kernels do not let systemtap guess whether module module signing
2061 is in effect. On such machines, set the SYSTEMTAP_SIGN environment
2062 variable to any value while running stap.
2063 RESOURCE LIMITS
2066 Many resource use limits are set by macros in the generated C code.
2067 These may be overridden with -D flags. A selection of these is as fol‐
2068 lows:
2069 MAXNESTING
2071 Maximum number of nested function calls. Default determined by
2072 script analysis, with a bonus 10 slots added for recursive
2073 scripts.
2074 MAXSTRINGLEN
2076 Maximum length of strings, default 128.
2077 MAXTRYLOCK
2079 Maximum number of iterations to wait for locks on global vari‐
2080 ables before declaring possible deadlock and skipping the probe,
2081 default 1000.
2082 MAXACTION
2084 Maximum number of statements to execute during any single probe
2085 hit (with interrupts disabled), default 1000. Note that for
2086 straight-through probe handlers lacking loops or recursion, due
2087 to optimization, this parameter may be interpreted too conserva‐
2088 tively.
2089 MAXACTION_INTERRUPTIBLE
2091 Maximum number of statements to execute during any single probe
2092 hit which is executed with interrupts enabled (such as begin/end
2093 probes), default (MAXACTION * 10).
2094 MAXBACKTRACE
2096 Maximum number of stack frames that will be be processed by the
2097 stap runtime unwinder as produced by the backtrace functions in
2098 the [u]context-unwind.stp tapsets, default 20.
2099 MAXMAPENTRIES
2101 Maximum number of rows in any single global array, default 2048.
2102 Individual arrays may be declared with a larger or smaller limit
2103 instead:
2104 global big[10000],little[5]
2106 or denoted with % to make them wrap-around (replace old entries)
2108 automatically, as in
2109 global big%
2111 or both.
2113 MAPHASHBIAS
2115 The number of powers-of-two to add or subtract from the natural
2116 size of the hash table backing each global associative array.
2117 Default is 0. Try small positive numbers to get extra perfor‐
2118 mance at the cost of more memory consumption, because that
2119 should reduce hash table collisions. Try small negative numbers
2120 for the opposite tradeoff.
2121 MAXERRORS
2123 Maximum number of soft errors before an exit is triggered, de‐
2124 fault 0, which means that the first error will exit the script.
2125 Note that with the --suppress-handler-errors option, this limit
2126 is not enforced.
2127 MAXSKIPPED
2129 Maximum number of skipped probes before an exit is triggered,
2130 default 100. Running systemtap with -t (timing) mode gives more
2131 details about skipped probes. With the default -DINTERRUPT‐
2132 IBLE=1 setting, probes skipped due to reentrancy are not accumu‐
2133 lated against this limit. Note that with the --suppress-han‐
2134 dler-errors option, this limit is not enforced.
2135 MINSTACKSPACE
2137 Minimum number of free kernel stack bytes required in order to
2138 run a probe handler, default 1024. This number should be large
2139 enough for the probe handler's own needs, plus a safety margin.
2140 MAXUPROBES
2142 Maximum number of concurrently armed user-space probes (up‐
2143 robes), default somewhat larger than the number of user-space
2144 probe points named in the script. This pool needs to be poten‐
2145 tially large because individual uprobe objects (about 64 bytes
2146 each) are allocated for each process for each matching script-
2147 level probe.
2148 STP_MAXMEMORY
2150 Maximum amount of memory (in kilobytes) that the systemtap mod‐
2151 ule should use, default unlimited. The memory size includes the
2152 size of the module itself, plus any additional allocations.
2153 This only tracks direct allocations by the systemtap runtime.
2154 This does not track indirect allocations (as done by kprobes/up‐
2155 robes/etc. internals).
2156 STP_OVERLOAD_THRESHOLD, STP_OVERLOAD_INTERVAL
2158 Maximum number of machine cycles spent in probes on any cpu per
2159 given interval, before an overload condition is declared and the
2160 script shut down. The defaults are 500 million and 1 billion,
2161 so as to limit stap script cpu consumption at around 50%.
2162 STP_PROCFS_BUFSIZE
2164 Size of procfs probe read buffers (in bytes). Defaults to
2165 MAXSTRINGLEN. This value can be overridden on a per-procfs file
2166 basis using the procfs read probe .maxsize(MAXSIZE) parameter.
2167 With scripts that contain probes on any interrupt path, it is possible
2169 that those interrupts may occur in the middle of another probe handler.
2170 The probe in the interrupt handler would be skipped in this case to
2171 avoid reentrance. To work around this issue, execute stap with the op‐
2172 tion -DINTERRUPTIBLE=0 to mask interrupts throughout the probe handler.
2173 This does add some extra overhead to the probes, but it may prevent
2174 reentrance for common problem cases. However, probes in NMI handlers
2175 and in the callpath of the stap runtime may still be skipped due to
2176 reentrance.
2177 In case something goes wrong with stap or staprun after a probe has al‐
2180 ready started running, one may safely kill both user processes, and re‐
2181 move the active probe kernel module with rmmod. Any pending trace mes‐
2182 sages may be lost.
2183
2186 Systemtap exposes kernel internal data structures and potentially pri‐
2187 vate user information. Because of this, use of systemtap's full capa‐
2188 bilities are restricted to root and to users who are members of the
2189 groups stapdev and stapusr.
2190 However, a restricted set of systemtap's features can be made available
2192 to trusted, unprivileged users. These users are members of the group
2193 stapusr only, or members of the groups stapusr and stapsys. These
2194 users can load systemtap modules which have been compiled and certified
2195 by a trusted systemtap compile-server. See the descriptions of the op‐
2196 tions --privilege and --use-server. See README.unprivileged in the sys‐
2197 temtap source code for information about setting up a trusted compile
2198 server.
2199 The restrictions enforced when --privilege=stapsys is specified are de‐
2201 signed to prevent unprivileged users from:
2202 • harming the system maliciously.
2204 The restrictions enforced when --privilege=stapusr is specified are de‐
2206 signed to prevent unprivileged users from:
2207 • harming the system maliciously.
2209 • gaining access to information which would not normally be
2211 available to an unprivileged user.
2212 • disrupting the performance of processes owned by other users
2214 of the system. Some overhead to the system in general is
2215 unavoidable since the unprivileged user's probes will be
2216 triggered at the appropriate times. What we would like to
2217 avoid is targeted interruption of another user's processes
2218 which would not normally be possible by an unprivileged us‐
2219 er.
2220 PROBE RESTRICTIONS
2223 A member of the groups stapusr and stapsys may use all probe points.
2224 A member of only the group stapusr may use only the following probes:
2226 • begin, begin(n)
2228 • end, end(n)
2230 • error(n)
2232 • never
2234 • process.*, where the target process is owned by the user.
2236 • timer.{jiffies,s,sec,ms,msec,us,usec,ns,nsec}(n)*
2238 • timer.hz(n)
2240 SCRIPT LANGUAGE RESTRICTIONS
2243 The following scripting language features are unavailable to all un‐
2244 privileged users:
2245 • any feature enabled by the Guru Mode (-g) option.
2248 • embedded C code.
2250 RUNTIME RESTRICTIONS
2253 The following runtime restrictions are placed upon all unprivileged
2254 users:
2255 • Only the default runtime code (see -R) may be used.
2257 Additional restrictions are placed on members of only the group sta‐
2259 pusr:
2260 • Probing of processes owned by other users is not permitted.
2262 • Access of kernel memory (read and write) is not permitted.
2264 COMMAND LINE OPTION RESTRICTIONS
2267 Some command line options provide access to features which must not be
2268 available to all unprivileged users:
2269 • -g may not be specified.
2272 • The following options may not be used by the compile-server
2274 client:
2275 -a, -B, -D, -I, -r, -R
2277 ENVIRONMENT RESTRICTIONS
2281 The following environment variables must not be set for all unprivi‐
2282 leged users:
2283 SYSTEMTAP_RUNTIME
2285 SYSTEMTAP_TAPSET
2286 SYSTEMTAP_DEBUGINFO_PATH
2287 TAPSET RESTRICTIONS
2291 In general, tapset functions are only available for members of the
2292 group stapusr when they do not gather information that an ordinary pro‐
2293 gram running with that user's privileges would be denied access to.
2294 There are two categories of unprivileged tapset functions. The first
2296 category consists of utility functions that are unconditionally avail‐
2297 able to all users; these include such things as:
2298 cpu:long ()
2300 exit ()
2301 str_replace:string (prnt_str:string, srch_str:string, rplc_str:string)
2302 The second category consists of so-called myproc-unprivileged functions
2305 that can only gather information within their own processes. Scripts
2306 that wish to use these functions must test the result of the tapset
2307 function is_myproc and only call these functions if the result is 1.
2308 The script will exit immediately if any of these functions are called
2309 by an unprivileged user within a probe within a process which is not
2310 owned by that user. Examples of myproc-unprivileged functions include:
2311 print_usyms (stk:string)
2313 user_int:long (addr:long)
2314 usymname:string (addr:long)
2315 A compile error is triggered when any function not in either of the
2318 above categories is used by members of only the group stapusr.
2319 No other built-in tapset functions may be used by members of only the
2321 group stapusr.
2322
2325 As described above, systemtap's default runtime mode involves building
2326 and loading kernel modules, with various security tradeoffs presented.
2327 Systemtap now includes two new prototype backends: --runtime=dyninst
2328 and --runtime=bpf.
2329 --runtime=dyninst uses Dyninst to instrument a user's own processes at
2331 runtime. This backend does not use kernel modules, and does not require
2332 root privileges, but is restricted with respect to the kinds of probes
2333 and other constructs that a script may use. dyninst runtime operates in
2334 target-attach mode, so it does require a -c COMMAND or -x PID process.
2335 For example:
2336 stap --runtime=dyninst -c 'stap -V' \
2338 -e 'probe process.function("main")
2339 { println("hi from dyninst!") }'
2340 It may be necessary to disable a conflicting selinux check with
2343 # setsebool allow_execstack 1
2345 --runtime=bpf compiles the user script into extended Berkeley Packet
2348 Filter (eBPF) programs instead of a kernel module. eBPF programs are
2349 verified by the kernel for safety and are executed by an in-kernel vir‐
2350 tual machine. This runtime is in an early stage of development and
2351 currently lacks support for a number of features available in the de‐
2352 fault runtime. Please see the stapbpf(8) man page for more information.
2353
2356 The systemtap translator generally returns with a success code of 0 if
2357 the requested script was processed and executed successfully through
2358 the requested pass. Otherwise, errors may be printed to stderr and a
2359 failure code is returned. Use -v or -vp N to increase (global or per-
2360 pass) verbosity to identify the source of the trouble.
2361 In listings mode (-l and -L), error messages are normally suppressed.
2363 A success code of 0 is returned if at least one matching probe was
2364 found.
2365 A script executing in pass 5 that is interrupted with ^C / SIGINT is
2367 considered to be successful.
2368
2371 Over time, some features of the script language and the tapset library
2372 may undergo incompatible changes, so that a script written against an
2373 old version of systemtap may no longer run. In these cases, it may
2374 help to run systemtap with the --compatible VERSION flag, specifying
2375 the last known working version. Running systemtap with the
2376 --check-version flag will output a warning if any possible incompatible
2377 elements have been parsed. Deprecation historical details may be found
2378 in the NEWS file.
2379 The purpose of deprecation facility is to improve the experience of
2381 scripts written for newer versions of systemtap (by adding better al‐
2382 ternatives and removing conflicting or messy older alternatives), while
2383 at the same time permitting scripts written for older versions of sys‐
2384 temtap to continue running. Deprecation is thus intended a service to
2385 users (and an inconvenience to systemtap's developers), rather than the
2386 other way around.
2387 Please note that underscore-prefixed identifiers in the tapset some‐
2389 times undergo such changes that are difficult to preserve compatibility
2390 for, even with the deprecation mechanisms. Avoid relying on these in
2391 your scripts; instead propose them for promotion to non-underscored
2392 status.
2393
2397 Important files and their corresponding paths can be located in the
2398 stappaths (7) manual page.
2399
2402 stapprobes(3stap),
2403 function::*[24m(3stap),
2404 probe::*[24m(3stap),
2405 tapset::*[24m(3stap),
2406 stappaths(7),
2407 staprun(8),
2408 stapdyn(8),
2409 systemtap(8),
2410 stapvars(3stap),
2411 stapex(3stap),
2412 stap-server(8),
2413 stap-prep(1),
2414 stapref(1),
2415 awk(1),
2416 gdb(1)
2417
2420 Use the Bugzilla link of the project web page or our mailing list.
2421 http://sourceware.org/systemtap/, <systemtap@sourceware.org>.
2422 error::reporting(7stap),
2424 https://sourceware.org/systemtap/wiki/HowToReportBugs
2425 STAP(1)