1prex(1) User Commands prex(1)
2
3
4
6 prex - control tracing and manipulate probe points in a process or the
7 kernel
8
10 prex [-o trace_file_name] [-l libraries] [-s kbytes_size] cmd
11 [cmd-args]...
12
13
14 prex [-o trace_file_name] [-l libraries] [-s kbytes_size] -p pid
15
16
17 prex -k [-s kbytes_size]
18
19
21 The prex command is the part of the Solaris tracing architecture that
22 controls probes in a process or the kernel. See tracing(3TNF) for an
23 overview of this tracing architecture, including example source code
24 using it.
25
26
27 prex is the application used for external control of probes. It auto‐
28 matically preloads the libtnfprobe library. prex locates all the probes
29 in a target executable or the kernel and provides an interface for the
30 user to manipulate them. It allows a probe to be turned on for tracing,
31 debugging, or both. Tracing generates a TNF (Trace Normal Form) trace
32 file that can be converted to ASCII by tnfdump(1) and used for perfor‐
33 mance analysis. Debugging generates a line to standard error whenever
34 the probe is hit at run time.
35
36
37 prex does not work on static executables. It only works on dynamic exe‐
38 cutables.
39
40 Invoking prex
41 There are three ways to invoke prex:
42
43 1. Use prex to start the target application cmd. In this case,
44 the target application need not be built with a dependency
45 on libtnfprobe. See TNF_PROBE(3TNF). prex sets the environ‐
46 ment variable LD_PRELOAD to load libtnfprobe into the target
47 process. See ld(1). prex then uses the environment variable
48 PATH to find the target application.
49
50 2. Attach prex to a running application. In this case, the run‐
51 ning target application should have libtnfprobe already
52 linked in. Alternatively, the user may manually set
53 LD_PRELOAD to include libtnfprobe.so.1 prior to invoking the
54 target.
55
56 3. Use prex with the -k option to set prex to kernel mode. prex
57 can then be used to control probes in the Solaris kernel. In
58 kernel mode, additional commands are defined, and some com‐
59 mands that are valid in other modes are invalid. See Kernel
60 Mode below.
61
62 Control File Format and Command Language
63 In a future release of prex, the command language may be moved to a
64 syntax that is supported by an existing scripting language like ksh(1).
65 In the meantime, the interface to prex is uncommitted.
66
67 o Commands should be in ASCII.
68
69 o Each command is terminated with the NEWLINE character.
70
71 o A command can be continued onto the next line by ending the
72 previous line with the backslash ("\") character.
73
74 o Tokens in a command must be separated by whitespace (one or
75 more spaces or tabs).
76
77 o The "#" character implies that the rest of the line is a
78 comment.
79
80 Basic prex Commands
81 Command Result
82 ───────────────────────────────────────────────────────────
83 % prex a.out Attaches prex to your pro‐
84 gram and starts prex.
85 prex> enable $all Enables all the probes.
86 prex> quit resume Quits prex and resumes exe‐
87 cution of program.
88
89
90 Control File Search Path
91 There are two different methods of communicating with prex:
92
93 o By specifications in a control file. During start-up, prex
94 searches for a file named .prexrc in the directories speci‐
95 fied below. prex does not stop at the first one it finds.
96 This way a user can override any defaults that are set up.
97 The search order is:
98
99 $HOME/
100 ./
101
102
103
104 o By typing commands at the prex prompt.
105
106
107 The command language for both methods is the same and is specified in
108 USAGE. The commands that return output will not make sense in a control
109 file. The output will go to standard output.
110
111
112 When using prex on a target process, the target will be in one of two
113 states, running or stopped. This can be detected by the presence or
114 absence of the prex> prompt. If the prompt is absent, it means that the
115 target process is running. Typing Control-C will stop the target pr
116 ocess and return the user to the prompt. There is no guarantee that
117 Control-C will return to a prex prompt immediately. For example, if the
118 target process is stopped on a job control stop (SIGSTOP), then Con‐
119 trol-C in prex will wait until the target has been continued (SIGCONT).
120 See Signals to Target Program below for more information on signals and
121 the target process.
122
124 The following options are supported:
125
126 -k kernel mode: prex is used to control probes in
127 the Solaris kernel. In kernel mode, additional
128 commands are defined, and some commands valid in
129 other modes are invalid. See Kernel Mode below.
130
131
132 -l libraries The libraries mentioned are linked in to the tar‐
133 get application using LD_PRELOAD (see ld(1)).
134 This option cannot be used when attaching to a
135 running process. The argument to the -l option
136 should be a space-separated string enclosed in
137 double quotes. Each token in the string is a
138 library name. It follows the LD_PRELOAD rules on
139 how libraries should be specified and where they
140 will be found.
141
142
143 -o trace_file_name File to be used for the trace output.
144 trace_file_name is assumed to be relative to the
145 current working directory of prex (that is, the
146 directory that the user was in when prex was
147 started).
148
149 If prex attaches to a process that is already
150 tracing, the new trace_file_name (if provided)
151 will not be used. If no trace_file_name is speci‐
152 fied, the default is /$TMPDIR/trace-pid where pid
153 is the process id of the target program. If
154 TMPDIR is not set, /tmp is used.
155
156
157 -s kbytes_size Maximum size of the output trace file in Kbytes.
158 The default size of the trace kbytes_size is 4096
159 (2^10) bytes or 4 Mbytes for normal usage, and
160 384 or 384 kbytes in kernel mode. The minimum
161 size that can be specified is 128 Kbytes. The
162 trace file can be thought of as a least recently
163 used circular buffer. Once the file has been
164 filled, newer events will overwrite the older
165 ones.
166
167
169 This section describes the usage of the prex utility.
170
171 Grammar
172 Probes are specified by a list of space-separated selectors. Selectors
173 are of the form:
174
175 attribute=value
176
177
178
179 (See TNF_PROBE(3TNF)). The "attribute=" is optional. If it is not spec‐
180 ified, it defaults to "keys=".
181
182
183 The attribute or value (generically called "spec") can be any of the
184 following:
185
186 IDENT Any sequence of letters, digits, _, \, ., % not beginning
187 with a digit. IDENT implies an exact match.
188
189
190 QUOTED_STR Usually used to escape reserved words (any commands in
191 the command language). QUOTED_STR implies an exact match
192 and has to be enclosed in single quotes (' ').
193
194
195 REGEXP An ed(1) regular expression pattern match. REGEXP has to
196 be enclosed in slashes (/ /), A / can be included in a
197 REGEXP by escaping it with a backslash \.
198
199
200
201 The following grammar explains the syntax.
202
203 selector_list ::= | /* empty */
204 selector_list selector
205 selector ::= spec=spec | /* whitespace around `=' opt */
206 spec
207 spec ::= IDENT |
208 QUOTED_STR |
209 REGEXP
210
211
212
213 The terminals in the above grammar are:
214
215 IDENT = [a-zA-Z_\.%]{[a-zA-Z0-9_\.%]}+
216 QUOTED_STR = '[^\n']*' /* any string in single quotes */
217 REGEXP = /[^\n/]*/ /* regexp's have to be in / / */
218
219
220
221 This is a list of the remaining grammar that is needed to understand
222 the syntax of the command language (defined in next subsection):
223
224 filename ::= QUOTED_STR /* QUOTED_STR defined above */
225 spec_list ::= /* empty */ |
226 spec_list spec /* spec defined above */
227 fcn_handle ::= &IDENT /* IDENT defined above */
228 set_name ::= $IDENT /* IDENT defined above */
229
230
231 Command Language
232 1. Set Creation and Set Listing
233
234 create $set_name selector_list
235 list sets # list the defined sets
236
237
238 create can be used to define a set which contains probes
239 that match the selector_list. The set $all is pre-defined as
240 /.*/ and it matches all the probes.
241
242 2. Function Listing
243
244 list fcns # list the available fcn_handle
245
246
247 The user can list the different functions that can be con‐
248 nected to probe points. Currently, only the debug function
249 called &debug is available.
250
251 3. Commands to Connect and Disconnect Probe Functions
252
253 connect &fcn_handle $set_name
254 connect &fcn_handle selector_list
255 clear $set_name
256 clear selector_list
257
258
259 The connect command is used to connect probe functions
260 (which must be prefixed by `&') to probes. The probes are
261 specified either as a single set (with a `$'), or by explic‐
262 itly listing the probe selectors in the command. The probe
263 function has to be one that is listed by the list fcns com‐
264 mand. This command does not enable the probes.
265
266 The clear command is used to disconnect all connected probe
267 functions from the specified probes.
268
269 4. Commands to Toggle the Tracing Mode
270
271 trace $set_name
272 trace selector_list
273 untrace $set_name
274 untrace selector_list
275
276
277 The trace and untrace commands are used to toggle the trac‐
278 ing action of a probe point (that is, whether a probe will
279 emit a trace record or not if it is hit). This command does
280 not enable the probes specified. Probes have tracing on by
281 default. The most efficient way to turn off tracing is by
282 using the disable command. untrace is useful if you want
283 debug output but no tracing. If so, set the state of the
284 probe to enabled, untraced, and the debug function con‐
285 nected.
286
287 5. Commands to Enable and Disable Probes
288
289 enable $set_name
290 enable selector_list
291 disable $set_name
292 disable selector_list
293
294
295 The enable and disable commands are used to control whether
296 the probes perform the action that they have been set up
297 for. To trace a probe, it has to be both enabled and traced
298 (using the trace command). Probes are disabled by default.
299 The list history command is used to list the probe control
300 commands issued: connect, clear, trace, untrace, enable, and
301 disable. These are the commands that are executed whenever a
302 new shared object is brought in to the target program by
303 dlopen(3C). See the subsection, dlopen'ed Libraries, below
304 for more information.
305
306 The following table shows the actions that result from spe‐
307 cific combinations of tracing, enabling, and connecting:
308
309 Enabled or Tracing State Debug State Results
310 Disabled (On/Off) (Connected/Cleared) In
311 ------------------------------------------------------------
312 Enabled On Connected Tracing and
313 Debugging
314
315 Enabled On Cleared Tracing only
316
317 Enabled Off Connected Debugging only
318
319 Enabled Off Cleared Nothing
320
321 Disabled On Connected Nothing
322
323 Disabled On Cleared Nothing
324
325 Disabled Off Connected Nothing
326
327 Disabled Off Cleared Nothing
328
329
330
331 6. List History
332
333 list history # lists probe control command history
334
335
336 The list history command displays a list of the probe con‐
337 trol commands previously issued in the tracing session, for
338 example, connect, clear, trace, disable. Commands in the
339 history list are executed wherever a new shared object is
340 brought into the target program by dlopen(3C).
341
342 7. Commands to List Probes, List Values, or List Trace File
343 Name
344
345 list spec_list probes $set_name # list probes $all
346 list spec_list probes selector_list # list name probes file=test.c
347 list values spec_list # list values keys given in spec_list
348 list tracefile # list tracefile
349
350
351 The first two commands list the selected attributes and val‐
352 ues of the specified probes. They can be used to check the
353 state of a probe. The third command lists the various values
354 associated with the selected attributes. The fourth command
355 lists the current tracefile.
356
357 8. Help Command
358
359 help topic
360
361
362 To get a list of the help topics that are available, invoke
363 the help command with no arguments. If a topic argument is
364 specified, help is printed for that topic.
365
366 9. Source a File
367
368 source filename
369
370
371 The source command can be used to source a file of prex com‐
372 mands. source can be nested (that is, a file can source
373 another file). filename is a quoted string.
374
375 10. Process Control
376
377 continue # resumes the target process
378 quit kill # quit prex, kill target
379 quit resume # quit prex, continue target
380 quit suspend # quit prex, leave target suspended
381 quit # quit prex (continue or kill target)
382
383
384 The default quit will continue the target process if prex
385 attached to it. Instead, if prex had started the target pro‐
386 gram, quit will kill the target process.
387
388 dlopen'ed Libraries
389 Probes in shared objects that are brought in by dlopen(3C) are automat‐
390 ically set up according to the command history of prex. When a shared
391 object is removed by a dlclose(3C), prex again needs to refresh its
392 understanding of the probes in the target program. This implies that
393 there is more work to do for dlopen(3C) and dlclose(3C) —so they will
394 take slightly longer. If a user is not interested in this feature and
395 doesn't want to interfere with dlopen(3C) and dlclose(3C), detach prex
396 from the target to inhibit this feature.
397
398 Signals to Target Program
399 prex does not interfere with signals that are delivered directly to the
400 target program. However, prex receives all signals normally generated
401 from the terminal, for example, Control-C (SIGINT), and Control-Z
402 (SIGSTOP), and does not forward them to the target program. To signal
403 the target program, use the kill(1) command from a shell.
404
405 Interactions with Other Applications
406 Process managing applications like dbx, truss(1), and prex cannot oper‐
407 ate on the same target program simultaneously. prex will not be able to
408 attach to a target which is being controlled by another application. A
409 user can trace and debug a program serially by the following method:
410 first attach prex to target (or start target through prex), set up the
411 probes using the command language, and then type quit suspend. The user
412 can then attach dbx to the suspended process and debug it. A user can
413 also suspend the target by sending it a SIGSTOP signal, and then by
414 typing quit resume to prex. In this case, the user should also send a
415 SIGCONT signal after invoking dbx on the stopped process (else dbx will
416 be hung).
417
418 Failure of Event Writing Operations
419 There are a few failure points that are possible when writing out
420 events to a trace file, for example, system call failures. These fail‐
421 ures result in a failure code being set in the target process. The tar‐
422 get process continues normally, but no trace records are written. When‐
423 ever a user enters Control-C to prex to get to a prex prompt, prex will
424 check the failure code in the target and inform the user if there was a
425 tracing failure.
426
427 Target Executing a Fork or exec
428 If the target program does a fork(2), any probes that the child encoun‐
429 ters will cause events to be logged to the same trace file. Events are
430 annotated with a process id, so it will be possible to determine which
431 process a particular event came from. In multi-threaded programs, there
432 is a race condition with a thread doing a fork while the other threads
433 are still running. For the trace file not to get corrupted, the user
434 should either use fork1(2), or make sure that all other threads are
435 quiescent when doing a fork(2),
436
437
438 If the target program itself (not any children it may fork(2)) does an
439 exec(2), prex detaches from the target and exits. The user can recon‐
440 nect prex with prex -p pid.
441
442
443 A vfork(2) is generally followed quickly by an exec(2) in the child,
444 and in the interim, the child borrows the parent's process while the
445 parent waits for the exec(2). Any events logged by the child from the
446 parent process will appear to have been logged by the parent.
447
448 Kernel Mode
449 Invoking prex with the -k flag causes prex to run in kernel mode. In
450 kernel mode, prex controls probes in the Solaris kernel. See tnf_ker‐
451 nel_probes(4) for a list of available probes in the Solaris kernel. A
452 few prex commands are unavailable in kernel mode; many other commands
453 are valid in kernel mode only.
454
455
456 The -l, -o, and -p command-line options are not valid in kernel mode
457 (that is, they may not be combined with the -k flag).
458
459
460 The rest of this section describes the differences in the prex command
461 language when running prex in kernel mode.
462
463 1. prex will not stop the kernel
464
465 When prex attaches to a running user program, it stops the
466 user program. Obviously, it cannot do this when attaching to
467 the kernel. Instead, prex provides a ``tracing master
468 switch'': no probes will have any effect unless the tracing
469 master switch is on. This allows the user to iteratively
470 select probes to enable, then enable them all at once by
471 turning on the master switch.
472
473 The command
474
475 ktrace [ on | off ]
476
477
478 is used to inspect and set the value of the master switch.
479 Without an argument, prex reports the current state of the
480 master switch.
481
482 Since prex will not stop or kill the kernel, the
483
484 quit resume
485
486
487 and
488
489 quit kill
490
491
492 commands are not valid in kernel mode.
493
494 2. No functions may be attached to probes in the kernel
495
496 In particular, the debug function is unavailable in kernel
497 mode.
498
499 3. Trace output is written to an in-core buffer
500
501 In kernel mode, a trace output file is not generated
502 directly, in order to allow probes to be placed in time-
503 critical code. Instead, trace output is written to an in-
504 core buffer, and copied out by a separate program, tnfx‐
505 tract(1).
506
507 The in-core buffer is not automatically created. The follow‐
508 ing prex command controls buffer allocation and dealloca‐
509 tion:
510
511 buffer [ alloc [ size ] | dealloc ]
512
513
514 Without an argument, the buffer command reports the size of
515 the currently allocated buffer, if any. With an argument of
516 alloc [size], prex allocates a buffer of the given size.
517 size is in bytes, with an optional suffix of 'k' or 'm'
518 specifying a multiplier of 1024 or 1048576, respectively. If
519 no size is specified, the size specified on the command
520 line with the -s option is used as a default. If the -s com‐
521 mand line option was not used, the ``default default'' is
522 384 kilobytes.
523
524 With an argument of dealloc, prex deallocates the trace buf‐
525 fer in the kernel.
526
527 prex will reject attempts to turn the tracing master switch
528 on when no buffer is allocated, and to deallocate the buffer
529 when the tracing master switch is on. prex will refuse to
530 allocate a buffer when one is already allocated; use buffer
531 dealloc first.
532
533 prex will not allocate a buffer larger than one-half of a
534 machine's physical memory.
535
536 4. prex supports per-process probe enabling in the kernel
537
538 In kernel mode, it is possible to select a set of processes
539 for which probes are enabled. No trace output will be writ‐
540 ten when other processes traverse these probe points. This
541 is called "process filter mode". By default, process filter
542 mode is off, and all processes cause the generation of trace
543 records when they hit an enabled probe.
544
545 Some kernel events such as interrupts cannot be associated
546 with a particular user process. By convention, these events
547 are considered to be generated by process id 0.
548
549 prex provides commands to turn process filter mode on and
550 off, to get the current status of the process filter mode
551 switch, to add and delete processes (by process id) from the
552 process filter set, and to list the current process filter
553 set.
554
555 The process filter set is maintained even when process fil‐
556 ter mode is off, but has no effect unless process filter
557 mode is on.
558
559 When a process in the process filter set exits, its process
560 id is automatically deleted from the process filter set.
561
562 The command:
563
564 pfilter [ on | off | add pidlist | delete pidlist ]
565
566
567 controls the process filter switch, and process filter set
568 membership. With no arguments, pfilter prints the current
569 process filter set and the state of the process filter mode
570 switch:
571
572 on or off set the state of the process filter mode switch.
573
574
575 add pidlist add or delete processes from the process filter
576 delete pidlist set. pidlist is a comma-separated list of one or
577 more process ids.
578
579
581 See tracing(3TNF) for complete examples showing, among other things,
582 the use of prex to do simple probe control.
583
584
585 When either the process or kernel is started, all probes are disabled.
586
587 Example 1 Set creation and set listing
588
589 create $out name=/out/ # $out = probes with "out" in
590 # value of "name" attribute
591 create $foo /page/ name=biodone # $foo = union of
592 # probes with "page" in value of keys attribute
593 # probes with "biodone" as value of "name" attribute
594 list sets # list the defined sets
595 list fcns # list the defined probe fcns
596
597
598
599 Example 2 Commands to trace and connect probe functions
600
601 trace foobar='on' # exact match on foobar attribute
602 trace $all # trace all probes (predefined set $all)
603 connect &debug $foo # connect debug func to probes in $foo
604
605
606
607 Example 3 Commands to enable and disable probes
608
609 enable $all # enable all probes
610 enable /vm/ name=alloc # enable the specified probes
611 disable $foo # disable probes in set $foo
612 list history # list probe control commands issued
613
614
615
616 Example 4 Process control
617
618 continue # resumes the target process
619 ^C # stop target; give control to prex
620 quit resume # exit prex, leave process running
621 # and resume execution of program
622
623
624
625 Example 5 Kernel mode
626
627 buffer alloc 2m # allocate a 2 Megabyte buffer
628 enable $all # enable all probes
629 trace $all # trace all probes
630 ktrace on # turn tracing on
631 ktrace off # turn tracing back off
632 pfilter on # turn process filter mode on
633 pfilter add 1379 # add pid 1379 to process filter
634 ktrace on # turn tracing on
635 # (only pid 1379 will be traced)
636
637
638
640 .prexrc local prex initialization file
641
642
643 ~/.prexrc user's prex initialization file
644
645
646 /proc/nnnnn process files
647
648
650 See attributes(5) for descriptions of the following attributes:
651
652
653
654
655 ┌─────────────────────────────┬─────────────────────────────┐
656 │ ATTRIBUTE TYPE │ ATTRIBUTE VALUE │
657 ├─────────────────────────────┼─────────────────────────────┤
658 │Availability │SUNWtnfc │
659 └─────────────────────────────┴─────────────────────────────┘
660
662 ed(1), kill(1), ksh(1), ld(1), tnfdump(1), tnfxtract(1), truss(1),
663 exec(2), fork(2), fork1(2), vfork(2), TNF_DECLARE_RECORD(3TNF),
664 TNF_PROBE(3TNF), dlclose(3C), dlopen(3C), gethrtime(3C), libtn‐
665 fctl(3TNF), tnf_process_disable(3TNF), tracing(3TNF), tnf_ker‐
666 nel_probes(4), attributes(5)
667
669 Currently, the only probe function that is available is the &debug
670 function. When this function is executed, it prints out the arguments
671 sent in to the probe as well as the value associated with the
672 sunw%debug attribute in the detail field (if any) to stderr.
673
674
675 For example, for the following probe point:
676
677 TNF_PROBE_2(input_values, "testapp main",
678 "sunw%debug 'have read input values successfully'",
679 tnf_long, int_input, x,
680 tnf_string, string_input, input);
681
682
683
684
685 If x was 100 and input was the string "success", then the output of the
686 debug probe function would be:
687
688 probe input_values; sunw%debug "have read input values successfully";
689 int_input=100; string_input="success";
690
691
692
693
694 Some non-SPARC hardware lacks a true high-resolution timer, causing
695 gethrtime() to return the same value multiple times in succession. This
696 can lead to problems in how some tools interpret the trace file. This
697 situation can be improved by interposing a version of gethrtime(),
698 which causes these successive values to be artificially incremented by
699 one nanosecond:
700
701 hrtime_t
702 gethrtime()
703 {
704 static mutex_t lock;
705 static hrtime_t (*real_gethrtime)(void) = NULL;
706 static hrtime_t last_time = 0;
707
708 hrtime_t this_time;
709
710 if (real_gethrtime == NULL) {
711 real_gethrtime =
712 (hrtime_t (*)(void)) dlsym(RTLD_NEXT, "gethrtime");
713 }
714 this_time = real_gethrtime();
715
716 mutex_lock(&lock);
717 if (this_time <= last_time)
718 this_time = ++last_time;
719 else
720 last_time = this_time;
721 mutex_unlock(&lock);
722
723 return (this_time);
724 }
725
726
727
728
729 Of course, this does not increase the resolution of the timer, so time‐
730 stamps for individual events are still relatively inaccurate. But this
731 technique maintains ordering, so that if event A causes event B, B
732 never appears to happen before or at the same time as A.
733
734
735 dbx is available with the Sun Workshop Products.
736
738 prex should issue a notification when a process id has been automati‐
739 cally deleted from the filter set.
740
741
742 There is a known bug in prex which can result in this message:
743
744 Tracing shut down in target program due to an internal
745 error - Please restart prex and target
746
747
748
749
750 When prex runs as root, and the target process is not root, and the
751 tracefile is placed in a directory where it cannot be removed and re-
752 created (a directory with the sticky bit on, like /tmp),mm then the
753 target process will not be able to open the tracefile when it needs to.
754 This results in tracing being disabled.
755
756
757 Changing any of the circumstances listed above should fix the problem.
758 Either don't run prex as root, or run the target process as root, or
759 specify the tracefile in a directory other than /tmp.
760
761
762
763SunOS 5.11 1 Mar 2004 prex(1)