1LTTNG-CONCEPTS(7)                LTTng Manual                LTTNG-CONCEPTS(7)
2
3
4

NAME

6       lttng-concepts - LTTng concepts
7

DESCRIPTION

9       This manual page documents the concepts of LTTng.
10
11       Many other LTTng manual pages refer to this one so that you can
12       understand what are the various LTTng objects and how they relate to
13       each other.
14
15       The concepts of LTTng 2.13.4 are:
16
17       •   Instrumentation point, event rule, and event
18
19       •   Trigger
20
21       •   Recording session
22
23       •   Tracing domain
24
25       •   Channel and ring buffer
26
27       •   Recording event rule and event record
28

INSTRUMENTATION POINT, EVENT RULE, AND EVENT

30       An instrumentation point is a point, within a piece of software, which,
31       when executed, creates an LTTng event.
32
33       LTTng offers various types of instrumentation; see the “Instrumentation
34       point types” section below to learn about them.
35
36       An event rule is a set of conditions to match a set of events.
37
38       When LTTng creates an event E, an event rule ER is said to match E
39       when E satisfies all the conditions of ER. This concept is similar to a
40       regular expression which matches a set of strings.
41
42       When an event rule matches an event, LTTng emits the event, therefore
43       attempting to execute one or more actions.
44
45       Important
46           The event creation and emission processes are documentation
47           concepts to help understand the journey from an instrumentation
48           point to the execution of actions.
49
50           The actual creation of an event can be costly because LTTng needs
51           to evaluate the arguments of the instrumentation point.
52
53           In practice, LTTng implements various optimizations for the Linux
54           kernel and user space tracing domains (see the “TRACING DOMAIN”
55           section below) to avoid actually creating an event when the tracer
56           knows, thanks to properties which are independent from the event
57           payload and current context, that it would never emit such an
58           event. Those properties are:
59
60           •   The instrumentation point type (see the “Instrumentation point
61               types” section below).
62
63           •   The instrumentation point name.
64
65           •   The instrumentation point log level.
66
67           •   For a recording event rule (see the “RECORDING EVENT RULE AND
68               EVENT RECORD” section below):
69
70               •   The status of the rule itself.
71
72               •   The status of the channel (see the “CHANNEL AND RING
73                   BUFFER” section below).
74
75               •   The activity of the recording session (started or stopped;
76                   see the “RECORDING SESSION” section below).
77
78               •   Whether or not the process for which LTTng would create the
79                   event is allowed to record events (see lttng-track(1)).
80
81           In other words: if, for a given instrumentation point IP, the LTTng
82           tracer knows that it would never emit an event, executing IP
83           represents a simple boolean variable check and, for a Linux kernel
84           recording event rule, a few process attribute checks.
85
86       As of LTTng 2.13.4, there are two places where you can find an event
87       rule:
88
89       Recording event rule
90           A specific type of event rule of which the action is to record the
91           matched event as an event record.
92
93           See the “RECORDING EVENT RULE AND EVENT RECORD” section below.
94
95           Create or enable a recording event rule with the lttng-enable-
96           event(1) command.
97
98           List the recording event rules of a specific recording session
99           and/or channel with the lttng-list(1) and lttng-status(1) commands.
100
101       “Event rule matches” trigger condition (since LTTng 2.13)
102           When the event rule of the trigger condition matches an event,
103           LTTng can execute user-defined actions such as sending an LTTng
104           notification, starting a recording session, and more.
105
106           See lttng-add-trigger(1) and lttng-event-rule(7).
107
108       For LTTng to emit an event E, E must satisfy all the basic conditions
109       of an event rule ER, that is:
110
111       •   The instrumentation point from which LTTng creates E has a specific
112           type.
113
114           See the “Instrumentation point types” section below.
115
116       •   A pattern matches the name of E while another pattern doesn’t.
117
118       •   The log level of the instrumentation point from which LTTng
119           creates E is at least as severe as some value, or is exactly some
120           value.
121
122       •   The fields of the payload of E and the current context fields
123           satisfy a filter expression.
124
125       A recording event rule has additional, implicit conditions to satisfy.
126       See the “RECORDING EVENT RULE AND EVENT RECORD” section below to learn
127       more.
128
129   Instrumentation point types
130       As of LTTng 2.13.4, the available instrumentation point types are,
131       depending on the tracing domain (see the “TRACING DOMAIN” section
132       below):
133
134       Linux kernel
135
136           LTTng tracepoint
137               A statically defined point in the source code of the kernel
138               image or of a kernel module using the LTTng-modules macros.
139
140               List the available Linux kernel tracepoints with lttng list
141               --kernel. See lttng-list(1) to learn more.
142
143           Linux kernel system call
144               Entry, exit, or both of a Linux kernel system call.
145
146               List the available Linux kernel system call instrumentation
147               points with lttng list --kernel --syscall. See lttng-list(1) to
148               learn more.
149
150           Linux kprobe
151               A single probe dynamically placed in the compiled kernel code.
152
153               When you create such an instrumentation point, you set its
154               memory address or symbol name.
155
156           Linux user space probe
157               A single probe dynamically placed at the entry of a compiled
158               user space application/library function through the kernel.
159
160               When you create such an instrumentation point, you set:
161
162               With the ELF method
163                   Its application/library path and its symbol name.
164
165               With the USDT method
166                   Its application/library path, its provider name, and its
167                   probe name.
168
169                   “USDT” stands for SystemTap User-level Statically Defined
170                   Tracing, a DTrace-style marker.
171
172               As of LTTng 2.13.4, LTTng only supports USDT probes which are
173               NOT reference-counted.
174
175           Linux kretprobe
176               Entry, exit, or both of a Linux kernel function.
177
178               When you create such an instrumentation point, you set the
179               memory address or symbol name of its function.
180
181       User space
182
183           LTTng tracepoint
184               A statically defined point in the source code of a C/C++
185               application/library using the LTTng-UST macros.
186
187               List the available Linux kernel tracepoints with lttng list
188               --userspace. See lttng-list(1) to learn more.
189
190       java.util.logging, Apache log4j, and Python
191
192           Java or Python logging statement
193               A method call on a Java or Python logger attached to an
194               LTTng-UST handler.
195
196               List the available Java and Python loggers with lttng list
197               --jul, lttng list --log4j, and lttng list --python. See lttng-
198               list(1) to learn more.
199

TRIGGER

201       A trigger associates a condition to one or more actions.
202
203       When the condition of a trigger is satisfied, LTTng attempts to execute
204       its actions.
205
206       As of LTTng 2.13.4, the available trigger conditions and actions are:
207
208       Conditions
209
210           •   The consumed buffer size of a given recording session (see the
211               “RECORDING SESSION” section below) becomes greater than some
212               value.
213
214           •   The buffer usage of a given channel (see the “CHANNEL AND RING
215               BUFFER” section below) becomes greater than some value.
216
217           •   The buffer usage of a given channel becomes less than some
218               value.
219
220           •   There’s an ongoing recording session rotation (see the
221               “Recording session rotation” section below).
222
223           •   A recording session rotation becomes completed.
224
225           •   An event rule matches an event.
226
227               As of LTTng 2.13.4, this is the only available condition when
228               you add a trigger with the lttng-add-trigger(1) command. The
229               other ones are available through the liblttng-ctl C API.
230
231       Actions
232
233           •   Send a notification to a user application.
234
235           •   Start a given recording session, like lttng-start(1) would do.
236
237           •   Stop a given recording session, like lttng-stop(1) would do.
238
239           •   Archive the current trace chunk of a given recording session
240               (rotate), like lttng-rotate(1) would do.
241
242           •   Take a snapshot of a given recording session, like lttng-
243               snapshot(1) would do.
244
245       A trigger belongs to a session daemon (see lttng-sessiond(8)), not to a
246       specific recording session. For a given session daemon, each Unix user
247       has its own, private triggers. Note, however, that the root Unix user
248       may, for the root session daemon:
249
250       •   Add a trigger as another Unix user.
251
252       •   List all the triggers, regardless of their owner.
253
254       •   Remove a trigger which belongs to another Unix user.
255
256       For a given session daemon and Unix user, a trigger has a unique name.
257
258       Add a trigger to a session daemon with the lttng-add-trigger(1)
259       command.
260
261       List the triggers of your Unix user (or of all users if your Unix user
262       is root) with the lttng-list-triggers(1) command.
263
264       Remove a trigger with the lttng-remove-trigger(1) command.
265

RECORDING SESSION

267       A recording session (named “tracing session” prior to LTTng 2.13) is a
268       stateful dialogue between you and a session daemon (see lttng-
269       sessiond(8)) for everything related to event recording.
270
271       Everything that you do when you control LTTng tracers to record events
272       happens within a recording session. In particular, a recording session:
273
274       •   Has its own name, unique for a given session daemon.
275
276       •   Has its own set of trace files, if any.
277
278       •   Has its own state of activity (started or stopped).
279
280           An active recording session is an implicit recording event rule
281           condition (see the “RECORDING EVENT RULE AND EVENT RECORD” section
282           below).
283
284       •   Has its own mode (local, network streaming, snapshot, or live).
285
286           See the “Recording session modes” section below to learn more.
287
288       •   Has its own channels (see the “CHANNEL AND RING BUFFER” section
289           below) to which are attached their own recording event rules.
290
291       •   Has its own process attribute inclusion sets (see lttng-track(1)).
292
293       Those attributes and objects are completely isolated between different
294       recording sessions.
295
296       A recording session is like an ATM session: the operations you do on
297       the banking system through the ATM don’t alter the data of other users
298       of the same system. In the case of the ATM, a session lasts as long as
299       your bank card is inside. In the case of LTTng, a recording session
300       lasts from the lttng-create(1) command to the lttng-destroy(1) command.
301
302       A recording session belongs to a session daemon (see lttng-
303       sessiond(8)). For a given session daemon, each Unix user has its own,
304       private recording sessions. Note, however, that the root Unix user may
305       operate on or destroy another user’s recording session.
306
307       Create a recording session with the lttng-create(1) command.
308
309       List the recording sessions of the connected session daemon with the
310       lttng-list(1) command.
311
312       Start and stop a recording session with the lttng-start(1) and lttng-
313       stop(1) commands.
314
315       Save and load a recording session with the lttng-save(1) and lttng-
316       load(1) commands.
317
318       Archive the current trace chunk of (rotate) a recording session with
319       the lttng-rotate(1) command.
320
321       Destroy a recording session with the lttng-destroy(1) command.
322
323   Current recording session
324       When you run the lttng-create(1) command, LTTng creates the
325       $LTTNG_HOME/.lttngrc file if it doesn’t exist ($LTTNG_HOME defaults to
326       $HOME).
327
328       $LTTNG_HOME/.lttngrc contains the name of the current recording
329       session.
330
331       When you create a new recording session with the create command, LTTng
332       updates the current recording session.
333
334       The following lttng(1) commands select the current recording session if
335       you don’t specify one:
336
337lttng-add-context(1)
338
339lttng-clear(1)
340
341lttng-destroy(1)
342
343lttng-disable-channel(1)
344
345lttng-disable-event(1)
346
347lttng-disable-rotation(1)
348
349lttng-enable-channel(1)
350
351lttng-enable-event(1)
352
353lttng-enable-rotation(1)
354
355lttng-regenerate(1)
356
357lttng-rotate(1)
358
359lttng-save(1)
360
361lttng-snapshot(1)
362
363lttng-start(1)
364
365lttng-status(1)
366
367lttng-stop(1)
368
369lttng-track(1)
370
371lttng-untrack(1)
372
373lttng-view(1)
374
375       Set the current recording session manually with the lttng-set-
376       session(1) command, without having to edit the .lttngrc file.
377
378   Recording session modes
379       LTTng offers four recording session modes:
380
381       Local mode
382           Write the trace data to the local file system.
383
384       Network streaming mode
385           Send the trace data over the network to a listening relay daemon
386           (see lttng-relayd(8)).
387
388       Snapshot mode
389           Only write the trace data to the local file system or send it to a
390           listening relay daemon (lttng-relayd(8)) when LTTng takes a
391           snapshot.
392
393           LTTng forces all the channels (see the “CHANNEL AND RING BUFFER”
394           section below) to be created to be configured to be snapshot-ready.
395
396           LTTng takes a snapshot of such a recording session when:
397
398           •   You run the lttng-snapshot(1) command.
399
400           •   LTTng executes a snapshot-session trigger action (see the
401               “TRIGGER” section above).
402
403       Live mode
404           Send the trace data over the network to a listening relay daemon
405           (see lttng-relayd(8)) for live reading.
406
407           An LTTng live reader (for example, babeltrace2(1)) can connect to
408           the same relay daemon to receive trace data while the recording
409           session is active.
410
411   Recording session rotation
412       A recording session rotation is the action of archiving the current
413       trace chunk of the recording session to the file system.
414
415       Once LTTng archives a trace chunk, it does NOT manage it anymore: you
416       can read it, modify it, move it, or remove it.
417
418       An archived trace chunk is a collection of metadata and data stream
419       files which form a self-contained LTTng trace. See the “Trace chunk
420       naming” section below to learn how LTTng names a trace chunk archive
421       directory.
422
423       The current trace chunk of a given recording session includes:
424
425       •   The stream files which LTTng already wrote to the file system, and
426           which are not part of a previously archived trace chunk, since the
427           most recent event amongst:
428
429           •   The first time the recording session was started, either with
430               the lttng-start(1) command or with a start-session trigger
431               action (see the “TRIGGER” section above).
432
433           •   The last rotation, performed with:
434
435               •   An lttng-rotate(1) command.
436
437               •   A rotation schedule previously set with lttng-enable-
438                   rotation(1).
439
440               •   An executed rotate-session trigger action (see the
441                   “TRIGGER” section above).
442
443       •   The content of all the non-flushed sub-buffers of the channels of
444           the recording session.
445
446   Trace chunk archive naming
447       A trace chunk archive is a subdirectory of the archives subdirectory
448       within the output directory of a recording session (see the --output
449       option of the lttng-create(1) command and of lttng-relayd(8)).
450
451       A trace chunk archive contains, through tracing domain and possibly
452       UID/PID subdirectories, metadata and data stream files.
453
454       A trace chunk archive is, at the same time:
455
456       •   A self-contained LTTng trace.
457
458       •   A member of a set of trace chunk archives which form the complete
459           trace of a recording session.
460
461       In other words, an LTTng trace reader can read both the recording
462       session output directory (all the trace chunk archives), or a single
463       trace chunk archive.
464
465       When LTTng performs a recording session rotation, it names the
466       resulting trace chunk archive as such, relative to the output directory
467       of the recording session:
468
469           archives/BEGIN-END-ID
470
471       BEGIN
472           Date and time of the beginning of the trace chunk archive with the
473           ISO 8601-compatible YYYYmmddTHHMMSS±HHMM form, where YYYYmmdd is
474           the date and HHMMSS±HHMM is the time with the time zone offset from
475           UTC.
476
477           Example: 20171119T152407-0500
478
479       END
480           Date and time of the end of the trace chunk archive with the
481           ISO 8601-compatible YYYYmmddTHHMMSS±HHMM form, where YYYYmmdd is
482           the date and HHMMSS±HHMM is the time with the time zone offset from
483           UTC.
484
485           Example: 20180118T152407+0930
486
487       ID
488           Unique numeric identifier of the trace chunk within its recording
489           session.
490
491       Trace chunk archive name example:
492
493           archives/20171119T152407-0500-20171119T151422-0500-3
494

TRACING DOMAIN

496       A tracing domain identifies a type of LTTng tracer.
497
498       A tracing domain has its own properties and features.
499
500       There are currently five available tracing domains:
501
502       ┌──────────────────┬─────────────────────┬──────────────────┐
503Tracing domain    “Event rule         Option for other 
504       │                  │ matches” trigger    CLI commands     
505       │                  │ condition option    │                  │
506       ├──────────────────┼─────────────────────┼──────────────────┤
507       │                  │                     │                  │
508       │Linux kernel      │ --type option       │ --kernel         
509       │                  │ starts with kernel: │                  │
510       ├──────────────────┼─────────────────────┼──────────────────┤
511       │                  │                     │                  │
512       │User space        │ --type option       │ --userspace      
513       │                  │ starts with user:   │                  │
514       ├──────────────────┼─────────────────────┼──────────────────┤
515       │                  │                     │                  │
516java.util.logging --type option       │ --jul            
517       │(JUL)             │ starts with jul:    │                  │
518       ├──────────────────┼─────────────────────┼──────────────────┤
519       │                  │                     │                  │
520       │Apache log4j      │ --type option       │ --log4j          
521       │                  │ starts with log4j:  │                  │
522       ├──────────────────┼─────────────────────┼──────────────────┤
523       │                  │                     │                  │
524       │Python            │ --type option       │ --python         
525       │                  │ starts with python: │                  │
526       └──────────────────┴─────────────────────┴──────────────────┘
527
528       You must specify a tracing domain to target a type of LTTng tracer when
529       using some lttng(1) commands to avoid ambiguity. For example, because
530       the Linux kernel and user space tracing domains support named
531       tracepoints as instrumentation points (see the “INSTRUMENTATION POINT,
532       EVENT RULE, AND EVENT” section above), you need to specify a tracing
533       domain when you create an event rule because both tracing domains could
534       have tracepoints sharing the same name.
535
536       You can create channels (see the “CHANNEL AND RING BUFFER” section
537       below) in the Linux kernel and user space tracing domains. The other
538       tracing domains have a single, default channel.
539

CHANNEL AND RING BUFFER

541       A channel is an object which is responsible for a set of ring buffers.
542
543       Each ring buffer is divided into multiple sub-buffers. When a recording
544       event rule (see the “RECORDING EVENT RULE AND EVENT RECORD” section
545       below) matches an event, LTTng can record it to one or more sub-buffers
546       of one or more channels.
547
548       When you create a channel with the lttng-enable-channel(1) command, you
549       set its final attributes, that is:
550
551       •   Its buffering scheme.
552
553           See the “Buffering scheme” section below.
554
555       •   What to do when there’s no space left for a new event record
556           because all sub-buffers are full.
557
558           See the “Event record loss mode” section below.
559
560       •   The size of each ring buffer and how many sub-buffers a ring buffer
561           has.
562
563           See the “Sub-buffer size and count” section below.
564
565       •   The size of each trace file LTTng writes for this channel and the
566           maximum count of trace files.
567
568           See the “Maximum trace file size and count” section below.
569
570       •   The periods of its read, switch, and monitor timers.
571
572           See the “Timers” section below.
573
574       •   For a Linux kernel channel: its output type (mmap(2) or splice(2)).
575
576           See the --output option of the lttng-enable-channel(1) command.
577
578       •   For a user space channel: the value of its blocking timeout.
579
580           See the --blocking-timeout option of the lttng-enable-channel(1)
581           command.
582
583       Note that the lttng-enable-event(1) command can automatically create a
584       default channel with sane defaults when no channel exists for the
585       provided tracing domain.
586
587       A channel is always associated to a tracing domain (see the “TRACING
588       DOMAIN” section below). The java.util.logging (JUL), log4j, and Python
589       tracing domains each have a default channel which you can’t configure.
590
591       A channel owns recording event rules.
592
593       List the channels of a given recording session with the lttng-list(1)
594       and lttng-status(1) commands.
595
596       Disable an enabled channel with the lttng-disable-channel(1) command.
597
598   Buffering scheme
599       A channel has at least one ring buffer per CPU. LTTng always records an
600       event to the ring buffer dedicated to the CPU which emits it.
601
602       The buffering scheme of a user space channel determines what has its
603       own set of per-CPU ring buffers:
604
605       Per-user buffering (--buffers-uid option of the lttng-enable-channel(1)
606       command)
607           Allocate one set of ring buffers (one per CPU) shared by all the
608           instrumented processes of:
609
610           If your Unix user is root
611               Each Unix user.
612
613           Otherwise
614               Your Unix user.
615
616       Per-process buffering (--buffers-pid option of the lttng-enable-
617       channel(1) command)
618           Allocate one set of ring buffers (one per CPU) for each
619           instrumented process of:
620
621           If your Unix user is root
622               All Unix users.
623
624           Otherwise
625               Your Unix user.
626
627       The per-process buffering scheme tends to consume more memory than the
628       per-user option because systems generally have more instrumented
629       processes than Unix users running instrumented processes. However, the
630       per-process buffering scheme ensures that one process having a high
631       event throughput won’t fill all the shared sub-buffers of the same Unix
632       user, only its own.
633
634       The buffering scheme of a Linux kernel channel is always to allocate a
635       single set of ring buffers for the whole system. This scheme is similar
636       to the per-user option, but with a single, global user “running” the
637       kernel.
638
639   Event record loss mode
640       When LTTng emits an event, LTTng can record it to a specific, available
641       sub-buffer within the ring buffers of specific channels. When there’s
642       no space left in a sub-buffer, the tracer marks it as consumable and
643       another, available sub-buffer starts receiving the following event
644       records. An LTTng consumer daemon eventually consumes the marked
645       sub-buffer, which returns to the available state.
646
647       In an ideal world, sub-buffers are consumed faster than they are
648       filled. In the real world, however, all sub-buffers can be full at some
649       point, leaving no space to record the following events.
650
651       By default, LTTng-modules and LTTng-UST are non-blocking tracers: when
652       there’s no available sub-buffer to record an event, it’s acceptable to
653       lose event records when the alternative would be to cause substantial
654       delays in the execution of the instrumented application. LTTng
655       privileges performance over integrity; it aims at perturbing the
656       instrumented application as little as possible in order to make the
657       detection of subtle race conditions and rare interrupt cascades
658       possible.
659
660       Since LTTng 2.10, the LTTng user space tracer, LTTng-UST, supports a
661       blocking mode. See the --blocking-timeout of the lttng-enable-
662       channel(1) command to learn how to use the blocking mode.
663
664       When it comes to losing event records because there’s no available
665       sub-buffer, or because the blocking timeout of the channel is reached,
666       the event record loss mode of the channel determines what to do. The
667       available event record loss modes are:
668
669       Discard mode
670           Drop the newest event records until a sub-buffer becomes available.
671
672           This is the only available mode when you specify a blocking
673           timeout.
674
675           With this mode, LTTng increments a count of lost event records when
676           an event record is lost and saves this count to the trace. A trace
677           reader can use the saved discarded event record count of the trace
678           to decide whether or not to perform some analysis even if trace
679           data is known to be missing.
680
681       Overwrite mode
682           Clear the sub-buffer containing the oldest event records and start
683           writing the newest event records there.
684
685           This mode is sometimes called flight recorder mode because it’s
686           similar to a flight recorder
687           <https://en.wikipedia.org/wiki/Flight_recorder>: always keep a
688           fixed amount of the latest data. It’s also similar to the roll mode
689           of an oscilloscope.
690
691           Since LTTng 2.8, with this mode, LTTng writes to a given sub-buffer
692           its sequence number within its data stream. With a local, network
693           streaming, or live recording session (see the “Recording session
694           modes” section above), a trace reader can use such sequence numbers
695           to report lost packets. A trace reader can use the saved discarded
696           sub-buffer (packet) count of the trace to decide whether or not to
697           perform some analysis even if trace data is known to be missing.
698
699           With this mode, LTTng doesn’t write to the trace the exact number
700           of lost event records in the lost sub-buffers.
701
702       Which mechanism you should choose depends on your context: prioritize
703       the newest or the oldest event records in the ring buffer?
704
705       Beware that, in overwrite mode, the tracer abandons a whole sub-buffer
706       as soon as a there’s no space left for a new event record, whereas in
707       discard mode, the tracer only discards the event record that doesn’t
708       fit.
709
710       Set the event record loss mode of a channel with the --discard and
711       --overwrite options of the lttng-enable-channel(1) command.
712
713       There are a few ways to decrease your probability of losing event
714       records. The “Sub-buffer size and count” section below shows how to
715       fine-tune the sub-buffer size and count of a channel to virtually stop
716       losing event records, though at the cost of greater memory usage.
717
718   Sub-buffer size and count
719       A channel has one or more ring buffer for each CPU of the target
720       system.
721
722       See the “Buffering scheme” section above to learn how many ring buffers
723       of a given channel are dedicated to each CPU depending on its buffering
724       scheme.
725
726       Set the size of each sub-buffer the ring buffers of a channel contain
727       with the --subbuf-size option of the lttng-enable-channel(1) command.
728
729       Set the number of sub-buffers each ring buffer of a channel contains
730       with the --num-subbuf option of the lttng-enable-channel(1) command.
731
732       Note that LTTng switching the current sub-buffer of a ring buffer
733       (marking a full one as consumable and switching to an available one for
734       LTTng to record the next events) introduces noticeable CPU overhead.
735       Knowing this, the following list presents a few practical situations
736       along with how to configure the sub-buffer size and count for them:
737
738       High event throughput
739           In general, prefer large sub-buffers to lower the risk of losing
740           event records.
741
742           Having larger sub-buffers also ensures a lower sub-buffer switching
743           frequency (see the “Timers” section below).
744
745           The sub-buffer count is only meaningful if you create the channel
746           in overwrite mode (see the “Event record loss mode” section above):
747           in this case, if LTTng overwrites a sub-buffer, then the other
748           sub-buffers are left unaltered.
749
750       Low event throughput
751           In general, prefer smaller sub-buffers since the risk of losing
752           event records is low.
753
754           Because LTTng emits events less frequently, the sub-buffer
755           switching frequency should remain low and therefore the overhead of
756           the tracer shouldn’t be a problem.
757
758       Low memory system
759           If your target system has a low memory limit, prefer fewer first,
760           then smaller sub-buffers.
761
762           Even if the system is limited in memory, you want to keep the
763           sub-buffers as large as possible to avoid a high sub-buffer
764           switching frequency.
765
766       Note that LTTng uses CTF <https://diamon.org/ctf/> as its trace format,
767       which means event record data is very compact. For example, the average
768       LTTng kernel event record weights about 32 bytes. Therefore, a
769       sub-buffer size of 1 MiB is considered large.
770
771       The previous scenarios highlight the major trade-off between a few
772       large sub-buffers and more, smaller sub-buffers: sub-buffer switching
773       frequency vs. how many event records are lost in overwrite mode.
774       Assuming a constant event throughput and using the overwrite mode, the
775       two following configurations have the same ring buffer total size:
776
777       Two sub-buffers of 4 MiB each
778           Expect a very low sub-buffer switching frequency, but if LTTng ever
779           needs to overwrite a sub-buffer, half of the event records so far
780           (4 MiB) are definitely lost.
781
782       Eight sub-buffers of 1 MiB each
783           Expect four times the tracer overhead of the configuration above,
784           but if LTTng needs to overwrite a sub-buffer, only the eighth of
785           event records so far (1 MiB) are definitely lost.
786
787       In discard mode, the sub-buffer count parameter is pointless: use two
788       sub-buffers and set their size according to your requirements.
789
790   Maximum trace file size and count
791       By default, trace files can grow as large as needed.
792
793       Set the maximum size of each trace file that LTTng writes of a given
794       channel with the --tracefile-size option of the lttng-enable-channel(1)
795       command.
796
797       When the size of a trace file reaches the fixed maximum size of the
798       channel, LTTng creates another file to contain the next event records.
799       LTTng appends a file count to each trace file name in this case.
800
801       If you set the trace file size attribute when you create a channel, the
802       maximum number of trace files that LTTng creates is unlimited by
803       default. To limit them, use the --tracefile-count option of lttng-
804       enable-channel(1). When the number of trace files reaches the fixed
805       maximum count of the channel, LTTng overwrites the oldest trace file.
806       This mechanism is called trace file rotation.
807
808       Important
809           Even if you don’t limit the trace file count, always assume that
810           LTTng manages all the trace files of the recording session.
811
812           In other words, there’s no safe way to know if LTTng still holds a
813           given trace file open with the trace file rotation feature.
814
815           The only way to obtain an unmanaged, self-contained LTTng trace
816           before you destroy the recording session is with the recording
817           session rotation feature (see the “Recording session rotation”
818           section above), which is available since LTTng 2.11.
819
820   Timers
821       Each channel can have up to three optional timers:
822
823       Switch timer
824           When this timer expires, a sub-buffer switch happens: for each ring
825           buffer of the channel, LTTng marks the current sub-buffer as
826           consumable and switches to an available one to record the next
827           events.
828
829           A switch timer is useful to ensure that LTTng consumes and commits
830           trace data to trace files or to a distant relay daemon (lttng-
831           relayd(8)) periodically in case of a low event throughput.
832
833           Such a timer is also convenient when you use large sub-buffers (see
834           the “Sub-buffer size and count” section above) to cope with a
835           sporadic high event throughput, even if the throughput is otherwise
836           low.
837
838           Set the period of the switch timer of a channel, or disable the
839           timer altogether, with the --switch-timer option of the lttng-
840           enable-channel(1) command.
841
842       Read timer
843           When this timer expires, LTTng checks for full, consumable
844           sub-buffers.
845
846           By default, the LTTng tracers use an asynchronous message mechanism
847           to signal a full sub-buffer so that a consumer daemon can consume
848           it.
849
850           When such messages must be avoided, for example in real-time
851           applications, use this timer instead.
852
853           Set the period of the read timer of a channel, or disable the timer
854           altogether, with the --read-timer option of the lttng-enable-
855           channel(1) command.
856
857       Monitor timer
858           When this timer expires, the consumer daemon samples some channel
859           statistics to evaluate the following trigger conditions:
860
861            1. The consumed buffer size of a given recording session becomes
862               greater than some value.
863
864            2. The buffer usage of a given channel becomes greater than some
865               value.
866
867            3. The buffer usage of a given channel becomes less than some
868               value.
869
870           If you disable the monitor timer of a channel C:
871
872           •   The consumed buffer size value of the recording session of C
873               could be wrong for trigger condition type 1: the consumed
874               buffer size of C won’t be part of the grand total.
875
876           •   The buffer usage trigger conditions (types 2 and 3) for C will
877               never be satisfied.
878
879           See the “TRIGGER” section above to learn more about triggers.
880
881           Set the period of the monitor timer of a channel, or disable the
882           timer altogether, with the --monitor-timer option of the lttng-
883           enable-channel(1) command.
884

RECORDING EVENT RULE AND EVENT RECORD

886       A recording event rule is a specific type of event rule (see the
887       “INSTRUMENTATION POINT, EVENT RULE, AND EVENT” section above) of which
888       the action is to serialize and record the matched event as an event
889       record.
890
891       Set the explicit conditions of a recording event rule when you create
892       it with the lttng-enable-event(1) command. A recording event rule also
893       has the following implicit conditions:
894
895       •   The recording event rule itself is enabled.
896
897           A recording event rule is enabled on creation.
898
899       •   The channel to which the recording event rule is attached is
900           enabled.
901
902           A channel is enabled on creation.
903
904           See the “CHANNEL AND RING BUFFER” section above.
905
906       •   The recording session of the recording event rule is active
907           (started).
908
909           A recording session is inactive (stopped) on creation.
910
911           See the “RECORDING SESSION” section above.
912
913       •   The process for which LTTng creates an event to match is allowed to
914           record events.
915
916           All processes are allowed to record events on recording session
917           creation.
918
919           Use the lttng-track(1) and lttng-untrack(1) commands to select
920           which processes are allowed to record events based on specific
921           process attributes.
922
923       You always attach a recording event rule to a channel, which belongs to
924       a recording session, when you create it.
925
926       When a recording event rule ER matches an event E, LTTng attempts to
927       serialize and record E to one of the available sub-buffers of the
928       channel to which E is attached.
929
930       When multiple matching recording event rules are attached to the same
931       channel, LTTng attempts to serialize and record the matched event once.
932       In the following example, the second recording event rule is redundant
933       when both are enabled:
934
935           $ lttng enable-event --userspace hello:world
936           $ lttng enable-event --userspace hello:world --loglevel=INFO
937
938       List the recording event rules of a specific recording session and/or
939       channel with the lttng-list(1) and lttng-status(1) commands.
940
941       Disable a recording event rule with the lttng-disable-event(1) command.
942
943       As of LTTng 2.13.4, you cannot remove a recording event rule: it exists
944       as long as its recording session exists.
945

RESOURCES

947       •   LTTng project website <https://lttng.org>
948
949       •   LTTng documentation <https://lttng.org/docs>
950
951       •   LTTng bug tracker <https://bugs.lttng.org>
952
953       •   Git repositories <https://git.lttng.org>
954
955       •   GitHub organization <https://github.com/lttng>
956
957       •   Continuous integration <https://ci.lttng.org/>
958
959       •   Mailing list <https://lists.lttng.org/> for support and
960           development: lttng-dev@lists.lttng.org
961
962       •   IRC channel <irc://irc.oftc.net/lttng>: #lttng on irc.oftc.net
963
965       This program is part of the LTTng-tools project.
966
967       LTTng-tools is distributed under the GNU General Public License
968       version 2 <http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html>.
969       See the LICENSE <https://github.com/lttng/lttng-
970       tools/blob/master/LICENSE> file for details.
971

THANKS

973       Special thanks to Michel Dagenais and the DORSAL laboratory
974       <http://www.dorsal.polymtl.ca/> at École Polytechnique de Montréal for
975       the LTTng journey.
976
977       Also thanks to the Ericsson teams working on tracing which helped us
978       greatly with detailed bug reports and unusual test cases.
979

SEE ALSO

981       lttng(1), lttng-relayd(8), lttng-sessiond(8)
982
983
984
985LTTng 2.13.4                     14 June 2021                LTTNG-CONCEPTS(7)
Impressum