1PDBTOOL(1)                  The pdbtool manual page                 PDBTOOL(1)
2
3
4

NAME

6       pdbtool - An application to test and convert syslog-ng pattern database
7       rules
8

SYNOPSIS

10       pdbtool [command] [options]
11

DESCRIPTION

13       This manual page is only an abstract, for the complete documentation of
14       syslog-ng and pdbtool, see The syslog-ng Administrator Guide[1].
15
16       The syslog-ng application can match the contents of the log messages to
17       a database of predefined message patterns (also called patterndb). By
18       comparing the messages to the known patterns, syslog-ng is able to
19       identify the exact type of the messages, tag the messages, and sort
20       them into message classes. The message classes can be used to classify
21       the type of the event described in the log message. The functionality
22       of the pattern database is similar to that of the logcheck project, but
23       the syslog-ng approach is faster, scales better, and is much easier to
24       maintain compared to the regular expressions of logcheck.
25
26       The pdbtool application is a utility that can be used to:
27
28       ·   test messages, or specific rules
29
30       ·   convert an older pattern database to the latest database format
31
32       ·   merge pattern databases into a single file
33
34       ·   automatically create pattern databases from a large amount of log
35           messages
36
37       ·   dump the RADIX tree built from the pattern database (or a part of
38           it) to explore how the pattern matching works.
39

THE DICTIONARY COMMAND

41       dictionary [options]
42
43       Lists every name-value pair that can be set by the rules of the pattern
44       database.
45
46       --dump-tags or -T
47           List the tags instead of the names of the name-value pairs.
48
49       --pdb <path-to-file> or -p <path-to-file>
50           Name of the pattern database file to use.
51
52       --program <programname> or -P <programname>
53           List only the name-value pairs that can be set for the messages of
54           the specified $PROGRAM application.
55

THE DUMP COMMAND

57       dump [options]
58
59       Display the RADIX tree built from the patterns. This shows how are the
60       patterns represented in syslog-ng and it might also help to track down
61       pattern-matching problems. The dump utility can dump the tree used for
62       matching the PROGRAM or the MESSAGE parts.
63
64       --debug or -d
65           Enable debug/diagnostic messages on stderr.
66
67       --pdb or -p
68           Name of the pattern database file to use.
69
70       --program or -P
71           Displays the RADIX tree built from the patterns belonging to the
72           ${PROGRAM} application.
73
74       --program-tree or -T
75           Display the ${PROGRAM} tree.
76
77       --verbose or -v
78           Enable verbose messages on stderr.
79
80       Example and sample output:
81
82           pdbtool dump -p patterndb.xml  -P 'sshd'
83
84           'p'
85              'assword for'
86                @QSTRING:@
87                  'from'
88                   @QSTRING:@
89                     'port '
90                       @NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b'
91                         ' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b'
92                            '2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b'
93               'ublickey for'
94                 @QSTRING:@
95                   'from'
96                    @QSTRING:@
97                      'port '
98                        @NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b'
99                          ' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b'
100                             '2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b'
101
102

THE MATCH COMMAND

104       match [options]
105
106       Use the match command to test the rules in a pattern database. The
107       command tries to match the specified message against the patterns of
108       the database, evaluates the parsers of the pattern, and also displays
109       which part of the message was parsed successfully. The command returns
110       with a 0 (success) or 1 (no match) return code and displays the
111       following information:
112
113       ·   the class assigned to the message (that is, system, violation, and
114           so on),
115
116       ·   the ID of the rule that matched the message, and
117
118       ·   the values of the parsers (if there were parsers in the matching
119           pattern).
120
121       The match command has the following options:
122
123       --color-out  or -c
124           Color the terminal output to highlight the part of the message that
125           was successfully parsed.
126
127       --debug or -d
128           Enable debug/diagnostic messages on stderr.
129
130       --debug-csv or -C
131           Print the debugging information returned by the --debug-pattern
132           option as comma-separated values.
133
134       --debug-pattern or -D
135           Print debugging information about the pattern matching. See also
136           the --debug-csv option.
137
138       --file=<filename-with-path> or -f
139           Process the messages of the specified log file with the pattern
140           database. This option allows to classify messages offline, and to
141           apply the pattern database to already existing logfiles. To read
142           the messages from the standard input (stdin), specify a hyphen (-)
143           character instead of a filename.
144
145       --filter=<filter-expression> or -F
146           Print only messages matching the specified syslog-ng filter
147           expression.
148
149       --message or -M
150           The text of the log message to match (only the ${MESSAGE} part
151           without the syslog headers).
152
153       --pdb or -p
154           Name of the pattern database file to use.
155
156       --program or -P
157           Name of the program to use, as contained in the ${PROGRAM} part of
158           the syslog message.
159
160       --template=<template-expression> or -T
161           A syslog-ng template expression that is used to format the output
162           messages.
163
164       --verbose or -v
165           Enable verbose messages on stderr.
166
167       Example: The following command checks if the patterndb.xml file
168       recognizes the Accepted publickey for myuser from 127.0.0.1 port 59357
169       ssh2 message:
170
171           pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for myuser from 127.0.0.1 port 59357 ssh2"
172
173       The following example applies the sshd.pdb pattern database file to the
174       log messages stored in the /var/log/messages file, and displays only
175       the messages that received a useracct tag.
176
177           pdbtool match -p sshd.pdb \
178             –file /var/log/messages \
179             –filter ‘tags(“usracct”);’
180

THE MERGE COMMAND

182       merge [options]
183
184       Use the merge command to combine separate pattern database files into a
185       single file (pattern databases are usually stored in separate files per
186       applications to simplify maintenance). If a file uses an older database
187       format, it is automatically updated to the latest format (V3). See the
188       The syslog-ng Administrator Guide[1] for details on the different
189       pattern database versions.
190
191       --debug or -d
192           Enable debug/diagnostic messages on stderr.
193
194       --directory or -D
195           The directory that contains the pattern database XML files to be
196           merged.
197
198       --glob or -G
199           Specify filenames to be merged using a glob pattern, for example,
200           using wildcards. For details on glob patterns, see man glob. This
201           pattern is applied only to the filenames, and not on directory
202           names.
203
204       --pdb or -p
205           Name of the output pattern database file.
206
207       --recursive or -r
208           Merge files from subdirectories as well.
209
210       --verbose or -v
211           Enable verbose messages on stderr.
212
213       Example:
214
215           pdbtool merge --recursive --directory /home/me/mypatterns/  --pdb /var/lib/syslog-ng/patterndb.xml
216
217       Currently it is not possible to convert a file without merging, so if
218       you only want to convert an older pattern database file to the latest
219       format, you have to copy it into an empty directory.
220

THE PATTERNIZE COMMAND

222       patternize [options]
223
224       Automatically create a pattern database from a log file containing a
225       large number of log messages. The resulting pattern database is printed
226       to the standard output (stdout). The pdbtool patternize command uses a
227       data clustering technique to find similar log messages and replacing
228       the differing parts with @ESTRING:: @ parsers. For details on pattern
229       databases and message parsers, see the The syslog-ng Administrator
230       Guide[1]. The patternize command is available only in version 3.2 and
231       later.
232
233       --debug or -d
234           Enable debug/diagnostic messages on stderr.
235
236       --file=<path> or -f
237           The logfile containing the log messages to create patterns from. To
238           receive the log messages from the standard input (stdin), use -.
239
240       --iterate-outliers or -o
241           Recursively iterate on the log lines to cover as many log messages
242           with patterns as possible.
243
244       --named-parsers or -n
245           The number of example log messages to include in the pattern
246           database for every pattern. Default value: 1
247
248       --no-parse or -p
249           Do not parse the input file, treat every line as the message part
250           of a log message.
251
252       --samples=<number-of-samples>
253           Include a generated name in the parsers, for example,
254           .dict.string1, .dict.string2, and so on.
255
256       --support=<number> or -S
257           A pattern is added to the output pattern database if at least the
258           specified percentage of log messages from the input logfile match
259           the pattern. For example, if the input logfile contains 1000 log
260           messages and the --support=3.0 option is used, a pattern is created
261           only if the pattern matches at least 3 percent of the log messages
262           (that is, 30 log messages). If patternize does not create enough
263           patterns, try to decrease the support value.
264
265           Default value: 4.0
266
267       --verbose or -v
268           Enable verbose messages on stderr.
269
270       Example:
271
272           pdbtool patternize --support=2.5 --file=/var/log/messages
273

THE TEST COMMAND

275       test [options]
276
277       Use the test command to validate a pattern database XML file. Note that
278       you must have the xmllint application installed. The test command is
279       available only in version 3.2 and later.
280
281       --color-out or -c
282           Enable coloring in terminal output.
283
284       --debug or -d
285           Enable debug/diagnostic messages on stderr.
286
287       --debug or -D
288           Print debugging information on non-matching patterns.
289
290       --rule-id or -r
291           Test only the patterndb rule (specified by its rule id) against its
292           example.
293
294       --validate
295           Validate a pattern database XML file.
296
297       --verbose or -v
298           Enable verbose messages on stderr.
299
300       Example:
301
302           pdbtool test --validate /home/me/mypatterndb.pdb
303

FILES

305       /usr/local/
306
307       /usr/local/etc/syslog-ng.conf
308

SEE ALSO

310       The syslog-ng Administrator Guide[1]
311
312       syslog-ng.conf(5)
313
314       syslog-ng(8)
315
316           Note
317           For the detailed documentation of see The 3.17 Administrator
318           Guide[2]
319
320           If you experience any problems or need help with syslog-ng, visit
321           the syslog-ng mailing list[3].
322
323           For news and notifications about of syslog-ng, visit the syslog-ng
324           blogs[4].
325

AUTHOR

327       This manual page was written by the Balabit Documentation Team
328       <documentation@balabit.com>.
329

NOTES

332        1. The syslog-ng Administrator Guide
333           https://www.balabit.com/support/documentation/
334
335        2. The  3.17 Administrator Guide
336           https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html
337
338        3. syslog-ng mailing list
339           https://lists.balabit.hu/mailman/listinfo/syslog-ng
340
341        4. syslog-ng blogs
342           https://syslog-ng.org/blogs/
343
344
345
3463.17                              08/10/2018                        PDBTOOL(1)
Impressum