1MK-LOG-PLAYER(1) User Contributed Perl Documentation MK-LOG-PLAYER(1)
2
6 mk-log-player - Replay MySQL query logs.
7
9 Usage: mk-log-player [OPTION...] [DSN]
10 mk-log-player splits and plays slow log files.
12 Split slow.log on Thread_id into 16 session files, save in ./sessions:
14 mk-log-player --split Thread_id --session-files 16 --base-dir ./sessions slow.log
16 Play all those sessions on host1, save results in ./results:
18 mk-log-player --play ./sessions --base-dir ./results h=host1
20 Use mk-query-digest to summarize the results:
22 mk-query-digest ./results/*
24
26 The following section is included to inform users about the potential
27 risks, whether known or unknown, of using this tool. The two main
28 categories of risks are those created by the nature of the tool (e.g.
29 read-only tools vs. read-write tools) and those created by bugs.
30 This tool is meant to load a server as much as possible, for stress-
32 testing purposes. It is not designed to be used on production servers.
33 At the time of this release there is a bug which causes mk-log-player
35 to exceed max open files during "--split".
36 The authoritative source for updated information is always the online
38 issue tracking system. Issues that affect this tool will be marked as
39 such. You can see a list of such issues at the following URL:
40 <http://www.maatkit.org/bugs/mk-log-player>.
41 See also "BUGS" for more information on filing bugs and getting help.
43
45 mk-log-player does two things: it splits MySQL query logs into session
46 files and it plays (executes) queries in session files on a MySQL
47 server. Only session files can be played; slow logs cannot be played
48 directly without being split.
49 A session is a group of queries from the slow log that all share a
51 common attribute, usually Thread_id. The common attribute is specified
52 with "--split". Multiple sessions are saved into a single session
53 file. See "--session-files", "--max-sessions", "--base-file-name" and
54 "--base-dir". These session files are played with "--play".
55 mk-log-player will "--play" session files in parallel using N number of
57 "--threads". (They're not technically threads, but we call them that
58 anyway.) Each thread will play all the sessions in its given session
59 files. The sessions are played as fast as possible--there are no
60 delays--because the goal is to stress-test and load-test the server.
61 So be careful using this script on a production server!
62 Each "--play" thread writes its results to a separate file. These
64 result files are in slow log format so they can be aggregated and
65 summarized with mk-query-digest. See "OUTPUT".
66
68 Both "--split" and "--play" have two outputs: status messages printed
69 to STDOUT to let you know what the script is doing, and session or
70 result files written to separate files saved in "--base-dir". You can
71 suppress all output to STDOUT for each with "--quiet", or increase
72 output with "--verbose".
73 The session files written by "--split" are simple text files containing
75 queries grouped into sessions. For example:
76 -- START SESSION 10
78 use foo
80 SELECT col FROM foo_tbl
82 The format of these session files is important: each query must be a
84 single line separated by a single blank line. And the "-- START
85 SESSION" comment tells mk-log-player where individual sessions begin
86 and end so that "--play" can correctly fake Thread_id in its result
87 files.
88 The result files written by "--play" are in slow log format with a
90 minimal header: the only attributes printed are Thread_id, Query_time
91 and Schema.
92
94 Specify at least one of "--play", "--split" or "--split-random".
95 "--play" and "--split" are mutually exclusive.
97 This tool accepts additional command-line arguments. Refer to the
99 "SYNOPSIS" and usage information for details.
100 --ask-pass
102 group: Play
103 Prompt for a password when connecting to MySQL.
105 --base-dir
107 type: string; default: ./
108 Base directory for "--split" session files and "--play" result
110 file.
111 --base-file-name
113 type: string; default: session
114 Base file name for "--split" session files and "--play" result
116 file.
117 Each "--split" session file will be saved as
119 <base-file-name>-N.txt, where N is a four digit, zero-padded
120 session ID. For example: session-0003.txt.
121 Each "--play" result file will be saved as
123 <base-file-name>-results-PID.txt, where PID is the process ID of
124 the executing thread.
125 All files are saved in "--base-dir".
127 --charset
129 short form: -A; type: string; group: Play
130 Default character set. If the value is utf8, sets Perl's binmode
132 on STDOUT to utf8, passes the mysql_enable_utf8 option to
133 DBD::mysql, and runs SET NAMES UTF8 after connecting to MySQL. Any
134 other value sets binmode on STDOUT without the utf8 layer, and runs
135 SET NAMES after connecting to MySQL.
136 --config
138 type: Array
139 Read this comma-separated list of config files; if specified, this
141 must be the first option on the command line.
142 --defaults-file
144 short form: -F; type: string
145 Only read mysql options from the given file.
147 --dry-run
149 Print which processes play which session files then exit.
150 --filter
152 type: string; group: Split
153 Discard "--split" events for which this Perl code doesn't return
155 true.
156 This option only works with "--split".
158 This option allows you to inject Perl code into the tool to affect
160 how the tool runs. Usually your code should examine $event to
161 decided whether or not to allow the event. $event is a hashref of
162 attributes and values of the event being filtered. Or, your code
163 could add new attribute-value pairs to $event for use by other
164 options that accept event attributes as their value. You can find
165 an explanation of the structure of $event at
166 <http://code.google.com/p/maatkit/wiki/EventAttributes>.
167 There are two ways to supply your code: on the command line or in a
169 file. If you supply your code on the command line, it is injected
170 into the following subroutine where $filter is your code:
171 sub {
173 MKDEBUG && _d('callback: filter');
174 my( $event ) = shift;
175 ( $filter ) && return $event;
176 }
177 Therefore you must ensure two things: first, that you correctly
179 escape any special characters that need to be escaped on the
180 command line for your shell, and two, that your code is
181 syntactically valid when injected into the subroutine above.
182 Here's an example filter supplied on the command line that discards
184 events that are not SELECT statements:
185 --filter '$event->{arg} =~ m/^select/i'
187 The second way to supply your code is in a file. If your code is
189 too complex to be expressed on the command line that results in
190 valid syntax in the subroutine above, then you need to put the code
191 in a file and give the file name as the value to "--filter". The
192 file should not contain a shebang ("#!/usr/bin/perl") line. The
193 entire contents of the file is injected into the following
194 subroutine:
195 sub {
197 MKDEBUG && _d('callback: filter');
198 my( $event ) = shift;
199 $filter && return $event;
200 }
201 That subroutine is almost identical to the one above except your
203 code is not wrapped in parentheses. This allows you to write
204 multi-line code like:
205 my $event_ok;
207 if (...) {
208 $event_ok = 1;
209 }
210 else {
211 $event_ok = 0;
212 }
213 $event_ok
214 Notice that the last line is not syntactically valid by itself, but
216 it becomes syntactically valid when injected into the subroutine
217 because it becomes:
218 $event_ok && return $event;
220 If your code doesn't compile, the tool will die with an error.
222 Even if your code compiles, it may crash to tool during runtime if,
223 for example, it tries a pattern match an undefined value. No
224 safeguards of any kind of provided so code carefully!
225 --help
227 Show help and exit.
228 --host
230 short form: -h; type: string; group: Play
231 Connect to host.
233 --iterations
235 type: int; default: 1; group: Play
236 How many times each thread should play all its session files.
238 --max-sessions
240 type: int; default: 5000000; group: Split
241 Maximum number of sessions to "--split".
243 By default, "mk-log-player" tries to split every session from the
245 log file. For huge logs, however, this can result in millions of
246 sessions. This option causes only the first N number of sessions
247 to be saved. All sessions after this number are ignored, but
248 sessions split before this number will continue to have their
249 queries split even if those queries appear near the end of the log
250 and after this number has been reached.
251 --only-select
253 group: Play
254 Play only SELECT and USE queries; ignore all others.
256 --password
258 short form: -p; type: string; group: Play
259 Password to use when connecting.
261 --pid
263 type: string
264 Create the given PID file. The file contains the process ID of the
266 script. The PID file is removed when the script exits. Before
267 starting, the script checks if the PID file already exists. If it
268 does not, then the script creates and writes its own PID to it. If
269 it does, then the script checks the following: if the file contains
270 a PID and a process is running with that PID, then the script dies;
271 or, if there is no process running with that PID, then the script
272 overwrites the file with its own PID and starts; else, if the file
273 contains no PID, then the script dies.
274 --play
276 type: string; group: Play
277 Play (execute) session files created by "--split".
279 The argument to play must be a comma-separated list of session
281 files created by "--split" or a directory. If the argument is a
282 directory, ALL files in that directory will be played.
283 --port
285 short form: -P; type: int; group: Play
286 Port number to use for connection.
288 --print
290 group: Play
291 Print queries instead of playing them; requires "--play".
293 You must also specify "--play" with "--print". Although the
295 queries will not be executed, "--play" is required to specify which
296 session files to read.
297 --quiet
299 short form: -q
300 Do not print anything; disables "--verbose".
302 --[no]results
304 default: yes
305 Print "--play" results to files in "--base-dir".
307 --session-files
309 type: int; default: 8; group: Split
310 Number of session files to create with "--split".
312 The number of session files should either be equal to the number of
314 "--threads" you intend to "--play" or be an even multiple of
315 "--threads". This number is important for maximum performance
316 because it:
317 * allows each thread to have roughly the same amount of sessions
318 to play
319 * avoids having to open/close many session files
320 * avoids disk IO overhead by doing large sequential reads
321 You may want to increase this number beyond "--threads" if each
323 session file becomes too large. For example, splitting a 20G log
324 into 8 sessions files may yield roughly eight 2G session files.
325 See also "--max-sessions".
327 --set-vars
329 type: string; group: Play; default: wait_timeout=10000
330 Set these MySQL variables. Immediately after connecting to MySQL,
332 this string will be appended to SET and executed.
333 --socket
335 short form: -S; type: string; group: Play
336 Socket file to use for connection.
338 --split
340 type: string; group: Split
341 Split log by given attribute to create session files.
343 Valid attributes are any which appear in the log: Thread_id,
345 Schema, etc.
346 --split-random
348 group: Split
349 Split log without an attribute, write queries round-robin to
351 session files.
352 This option, if specified, overrides "--split" and causes the log
354 to be split query-by-query, writing each query to the next session
355 file in round-robin style. If you don't care about "sessions" and
356 just want to split a lot into N many session files and the relation
357 or order of the queries does not matter, then use this option.
358 --threads
360 type: int; default: 2; group: Play
361 Number of threads used to play sessions concurrently.
363 Specifies the number of parallel processes to run. The default is
365 2. On GNU/Linux machines, the default is the number of times
366 'processor' appears in /proc/cpuinfo. On Windows, the default is
367 read from the environment. In any case, the default is at least 2,
368 even when there's only a single processor.
369 See also "--session-files".
371 --type
373 type: string; group: Split
374 The type of log to "--split" (default slowlog). The permitted
376 types are
377 binlog
379 Split the output of running "mysqlbinlog" against a binary log
380 file. Currently, splitting binary logs does not always work
381 well depending on what the binary logs contain. Be sure to
382 check the session files after splitting to ensure proper
383 "OUTPUT".
384 If the binary log contains row-based replication data, you need
386 to run "mysqlbinlog" with options "--base64-output=decode-rows
387 --verbose", else invalid statements will be written to the
388 session files.
389 genlog
391 Split a general log file.
392 slowlog
394 Split a log file in any variation of MySQL slow-log format.
395 --user
397 short form: -u; type: string; group: Play
398 User for login if not current user.
400 --verbose
402 short form: -v; cumulative: yes; default: 0
403 Increase verbosity; can be specified multiple times.
405 This option is disabled by "--quiet".
407 --version
409 Show version and exit.
410 --wait-between-sessions
412 type: array; default: 0; group: Play
413 Not implemented yet.
415 The wait time is given in seconds with microsecond precision and
417 can be either a single value or a range. A single value causes an
418 exact wait; example: 0.010 = wait 10 milliseconds. A range causes
419 a random wait between the given value times; example: 0.001,1 =
420 random wait from 1 millisecond to 1 second.
421 --[no]warnings
423 default: no; group: Play
424 Print warnings about SQL errors such as invalid queries to STDERR.
426
428 These DSN options are used to create a DSN. Each option is given like
429 "option=value". The options are case-sensitive, so P and p are not the
430 same option. There cannot be whitespace before or after the "=" and if
431 the value contains whitespace it must be quoted. DSN options are
432 comma-separated. See the maatkit manpage for full details.
433 • A
435 dsn: charset; copy: yes
437 Default character set.
439 • D
441 dsn: database; copy: yes
443 Default database.
445 • F
447 dsn: mysql_read_default_file; copy: yes
449 Only read default options from the given file
451 • h
453 dsn: host; copy: yes
455 Connect to host.
457 • p
459 dsn: password; copy: yes
461 Password to use when connecting.
463 • P
465 dsn: port; copy: yes
467 Port number to use for connection.
469 • S
471 dsn: mysql_socket; copy: yes
473 Socket file to use for connection.
475 • u
477 dsn: user; copy: yes
479 User for login if not current user.
481
483 You can download Maatkit from Google Code at
484 <http://code.google.com/p/maatkit/>, or you can get any of the tools
485 easily with a command like the following:
486 wget http://www.maatkit.org/get/toolname
488 or
489 wget http://www.maatkit.org/trunk/toolname
490 Where "toolname" can be replaced with the name (or fragment of a name)
492 of any of the Maatkit tools. Once downloaded, they're ready to run; no
493 installation is needed. The first URL gets the latest released version
494 of the tool, and the second gets the latest trunk code from Subversion.
495
497 The environment variable "MKDEBUG" enables verbose debugging output in
498 all of the Maatkit tools:
499 MKDEBUG=1 mk-....
501
503 You need Perl and some core packages that ought to be installed in any
504 reasonably new version of Perl.
505
507 For a list of known bugs see
508 <http://www.maatkit.org/bugs/mk-log-player>.
509 Please use Google Code Issues and Groups to report bugs or request
511 support: <http://code.google.com/p/maatkit/>. You can also join
512 #maatkit on Freenode to discuss Maatkit.
513 Please include the complete command-line used to reproduce the problem
515 you are seeing, the version of all MySQL servers involved, the complete
516 output of the tool when run with "--version", and if possible,
517 debugging output produced by running with the "MKDEBUG=1" environment
518 variable.
519
521 This program is copyright 2008-2011 Percona Inc. Feedback and
522 improvements are welcome.
523 THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
525 WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
526 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
527 This program is free software; you can redistribute it and/or modify it
529 under the terms of the GNU General Public License as published by the
530 Free Software Foundation, version 2; OR the Perl Artistic License. On
531 UNIX and similar systems, you can issue `man perlgpl' or `man
532 perlartistic' to read these licenses.
533 You should have received a copy of the GNU General Public License along
535 with this program; if not, write to the Free Software Foundation, Inc.,
536 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
537
539 Daniel Nichter
540
542 This tool is part of Maatkit, a toolkit for power users of MySQL.
543 Maatkit was created by Baron Schwartz; Baron and Daniel Nichter are the
544 primary code contributors. Both are employed by Percona. Financial
545 support for Maatkit development is primarily provided by Percona and
546 its clients.
547
549 This manual page documents Ver 1.0.9 Distrib 7540 $Revision: 7531 $.
550perl v5.34.0 2021-07-22 MK-LOG-PLAYER(1)