1MK-LOG-PLAYER(1) User Contributed Perl Documentation MK-LOG-PLAYER(1)
2
6 mk-log-player - Split and play MySQL slow logs.
7
9 Split slow.log on Thread_id into 16 session files, save in ./sessions:
10 mk-log-player --split Thread_id --session-files 16 --base-dir ./sessions slow.log
12 Play all those sessions on host1, save results in ./results:
14 mk-log-player --play ./sessions --base-dir ./results h=host1
16 Use mk-query-digest to summarize the results:
18 mk-query-digest ./results/*
20
22 The following section is included to inform users about the potential
23 risks, whether known or unknown, of using this tool. The two main
24 categories of risks are those created by the nature of the tool (e.g.
25 read-only tools vs. read-write tools) and those created by bugs.
26 "--play" is meant to load a server as much as possible, for stress-
28 testing purposes. It is not designed to be used on production servers.
29 At the time of this release there is a bug which causes mk-log-player
31 to exceed max open files during "--split".
32 The authoritative source for updated information is always the online
34 issue tracking system. Issues that affect this tool will be marked as
35 such. You can see a list of such issues at the following URL:
36 http://www.maatkit.org/bugs/mk-log-player
37 <http://www.maatkit.org/bugs/mk-log-player>.
38 See also "BUGS" for more information on filing bugs and getting help.
40
42 mk-log-player does two things: it splits MySQL slow logs into session
43 files and it plays (executes) queries in session files on a MySQL
44 server. Only session files can be played; slow logs cannot be played
45 directly without being split.
46 A session is a group of queries from the slow log that all share a
48 common attribute, usually Thead_id. The common attribute is specified
49 with "--split". Multiple sessions are saved into a single sesssion
50 file. See "--session-files", "--max-sessions", "--base-file-name" and
51 "--base-dir". These session files are played with "--play".
52 mk-log-player will "--play" session files in parallel using N number of
54 "--threads". (They're not technically threads, but we call them that
55 anyway.) Each thread will play all the sessions in its given session
56 files. The sessions are played as fast as possible--there are no
57 delays--because the goal is to stress-test and load-test the server.
58 So be careful using this script on a production server!
59 Each "--play" thread writes its results to a separate file. These
61 result files are in slow log format so they can be aggregated and
62 summarized with mk-query-digest. See "OUTPUT".
63
65 Both "--split" and "--play" have two outputs: status messages printed
66 to STDOUT to let you know what the script is doing, and session or
67 result files written to separate files saved in "--base-dir". You can
68 suppress all output to STDOUT for each with "--quiet", or increase
69 output with "--verbose".
70 The session files written by "--split" are simple text files containing
72 queries grouped into sessions. For example:
73 -- START SESSION 10
75 use foo
77 SELECT col FROM foo_tbl
79 The format of these session files is important: each query must be a
81 single line separated by a single blank line. And the "-- START
82 SESSION" comment tells mk-log-player where individual sessions begin
83 and end so that "--play" can correctly fake Thread_id in its result
84 files.
85 The result files written by "--play" are in slow log format with a
87 minimal header: the only attributes printed are Thread_id, Query_time
88 and Schema.
89
91 Specify at least one of "--play", "--split" or "--split-random".
92 "--play" and "--split" are mutually exclusive.
94 --ask-pass
96 group: Play
97 Prompt for a password when connecting to MySQL.
99 --base-dir
101 type: string; default: ./
102 Base directory for "--split" session files and "--play" result
104 file.
105 --base-file-name
107 type: string; default: session
108 Base file name for "--split" session files and "--play" result
110 file.
111 Each "--split" session file will be saved as
113 <base-file-name>-N.txt, where N is a four digit, zero-padded
114 session ID. For example: session-0003.txt.
115 Each "--play" result file will be saved as
117 <base-file-name>-results-PID.txt, where PID is the process ID of
118 the executing thread.
119 All files are saved in "--base-dir".
121 --charset
123 group: Play
124 short form: -A; type: string
126 Default character set. If the value is utf8, sets Perl's binmode
128 on STDOUT to utf8, passes the mysql_enable_utf8 option to
129 DBD::mysql, and runs SET NAMES UTF8 after connecting to MySQL. Any
130 other value sets binmode on STDOUT without the utf8 layer, and runs
131 SET NAMES after connecting to MySQL.
132 --config
134 type: Array
135 Read this comma-separated list of config files; if specified, this
137 must be the first option on the command line.
138 --defaults-file
140 short form: -F; type: string
141 Only read mysql options from the given file.
143 --dry-run
145 Print which processes play which session files then exit.
146 --filter
148 type: string; group: Split
149 Discard "--split" events for which this Perl code doesn't return
151 true.
152 This option only works with "--split".
154 This option is a string of Perl code or a file containing Perl code
156 that gets compiled into a subroutine with one argument: $event.
157 This is a hashref. If the given value is a readable file, then mk-
158 log-player reads the entire file and uses its contents as the code.
159 The file should not contain a shebang (#!/usr/bin/perl) line.
160 If the code returns true, the query is split; otherwise it is
162 discarded. The code is the last statement in the subroutine other
163 than "return $event". The subroutine template is:
164 sub { $event = shift; filter && return $event; }
166 Filters given on the command line are wrapped inside parentheses
168 like like "( filter )". For complex, multi-line filters, you must
169 put the code inside a file so it will not be wrapped inside
170 parentheses. Either way, the filter must produce syntactically
171 valid code given the template. For example, an if-else branch
172 given on the command line would not be valid:
173 --filter 'if () { } else { }' # WRONG
175 Since it's given on the command line, the if-else branch would be
177 wrapped inside parentheses which is not syntactically valid. So to
178 accomplish something more complex like this would require putting
179 the code in a file, for example filter.txt:
180 my $event_ok; if (...) { $event_ok=1; } else { $event_ok=0; } $event_ok
182 Then specify "--filter filter.txt" to read the code from
184 filter.txt.
185 If the filter code won't compile, mk-log-player will die with an
187 error. If the filter code does compile, an error may still occur
188 at runtime if the code tries to do something wrong (like pattern
189 match an undefined value). mk-log-player does not provide any
190 safeguards so code carefully!
191 An example filter that discards everything but SELECT statements:
193 --filter '$event->{arg} =~ m/^select/i'
195 This is compiled into a subroutine like the following:
197 sub { $event = shift; ( $event->{arg} =~ m/^select/i ) && return $event; }
199 You can find an explanation of the structure of $event at
201 <http://code.google.com/p/maatkit/wiki/EventAttributes>.
202 --help
204 Show help and exit.
205 --host
207 short form: -h; type: string; group: Play
208 Connect to host.
210 --iterations
212 type: int; default: 1; group: Play
213 How many times each thread should play all its session files.
215 --max-sessions
217 type: int; default: 5000000; group: Split
218 Maximum number of sessions to "--split".
220 By default, "mk-log-player" tries to split every session from the
222 log file. For huge logs, however, this can result in millions of
223 sessions. This option causes only the first N number of sessions
224 to be saved. All sessions after this number are ignored, but
225 sessions split before this number will continue to have their
226 queries split even if those queries appear near the end of the log
227 and after this number has been reached.
228 --only-select
230 group: Play
231 Play only SELECT and USE queries; ignore all others.
233 --password
235 short form: -p; type: string; group: Play
236 Password to use when connecting.
238 --pid
240 type: string
241 Create the given PID file. The file contains the process ID of the
243 script. The PID file is removed when the script exits. Before
244 starting, the script checks if the PID file already exists. If it
245 does not, then the script creates and writes its own PID to it. If
246 it does, then the script checks the following: if the file contains
247 a PID and a process is running with that PID, then the script dies;
248 or, if there is no process running with that PID, then the script
249 overwrites the file with its own PID and starts; else, if the file
250 contains no PID, then the script dies.
251 --play
253 type: string; group: Play
254 Play (execute) session files created by "--split".
256 The argument to play must be a commaxn-separated list of session
258 files created by "--split" or a directory. If the argument is a
259 directory, ALL files in that directory will be played.
260 --port
262 short form: -P; type: int; group: Play
263 Port number to use for connection.
265 --print
267 group: Play
268 Print queries instead of playing them; requires "--play".
270 You must also specify "--play" with "--print". Although the
272 queries will not be executed, "--play" is required to specify which
273 session files to read.
274 --quiet
276 Do not print anything; disables "--verbose".
277 --[no]results
279 default: yes
280 Print "--play" results to files in "--base-dir".
282 --session-files
284 type: int; default: 8; group: Split
285 Number of session files to create with "--split".
287 The number of session files should either be equal to the number of
289 "--threads" you intend to "--play" or be an even multiple of
290 "--threads". This number is important for maximum performance
291 because it:
292 * allows each thread to have roughly the same amount of sessions
293 to play
294 * avoids having to open/close many session files
295 * avoids disk IO overhead by doing large sequential reads
296 You may want to increase this number beyond "--threads" if each
298 session file becomes too large. For example, splitting a 20G log
299 into 8 sessions files may yield roughly eight 2G session files.
300 See also "--max-sessions".
302 --set-vars
304 type: string; group: Play; default: wait_timeout=10000
305 Set these MySQL variables. Immediately after connecting to MySQL,
307 this string will be appended to SET and executed.
308 --socket
310 short form: -S; type: string; group: Play
311 Socket file to use for connection.
313 --split
315 type: string; group: Split
316 Split log by given attribute to create session files.
318 Valid attributes are any which appear in the log: Thread_id,
320 Schema, etc.
321 --split-random
323 group: Split
324 Split log without an attribute, write queries round-robin to
326 session files.
327 This option, if specified, overrides "--split" and causes the log
329 to be split query-by-query, writing each query to the next session
330 file in round-robin style. If you don't care about "sessions" and
331 just want to split a lot into N many session files and the relation
332 or order of the queries does not matter, then use this option.
333 --threads
335 type: int; default: 2; group: Play
336 Number of threads used to play sessions concurrently.
338 Specifies the number of parallel processes to run. The default is
340 2. On GNU/Linux machines, the default is the number of times
341 'processor' appears in /proc/cpuinfo. On Windows, the default is
342 read from the environment. In any case, the default is at least 2,
343 even when there's only a single processor.
344 See also "--session-files".
346 --type
348 type: string; group: Split
349 The type of log to "--split" (default slowlog). The permitted
351 types are
352 binlog
354 Split a binary log file.
355 genlog
357 Split a general log file.
358 slowlog
360 Split a log file in any varation of MySQL slow-log format.
361 --user
363 short form: -u; type: string; group: Play
364 User for login if not current user.
366 --verbose
368 short form: -v; cumulative: yes; default: 0
369 Increase verbosity; can specifiy multiple times.
371 This option is disabled by "--quiet".
373 --version
375 Show version and exit.
376 --wait-between-sessions
378 type: array; default: 0; group: Play
379 Not implemented yet.
381 The wait time is given in seconds with microsecond precision and
383 can be either a single value or a range. A single value causes an
384 exact wait; example: 0.010 = wait 10 milliseconds. A range causes
385 a random wait between the given value times; example: 0.001,1 =
386 random wait from 1 millisecond to 1 second.
387 --[no]warnings
389 default: no; group: Play
390 Print warnings about SQL errors such as invalid queries to STDERR.
392
394 These DSN options are used to create a DSN. Each option is given like
395 "option=value". The options are case-sensitive, so P and p are not the
396 same option. There cannot be whitespace before or after the "=" and if
397 the value contains whitespace it must be quoted. DSN options are
398 comma-separated. See the maatkit manpage for full details.
399 · A
401 dsn: charset; copy: yes
403 Default character set.
405 · D
407 dsn: database; copy: yes
409 Default database.
411 · F
413 dsn: mysql_read_default_file; copy: yes
415 Only read default options from the given file
417 · h
419 dsn: host; copy: yes
421 Connect to host.
423 · p
425 dsn: password; copy: yes
427 Password to use when connecting.
429 · P
431 dsn: port; copy: yes
433 Port number to use for connection.
435 · S
437 dsn: mysql_socket; copy: yes
439 Socket file to use for connection.
441 · u
443 dsn: user; copy: yes
445 User for login if not current user.
447
449 You can download Maatkit from Google Code at
450 <http://code.google.com/p/maatkit/>, or you can get any of the tools
451 easily with a command like the following:
452 wget http://www.maatkit.org/get/toolname
454 or
455 wget http://www.maatkit.org/trunk/toolname
456 Where "toolname" can be replaced with the name (or fragment of a name)
458 of any of the Maatkit tools. Once downloaded, they're ready to run; no
459 installation is needed. The first URL gets the latest released version
460 of the tool, and the second gets the latest trunk code from Subversion.
461
463 The environment variable "MKDEBUG" enables verbose debugging output in
464 all of the Maatkit tools:
465 MKDEBUG=1 mk-....
467
469 You need Perl and some core packages that ought to be installed in any
470 reasonably new version of Perl.
471
473 For list of known bugs see http://www.maatkit.org/bugs/mk-log-player
474 <http://www.maatkit.org/bugs/mk-log-player>.
475 Please use Google Code Issues and Groups to report bugs or request
477 support: <http://code.google.com/p/maatkit/>. You can also join
478 #maatkit on Freenode to discuss Maatkit.
479 Please include the complete command-line used to reproduce the problem
481 you are seeing, the version of all MySQL servers involved, the complete
482 output of the tool when run with "--version", and if possible,
483 debugging output produced by running with the "MKDEBUG=1" environment
484 variable.
485
487 This program is copyright 2008-2010 Percona Inc. Feedback and
488 improvements are welcome.
489 THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
491 WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
492 MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE.
493 This program is free software; you can redistribute it and/or modify it
495 under the terms of the GNU General Public License as published by the
496 Free Software Foundation, version 2; OR the Perl Artistic License. On
497 UNIX and similar systems, you can issue `man perlgpl' or `man
498 perlartistic' to read these licenses.
499 You should have received a copy of the GNU General Public License along
501 with this program; if not, write to the Free Software Foundation, Inc.,
502 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
503
505 Daniel Nichter
506
508 This tool is part of Maatkit, a toolkit for power users of MySQL.
509 Maatkit was created by Baron Schwartz; Baron and Daniel Nichter are the
510 primary code contributors. Both are employed by Percona. Financial
511 support for Maatkit development is primarily provided by Percona and
512 its clients.
513
515 This manual page documents Ver 1.0.8 Distrib 6839 $Revision: 6836 $.
516perl v5.12.1 2010-08-01 MK-LOG-PLAYER(1)