1bench(n) Benchmarking/Performance tools bench(n)
2______________________________________________________________________________
6
8 bench - bench - Processing benchmark suites
9
11 package require Tcl 8.2
12 package require bench ?0.4?
14 ::bench::locate pattern paths
16 ::bench::run ?option value...? interp_list file...
18 ::bench::versions interp_list
20 ::bench::del bench_result column
22 ::bench::edit bench_result column newvalue
24 ::bench::merge bench_result...
26 ::bench::norm bench_result column
28 ::bench::out::raw bench_result
30______________________________________________________________________________
32
34 This package provides commands for the execution of benchmarks written
35 in the bench language, and for the processing of results generated by
36 such execution.
37 A reader interested in the bench language itself should start with the
39 bench language introduction and proceed from there to the formal bench
40 language specification.
41
43 BENCHMARK EXECUTION
44 ::bench::locate pattern paths
45 This command locates Tcl interpreters and returns a list con‐
46 taining their paths. It searches them in the list of paths spec‐
47 ified by the caller, using the glob pattern.
48 The command resolves soft links to find the actual executables
50 matching the pattern. Note that only interpreters which are
51 marked as executable and are actually executable on the current
52 platform are put into the result.
53 ::bench::run ?option value...? interp_list file...
55 This command executes the benchmarks declared in the set of
56 files, once per Tcl interpreter specified via the interp_list,
57 and per the configuration specified by the options, and then re‐
58 turns the accumulated timing results. The format of this result
59 is described in section Result format.
60 It is assumed that the contents of the files are written in the
62 bench language.
63 The available options are
65 -errors flag
67 The argument is a boolean value. If set errors in bench‐
68 marks are propagated to the command, aborting benchmark
69 execution. Otherwise they are recorded in the timing re‐
70 sult via a special result code. The default is to propa‐
71 gate and abort.
72 -threads n
74 The argument is a non-negative integer value declaring
75 the number of threads to use while executing the bench‐
76 marks. The default value is 0, to not use threads.
77 -match pattern
79 The argument is a glob pattern. Only benchmarks whose de‐
80 scription matches the pattern are executed. The default
81 is the empty string, to execute all patterns.
82 -rmatch pattern
84 The argument is a regular expression pattern. Only bench‐
85 marks whose description matches the pattern are executed.
86 The default is the empty string, to execute all patterns.
87 -iters n
89 The argument is positive integer number, the maximal num‐
90 ber of iterations for any benchmark. The default is 1000.
91 Individual benchmarks can override this.
92 -pkgdir path
94 The argument is a path to an existing, readable direc‐
95 tory. Multiple paths can be specified, simply use the op‐
96 tion multiple times, each time with one of the paths to
97 use.
98 If no paths were specified the system will behave as be‐
100 fore. If one or more paths are specified, say N, each of
101 the specified interpreters will be invoked N times, with
102 one of the specified paths. The chosen path is put into
103 the interpreters' auto_path, thus allowing it to find
104 specific versions of a package.
105 In this way the use of -pkgdir allows the user to bench‐
107 mark several different versions of a package, against one
108 or more interpreters.
109 Note: The empty string is allowed as a path and causes
111 the system to run the specified interpreters with an un‐
112 modified auto_path. In case the package in question is
113 available there as well.
114 ::bench::versions interp_list
117 This command takes a list of Tcl interpreters, identified by
118 their path, and returns a dictionary mapping from the inter‐
119 preters to their versions. Interpreters which are not actually
120 executable, or fail when interrogated, are not put into the re‐
121 sult. I.e the result may contain less interpreters than there in
122 the input list.
123 The command uses builtin command info patchlevel to determine
125 the version of each interpreter.
126 RESULT MANIPULATION
128 ::bench::del bench_result column
129 This command removes a column, i.e. all benchmark results for a
130 specific Tcl interpreter, from the specified benchmark result
131 and returns the modified result.
132 The benchmark results are in the format described in section Re‐
134 sult format.
135 The column is identified by an integer number.
137 ::bench::edit bench_result column newvalue
139 This command renames a column in the specified benchmark result
140 and returns the modified result. This means that the path of the
141 Tcl interpreter in the identified column is changed to an arbi‐
142 trary string.
143 The benchmark results are in the format described in section Re‐
145 sult format.
146 The column is identified by an integer number.
148 ::bench::merge bench_result...
150 This commands takes one or more benchmark results, merges them
151 into one big result, and returns that as its result.
152 All benchmark results are in the format described in section Re‐
154 sult format.
155 ::bench::norm bench_result column
157 This command normalizes the timing results in the specified
158 benchmark result and returns the modified result. This means
159 that the cell values are not times anymore, but factors showing
160 how much faster or slower the execution was relative to the
161 baseline.
162 The baseline against which the command normalizes are the timing
164 results in the chosen column. This means that after the normal‐
165 ization the values in this column are all 1, as these benchmarks
166 are neither faster nor slower than the baseline.
167 A factor less than 1 indicates a benchmark which was faster than
169 the baseline, whereas a factor greater than 1 indicates a slower
170 execution.
171 The benchmark results are in the format described in section Re‐
173 sult format.
174 The column is identified by an integer number.
176 ::bench::out::raw bench_result
178 This command formats the specified benchmark result for output
179 to a file, socket, etc. This specific command does no formatting
180 at all, it passes the input through unchanged.
181 For other formatting styles see the packages bench::out::text
183 and bench::out::csv which provide commands to format benchmark
184 results for human consumption, or as CSV data importable by
185 spread sheets, respectively.
186 Complementary, to read benchmark results from files, sockets
188 etc. look for the package bench::in and the commands provided by
189 it.
190 RESULT FORMAT
192 After the execution of a set of benchmarks the raw result returned by
193 this package is a Tcl dictionary containing all the relevant informa‐
194 tion. The dictionary is a compact representation, i.e. serialization,
195 of a 2-dimensional table which has Tcl interpreters as columns and
196 benchmarks as rows. The cells of the table contain the timing results.
197 The Tcl interpreters / columns are identified by their paths. The
198 benchmarks / rows are identified by their description.
199 The possible keys are all valid Tcl lists of two or three elements and
201 have one of the following forms:
202 {interp *}
204 The set of keys matching this glob pattern capture the informa‐
205 tion about all the Tcl interpreters used to run the benchmarks.
206 The second element of the key is the path to the interpreter.
207 The associated value is the version of the Tcl interpreter.
209 {desc *}
211 The set of keys matching this glob pattern capture the informa‐
212 tion about all the benchmarks found in the executed benchmark
213 suite. The second element of the key is the description of the
214 benchmark, which has to be unique.
215 The associated value is irrelevant, and set to the empty string.
217 {usec * *}
219 The set of keys matching this glob pattern capture the perfor‐
220 mance information, i.e. timing results. The second element of
221 the key is the description of the benchmark, the third element
222 the path of the Tcl interpreter which was used to run it.
223 The associated value is either one of several special result
225 codes, or the time it took to execute the benchmark, in mi‐
226 croseconds. The possible special result codes are
227 ERR Benchmark could not be executed, failed with a Tcl error.
229 BAD_RES
231 The benchmark could be executed, however the result from
232 its body did not match the declared expectations.
233
235 This document, and the package it describes, will undoubtedly contain
236 bugs and other problems. Please report such in the category bench of
237 the Tcllib Trackers [http://core.tcl.tk/tcllib/reportlist]. Please
238 also report any ideas for enhancements you may have for either package
239 and/or documentation.
240 When proposing code changes, please provide unified diffs, i.e the out‐
242 put of diff -u.
243 Note further that attachments are strongly preferred over inlined
245 patches. Attachments can be made by going to the Edit form of the
246 ticket immediately after its creation, and then using the left-most
247 button in the secondary navigation bar.
248
250 bench_intro, bench_lang_intro, bench_lang_spec, bench_read, bench_wcsv,
251 bench_wtext
252
254 benchmark, merging, normalization, performance, testing
255
257 Benchmark tools
258
260 Copyright (c) 2007-2008 Andreas Kupries <andreas_kupries@users.sourceforge.net>
261tcllib 0.4 bench(n)