1bench(n)                Benchmarking/Performance tools                bench(n)
2
3
4
5______________________________________________________________________________
6

NAME

8       bench - bench - Processing benchmark suites
9

SYNOPSIS

11       package require Tcl  8.2
12
13       package require bench  ?0.4?
14
15       ::bench::locate pattern paths
16
17       ::bench::run ?option value...? interp_list file...
18
19       ::bench::versions interp_list
20
21       ::bench::del bench_result column
22
23       ::bench::edit bench_result column newvalue
24
25       ::bench::merge bench_result...
26
27       ::bench::norm bench_result column
28
29       ::bench::out::raw bench_result
30
31______________________________________________________________________________
32

DESCRIPTION

34       This  package provides commands for the execution of benchmarks written
35       in the bench language, and for the processing of results  generated  by
36       such execution.
37
38       A  reader interested in the bench language itself should start with the
39       bench language introduction and proceed from there to the formal  bench
40       language specification.
41

PUBLIC API

43   BENCHMARK EXECUTION
44       ::bench::locate pattern paths
45              This  command  locates  Tcl interpreters and returns a list con‐
46              taining their paths. It searches them in the list of paths spec‐
47              ified by the caller, using the glob pattern.
48
49              The  command  resolves soft links to find the actual executables
50              matching the pattern. Note  that  only  interpreters  which  are
51              marked  as executable and are actually executable on the current
52              platform are put into the result.
53
54       ::bench::run ?option value...? interp_list file...
55              This command executes the benchmarks  declared  in  the  set  of
56              files,  once  per Tcl interpreter specified via the interp_list,
57              and per the configuration specified by the options, and then re‐
58              turns  the accumulated timing results. The format of this result
59              is described in section Result format.
60
61              It is assumed that the contents of the files are written in  the
62              bench language.
63
64              The available options are
65
66              -errors flag
67                     The  argument is a boolean value. If set errors in bench‐
68                     marks are propagated to the command,  aborting  benchmark
69                     execution.  Otherwise they are recorded in the timing re‐
70                     sult via a special result code. The default is to  propa‐
71                     gate and abort.
72
73              -threads n
74                     The  argument  is  a non-negative integer value declaring
75                     the number of threads to use while executing  the  bench‐
76                     marks. The default value is 0, to not use threads.
77
78              -match pattern
79                     The argument is a glob pattern. Only benchmarks whose de‐
80                     scription matches the pattern are executed.  The  default
81                     is the empty string, to execute all patterns.
82
83              -rmatch pattern
84                     The argument is a regular expression pattern. Only bench‐
85                     marks whose description matches the pattern are executed.
86                     The default is the empty string, to execute all patterns.
87
88              -iters n
89                     The argument is positive integer number, the maximal num‐
90                     ber of iterations for any benchmark. The default is 1000.
91                     Individual benchmarks can override this.
92
93              -pkgdir path
94                     The  argument  is  a path to an existing, readable direc‐
95                     tory. Multiple paths can be specified, simply use the op‐
96                     tion  multiple  times, each time with one of the paths to
97                     use.
98
99                     If no paths were specified the system will behave as  be‐
100                     fore.  If one or more paths are specified, say N, each of
101                     the specified interpreters will be invoked N times,  with
102                     one  of  the specified paths. The chosen path is put into
103                     the interpreters' auto_path, thus  allowing  it  to  find
104                     specific versions of a package.
105
106                     In  this way the use of -pkgdir allows the user to bench‐
107                     mark several different versions of a package, against one
108                     or more interpreters.
109
110                     Note:  The  empty  string is allowed as a path and causes
111                     the system to run the specified interpreters with an  un‐
112                     modified  auto_path.  In  case the package in question is
113                     available there as well.
114
115
116       ::bench::versions interp_list
117              This command takes a list of  Tcl  interpreters,  identified  by
118              their  path,  and  returns  a dictionary mapping from the inter‐
119              preters to their versions. Interpreters which are  not  actually
120              executable,  or fail when interrogated, are not put into the re‐
121              sult. I.e the result may contain less interpreters than there in
122              the input list.
123
124              The  command  uses  builtin command info patchlevel to determine
125              the version of each interpreter.
126
127   RESULT MANIPULATION
128       ::bench::del bench_result column
129              This command removes a column, i.e. all benchmark results for  a
130              specific  Tcl  interpreter,  from the specified benchmark result
131              and returns the modified result.
132
133              The benchmark results are in the format described in section Re‐
134              sult format.
135
136              The column is identified by an integer number.
137
138       ::bench::edit bench_result column newvalue
139              This  command renames a column in the specified benchmark result
140              and returns the modified result. This means that the path of the
141              Tcl  interpreter in the identified column is changed to an arbi‐
142              trary string.
143
144              The benchmark results are in the format described in section Re‐
145              sult format.
146
147              The column is identified by an integer number.
148
149       ::bench::merge bench_result...
150              This  commands  takes one or more benchmark results, merges them
151              into one big result, and returns that as its result.
152
153              All benchmark results are in the format described in section Re‐
154              sult format.
155
156       ::bench::norm bench_result column
157              This  command  normalizes  the  timing  results in the specified
158              benchmark result and returns the  modified  result.  This  means
159              that  the cell values are not times anymore, but factors showing
160              how much faster or slower the  execution  was  relative  to  the
161              baseline.
162
163              The baseline against which the command normalizes are the timing
164              results in the chosen column. This means that after the  normal‐
165              ization the values in this column are all 1, as these benchmarks
166              are neither faster nor slower than the baseline.
167
168              A factor less than 1 indicates a benchmark which was faster than
169              the baseline, whereas a factor greater than 1 indicates a slower
170              execution.
171
172              The benchmark results are in the format described in section Re‐
173              sult format.
174
175              The column is identified by an integer number.
176
177       ::bench::out::raw bench_result
178              This  command  formats the specified benchmark result for output
179              to a file, socket, etc. This specific command does no formatting
180              at all, it passes the input through unchanged.
181
182              For  other  formatting  styles see the packages bench::out::text
183              and bench::out::csv which provide commands to  format  benchmark
184              results  for  human  consumption,  or  as CSV data importable by
185              spread sheets, respectively.
186
187              Complementary, to read benchmark  results  from  files,  sockets
188              etc. look for the package bench::in and the commands provided by
189              it.
190
191   RESULT FORMAT
192       After the execution of a set of benchmarks the raw result  returned  by
193       this  package  is a Tcl dictionary containing all the relevant informa‐
194       tion.  The dictionary is a compact representation, i.e.  serialization,
195       of  a  2-dimensional  table  which  has Tcl interpreters as columns and
196       benchmarks as rows. The cells of the table contain the timing  results.
197       The  Tcl  interpreters  /  columns  are identified by their paths.  The
198       benchmarks / rows are identified by their description.
199
200       The possible keys are all valid Tcl lists of two or three elements  and
201       have one of the following forms:
202
203       {interp *}
204              The  set of keys matching this glob pattern capture the informa‐
205              tion about all the Tcl interpreters used to run the  benchmarks.
206              The second element of the key is the path to the interpreter.
207
208              The associated value is the version of the Tcl interpreter.
209
210       {desc *}
211              The  set of keys matching this glob pattern capture the informa‐
212              tion about all the benchmarks found in  the  executed  benchmark
213              suite.  The  second element of the key is the description of the
214              benchmark, which has to be unique.
215
216              The associated value is irrelevant, and set to the empty string.
217
218       {usec * *}
219              The set of keys matching this glob pattern capture  the  perfor‐
220              mance  information,  i.e.  timing results. The second element of
221              the key is the description of the benchmark, the  third  element
222              the path of the Tcl interpreter which was used to run it.
223
224              The  associated  value  is  either one of several special result
225              codes, or the time it took to  execute  the  benchmark,  in  mi‐
226              croseconds. The possible special result codes are
227
228              ERR    Benchmark could not be executed, failed with a Tcl error.
229
230              BAD_RES
231                     The  benchmark could be executed, however the result from
232                     its body did not match the declared expectations.
233

BUGS, IDEAS, FEEDBACK

235       This document, and the package it describes, will  undoubtedly  contain
236       bugs  and  other problems.  Please report such in the category bench of
237       the  Tcllib  Trackers  [http://core.tcl.tk/tcllib/reportlist].   Please
238       also  report any ideas for enhancements you may have for either package
239       and/or documentation.
240
241       When proposing code changes, please provide unified diffs, i.e the out‐
242       put of diff -u.
243
244       Note  further  that  attachments  are  strongly  preferred over inlined
245       patches. Attachments can be made by going  to  the  Edit  form  of  the
246       ticket  immediately  after  its  creation, and then using the left-most
247       button in the secondary navigation bar.
248

SEE ALSO

250       bench_intro, bench_lang_intro, bench_lang_spec, bench_read, bench_wcsv,
251       bench_wtext
252

KEYWORDS

254       benchmark, merging, normalization, performance, testing
255

CATEGORY

257       Benchmark tools
258
260       Copyright (c) 2007-2008 Andreas Kupries <andreas_kupries@users.sourceforge.net>
261
262
263
264
265tcllib                                0.4                             bench(n)
Impressum