1bench(n)                Benchmarking/Performance tools                bench(n)
2
3
4
5______________________________________________________________________________
6

NAME

8       bench - bench - Processing benchmark suites
9

SYNOPSIS

11       package require Tcl  8.2
12
13       package require bench  ?0.4?
14
15       ::bench::locate pattern paths
16
17       ::bench::run ?option value...? interp_list file...
18
19       ::bench::versions interp_list
20
21       ::bench::del bench_result column
22
23       ::bench::edit bench_result column newvalue
24
25       ::bench::merge bench_result...
26
27       ::bench::norm bench_result column
28
29       ::bench::out::raw bench_result
30
31______________________________________________________________________________
32

DESCRIPTION

34       This  package provides commands for the execution of benchmarks written
35       in the bench language, and for the processing of results  generated  by
36       such execution.
37
38       A  reader interested in the bench language itself should start with the
39       bench language introduction and proceed from there to the formal  bench
40       language specification.
41

PUBLIC API

43   BENCHMARK EXECUTION
44       ::bench::locate pattern paths
45              This  command  locates  Tcl interpreters and returns a list con‐
46              taining their paths. It searches them in the list of paths spec‐
47              ified by the caller, using the glob pattern.
48
49              The  command  resolves soft links to find the actual executables
50              matching the pattern. Note  that  only  interpreters  which  are
51              marked  as executable and are actually executable on the current
52              platform are put into the result.
53
54       ::bench::run ?option value...? interp_list file...
55              This command executes the benchmarks  declared  in  the  set  of
56              files,  once  per Tcl interpreter specified via the interp_list,
57              and per the configuration specified by  the  options,  and  then
58              returns  the  accumulated  timing  results.  The  format of this
59              result is described in section Result format.
60
61              It is assumed that the contents of the files are written in  the
62              bench language.
63
64              The available options are
65
66              -errors flag
67                     The  argument is a boolean value. If set errors in bench‐
68                     marks are propagated to the command,  aborting  benchmark
69                     execution.  Otherwise  they  are  recorded  in the timing
70                     result via a special result code. The default is to prop‐
71                     agate and abort.
72
73              -threads n
74                     The  argument  is  a non-negative integer value declaring
75                     the number of threads to use while executing  the  bench‐
76                     marks. The default value is 0, to not use threads.
77
78              -match pattern
79                     The  argument  is  a  glob pattern. Only benchmarks whose
80                     description matches the pattern are executed. The default
81                     is the empty string, to execute all patterns.
82
83              -rmatch pattern
84                     The argument is a regular expression pattern. Only bench‐
85                     marks whose description matches the pattern are executed.
86                     The default is the empty string, to execute all patterns.
87
88              -iters n
89                     The argument is positive integer number, the maximal num‐
90                     ber of iterations for any benchmark. The default is 1000.
91                     Individual benchmarks can override this.
92
93              -pkgdir path
94                     The  argument  is  a path to an existing, readable direc‐
95                     tory. Multiple paths can be  specified,  simply  use  the
96                     option multiple times, each time with one of the paths to
97                     use.
98
99                     If no paths were specified  the  system  will  behave  as
100                     before.   If one or more paths are specified, say N, each
101                     of the specified interpreters will be  invoked  N  times,
102                     with  one  of the specified paths. The chosen path is put
103                     into the interpreters' auto_path,  thus  allowing  it  to
104                     find specific versions of a package.
105
106                     In  this way the use of -pkgdir allows the user to bench‐
107                     mark several different versions of a package, against one
108                     or more interpreters.
109
110                     Note:  The  empty  string is allowed as a path and causes
111                     the system to run  the  specified  interpreters  with  an
112                     unmodified  auto_path. In case the package in question is
113                     available there as well.
114
115
116       ::bench::versions interp_list
117              This command takes a list of  Tcl  interpreters,  identified  by
118              their  path,  and  returns  a dictionary mapping from the inter‐
119              preters to their versions. Interpreters which are  not  actually
120              executable,  or  fail  when  interrogated,  are not put into the
121              result. I.e the result may contain less interpreters than  there
122              in the input list.
123
124              The  command  uses  builtin command info patchlevel to determine
125              the version of each interpreter.
126
127   RESULT MANIPULATION
128       ::bench::del bench_result column
129              This command removes a column, i.e. all benchmark results for  a
130              specific  Tcl  interpreter,  from the specified benchmark result
131              and returns the modified result.
132
133              The benchmark results are in the  format  described  in  section
134              Result format.
135
136              The column is identified by an integer number.
137
138       ::bench::edit bench_result column newvalue
139              This  command renames a column in the specified benchmark result
140              and returns the modified result. This means that the path of the
141              Tcl  interpreter in the identified column is changed to an arbi‐
142              trary string.
143
144              The benchmark results are in the  format  described  in  section
145              Result format.
146
147              The column is identified by an integer number.
148
149       ::bench::merge bench_result...
150              This  commands  takes one or more benchmark results, merges them
151              into one big result, and returns that as its result.
152
153              All benchmark results are in the  format  described  in  section
154              Result format.
155
156       ::bench::norm bench_result column
157              This  command  normalizes  the  timing  results in the specified
158              benchmark result and returns the  modified  result.  This  means
159              that  the cell values are not times anymore, but factors showing
160              how much faster or slower the  execution  was  relative  to  the
161              baseline.
162
163              The baseline against which the command normalizes are the timing
164              results in the chosen column. This means that after the  normal‐
165              ization the values in this column are all 1, as these benchmarks
166              are neither faster nor slower than the baseline.
167
168              A factor less than 1 indicates a benchmark which was faster than
169              the baseline, whereas a factor greater than 1 indicates a slower
170              execution.
171
172              The benchmark results are in the  format  described  in  section
173              Result format.
174
175              The column is identified by an integer number.
176
177       ::bench::out::raw bench_result
178              This  command  formats the specified benchmark result for output
179              to a file, socket, etc. This specific command does no formatting
180              at all, it passes the input through unchanged.
181
182              For  other  formatting  styles see the packages bench::out::text
183              and bench::out::csv which provide commands to  format  benchmark
184              results  for  human  consumption,  or  as CSV data importable by
185              spread sheets, respectively.
186
187              Complementary, to read benchmark  results  from  files,  sockets
188              etc. look for the package bench::in and the commands provided by
189              it.
190
191   RESULT FORMAT
192       After the execution of a set of benchmarks the raw result  returned  by
193       this  package  is a Tcl dictionary containing all the relevant informa‐
194       tion.  The dictionary is a compact representation, i.e.  serialization,
195       of  a  2-dimensional  table  which  has Tcl interpreters as columns and
196       benchmarks as rows. The cells of the table contain the timing  results.
197       The  Tcl  interpreters  /  columns  are identified by their paths.  The
198       benchmarks / rows are identified by their description.
199
200       The possible keys are all valid Tcl lists of two or three elements  and
201       have one of the following forms:
202
203       {interp *}
204              The  set of keys matching this glob pattern capture the informa‐
205              tion about all the Tcl interpreters used to run the  benchmarks.
206              The second element of the key is the path to the interpreter.
207
208              The associated value is the version of the Tcl interpreter.
209
210       {desc *}
211              The  set of keys matching this glob pattern capture the informa‐
212              tion about all the benchmarks found in  the  executed  benchmark
213              suite.  The  second element of the key is the description of the
214              benchmark, which has to be unique.
215
216              The associated value is irrelevant, and set to the empty string.
217
218       {usec * *}
219              The set of keys matching this glob pattern capture  the  perfor‐
220              mance  information,  i.e.  timing results. The second element of
221              the key is the description of the benchmark, the  third  element
222              the path of the Tcl interpreter which was used to run it.
223
224              The  associated  value  is  either one of several special result
225              codes, or  the  time  it  took  to  execute  the  benchmark,  in
226              microseconds. The possible special result codes are
227
228              ERR    Benchmark could not be executed, failed with a Tcl error.
229
230              BAD_RES
231                     The  benchmark could be executed, however the result from
232                     its body did not match the declared expectations.
233

BUGS, IDEAS, FEEDBACK

235       This document, and the package it describes, will  undoubtedly  contain
236       bugs  and  other problems.  Please report such in the category bench of
237       the  Tcllib  Trackers  [http://core.tcl.tk/tcllib/reportlist].   Please
238       also  report any ideas for enhancements you may have for either package
239       and/or documentation.
240
241       When proposing code changes, please provide unified diffs, i.e the out‐
242       put of diff -u.
243
244       Note  further  that  attachments  are  strongly  preferred over inlined
245       patches. Attachments can be made by going  to  the  Edit  form  of  the
246       ticket  immediately  after  its  creation, and then using the left-most
247       button in the secondary navigation bar.
248

SEE ALSO

250       bench_intro, bench_lang_intro, bench_lang_spec, bench_read, bench_wcsv,
251       bench_wtext
252

KEYWORDS

254       benchmark, merging, normalization, performance, testing
255

CATEGORY

257       Benchmark tools
258
260       Copyright (c) 2007-2008 Andreas Kupries <andreas_kupries@users.sourceforge.net>
261
262
263
264
265tcllib                                0.4                             bench(n)
Impressum