1Fsdb::Filter::dbcolmoviUnsgesrtaCtosn(t3r)ibuted Perl DoFcsudmbe:n:tFaitlitoenr::dbcolmovingstats(3)
2
6 dbcolmovingstats - compute moving statistics over a window of a column
7 of data
8
10 dbcolmovingstats [-am] [-w WINDOW] [-e EmptyValue] [-k KEY] column
11
13 Compute moving statistics over a COLUMN of data. Records containing
14 non-numeric data are considered null do not contribute to the stats
15 (optionally they are treated as zeros with "-a").
16 Statitics are computed over a WINDOW of samples of data.
18 [In progress 2020-11-12, but not completed: Alternatively, if a key
20 column is given with "-k KEY", then a we treat the key column as a time
21 value and compute the time-weighted mean.]
22 Currently we compute mean and sample standard deviation. (Note we only
24 compute sample standard deviation, not full population.) Optionally,
25 with "-m" we also compute median. (Currently there is no support for
26 generalized quantiles.)
27 Values before a sufficient number have been accumulated are given the
29 empty value (if specified with "-e"). If no empty value is given,
30 stats are computed on as many are possible if no empty value is
31 specified.
32 Dbcolmovingstats runs in O(1) memory, but must buffer a full window of
34 data. Quantiles currently will repeatedly sort the window and so may
35 perform poorly with wide windows.
36
38 -a or --include-non-numeric
39 Compute stats over all records (treat non-numeric records as zero
40 rather than just ignoring them).
41 -w or --window WINDOW
43 WINDOW of how many items to accumulate (defaults to 10). (For
44 compatibility with fsdb-1.x, -n is also supported.)
45 -k or --key KEY
47 The KEY specifies a field that is used to evaluate the window---a
48 window must span at most this range of value so the key field.
49 (For example, if KEY is the time and window is 60, then enough
50 samples will be added to make at most 60s of observations. With a
51 key, sampling can be irregular.) If key is specified, we also
52 output a moving_n field for how many samples are in each window.
53 -m or --median
55 Show median of the window in addition to mean.
56 -e E or --empty E
58 Give value E as the value for empty (null) records. This null
59 value is then output before a full window is accumulated.
60 -f FORMAT or --format FORMAT
62 Specify a printf(3)-style format for output mean and standard
63 deviation. Defaults to "%.5g".
64 Eventually we expect to support other options of dbcolstats.
66 This module also supports the standard fsdb options:
68 -d Enable debugging output.
70 -i or --input InputSource
72 Read from InputSource, typically a file name, or "-" for standard
73 input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue
74 objects.
75 -o or --output OutputDestination
77 Write to OutputDestination, typically a file name, or "-" for
78 standard output, or (if in Perl) a IO::Handle, Fsdb::IO or
79 Fsdb::BoundedQueue objects.
80 --autorun or --noautorun
82 By default, programs process automatically, but Fsdb::Filter
83 objects in Perl do not run until you invoke the run() method. The
84 "--(no)autorun" option controls that behavior within Perl.
85 --help
87 Show help.
88 --man
90 Show full manual.
91
93 Input:
94 #fsdb date epoch count
95 19980201 886320000 6
96 19980202 886406400 8
97 19980203 886492800 19
98 19980204 886579200 53
99 19980205 886665600 20
100 19980206 886752000 18
101 19980207 886838400 5
102 19980208 886924800 9
103 19980209 887011200 22
104 19980210 887097600 22
105 19980211 887184000 36
106 19980212 887270400 26
107 19980213 887356800 23
108 19980214 887443200 6
109 Command:
111 cat data.fsdb | dbmovingstats -e - -w 4 count
112 Output:
114 #fsdb date epoch count moving_mean moving_stddev
115 19980201 886320000 6 - -
116 19980202 886406400 8 - -
117 19980203 886492800 19 - -
118 19980204 886579200 53 21.5 21.764
119 19980205 886665600 20 25 19.442
120 19980206 886752000 18 27.5 17.02
121 19980207 886838400 5 24 20.445
122 19980208 886924800 9 13 7.1647
123 19980209 887011200 22 13.5 7.8528
124 19980210 887097600 22 14.5 8.8129
125 19980211 887184000 36 22.25 11.026
126 19980212 887270400 26 26.5 6.6081
127 19980213 887356800 23 26.75 6.3966
128 19980214 887443200 6 22.75 12.473
129 # | dbcolmovingstats -e - -n 4 count
130
132 Fsdb. dbcolstats. dbmultistats. dbrowdiff.
133
135 Currently there is no support for generalized quantiles.
136
138 new
139 $filter = new Fsdb::Filter::dbcolmovingstats(@arguments);
140 Create a new dbcolmovingstats object, taking command-line arguments.
142 set_defaults
144 $filter->set_defaults();
145 Internal: set up defaults.
147 parse_options
149 $filter->parse_options(@ARGV);
150 Internal: parse command-line arguments.
152 setup
154 $filter->setup();
155 Internal: setup, parse headers.
157 run
159 $filter->run();
160 Internal: run over each rows.
162
164 Copyright (C) 1991-2022 by John Heidemann <johnh@isi.edu>
165 This program is distributed under terms of the GNU general public
167 license, version 2. See the file COPYING with the distribution for
168 details.
169perl v5.36.0 2022-11-22 Fsdb::Filter::dbcolmovingstats(3)