1RRD-BEGINNERS(1) rrdtool RRD-BEGINNERS(1)
2
6 rrd-beginners - RRDtool Beginners' Guide
7
9 Helping new RRDtool users to understand the basics of RRDtool
10
12 This manual is an attempt to assist beginners in understanding the con‐
13 cepts of RRDtool. It sheds a light on differences between RRDtool and
14 other databases. With help of an example, it explains the structure of
15 RRDtool database. This is followed by an overview of the "graph" fea‐
16 ture of RRDtool. At the end, it has sample scripts that illustrate the
17 usage/wrapping of RRDtool within Shell or Perl scripts.
18 What makes RRDtool so special?
20 RRDtool is GNU licensed software developed by Tobias Oetiker, a system
22 manager at the Swiss Federal Institute of Technology. Though it is a
23 database, there are distinct differences between RRDtool databases and
24 other databases as listed below:
25 · RRDtool stores data; that makes it a back-end tool. The RRDtool
27 command set allows the creation of graphs; that makes it a front-
28 end tool as well. Other databases just store data and can not cre‐
29 ate graphs.
30 · In case of linear databases, new data gets appended at the bottom
32 of the database table. Thus its size keeps on increasing, whereas
33 the size of an RRDtool database is determined at creation time.
34 Imagine an RRDtool database as the perimeter of a circle. Data is
35 added along the perimeter. When new data reaches the starting
36 point, it overwrites existing data. This way, the size of an RRD‐
37 tool database always remains constant. The name "Round Robin" stems
38 from this behavior.
39 · Other databases store the values as supplied. RRDtool can be con‐
41 figured to calculate the rate of change from the previous to the
42 current value and store this information instead.
43 · Other databases get updated when values are supplied. The RRDtool
45 database is structured in such a way that it needs data at prede‐
46 fined time intervals. If it does not get a new value during the
47 interval, it stores an UNKNOWN value for that interval. So, when
48 using the RRDtool database, it is imperative to use scripts that
49 run at regular intervals to ensure a constant data flow to update
50 the RRDtool database.
51 RRDtool is designed to store time series of data. With every data
53 update, an associated time stamp is stored. Time is always expressed in
54 seconds passed since epoch (01-01-1970). RRDtool can be installed on
55 Unix as well as Windows. It comes with a command set to carry out vari‐
56 ous operations on RRD databases. This command set can be accessed from
57 the command line, as well as from Shell or Perl scripts. The scripts
58 act as wrappers for accessing data stored in RRDtool databases.
59 Understanding by an example
61 The structure of an RRD database is different than other linear data‐
63 bases. Other databases define tables with columns, and many other
64 parameters. These definitions sometimes are very complex, especially in
65 large databases. RRDtool databases are primarily used for monitoring
66 purposes and hence are very simple in structure. The parameters that
67 need to be defined are variables that hold values and archives of those
68 values. Being time sensitive, a couple of time related parameters are
69 also defined. Because of its structure, the definition of an RRDtool
70 database also includes a provision to specify specific actions to take
71 in the absence of update values. Data Source (DS), heartbeat, Date
72 Source Type (DST), Round Robin Archive (RRA), and Consolidation Func‐
73 tion (CF) are some of the terminologies related to RRDtool databases.
74 The structure of a database and the terminology associated with it can
76 be best explained with an example.
77 rrdtool create target.rrd \
79 --start 1023654125 \
80 --step 300 \
81 DS:mem:GAUGE:600:0:671744 \
82 RRA:AVERAGE:0.5:12:24 \
83 RRA:AVERAGE:0.5:288:31
84 This example creates a database named target.rrd. Start time
86 (1'023'654'125) is specified in total number of seconds since epoch
87 (time in seconds since 01-01-1970). While updating the database, the
88 update time is also specified. This update time MUST be large (later)
89 then start time and MUST be in seconds since epoch.
90 The step of 300 seconds indicates that database expects new values
92 every 300 seconds. The wrapper script should be scheduled to run every
93 step seconds so that it updates the database every step seconds.
94 DS (Data Source) is the actual variable which relates to the parameter
96 on the device that is monitored. Its syntax is
97 DS:variable_name:DST:heartbeat:min:max
99 DS is a key word. "variable_name" is a name under which the parameter
101 is saved in the database. There can be as many DSs in a database as
102 needed. After every step interval, a new value of DS is supplied to
103 update the database. This value is also called Primary Data Point
104 (PDP). In our example mentioned above, a new PDP is generated every 300
105 seconds.
106 Note, that if you do NOT supply new datapoints exactly every 300 sec‐
108 onds, this is not a problem, RRDtool will interpolate the data accord‐
109 ingly.
110 DST (Data Source Type) defines the type of the DS. It can be COUNTER,
112 DERIVE, ABSOLUTE, GAUGE. A DS declared as COUNTER will save the rate of
113 change of the value over a step period. This assumes that the value is
114 always increasing (the difference between the current and the previous
115 value is greater than 0). Traffic counters on a router are an ideal
116 candidate for using COUNTER as DST. DERIVE is the same as COUNTER, but
117 it allows negative values as well. If you want to see the rate of
118 change in free diskspace on your server, then you might want to use the
119 DERIVE data type. ABSOLUTE also saves the rate of change, but it
120 assumes that the previous value is set to 0. The difference between the
121 current and the previous value is always equal to the current value.
122 Thus it just stores the current value divided by the step interval (300
123 seconds in our example). GAUGE does not save the rate of change. It
124 saves the actual value itself. There are no divisions or calculations.
125 Memory consumption in a server is a typical example of gauge. The dif‐
126 ference between the different types DSTs can be explained better with
127 the following example:
128 Values = 300, 600, 900, 1200
130 Step = 300 seconds
131 COUNTER DS = 1, 1, 1, 1
132 DERIVE DS = 1, 1, 1, 1
133 ABSOLUTE DS = 1, 2, 3, 4
134 GAUGE DS = 300, 600, 900, 1200
135 The next parameter is heartbeat. In our example, heartbeat is 600 sec‐
137 onds. If the database does not get a new PDP within 300 seconds, it
138 will wait for another 300 seconds (total 600 seconds). If it doesn't
139 receive any PDP within 600 seconds, it will save an UNKNOWN value into
140 the database. This UNKNOWN value is a special feature of RRDtool - it
141 is much better than to assume a missing value was 0 (zero) or any other
142 number which might also be a valid data value. For example, the traf‐
143 fic flow counter on a router keeps increasing. Lets say, a value is
144 missed for an interval and 0 is stored instead of UNKNOWN. Now when the
145 next value becomes available, it will calculate the difference between
146 the current value and the previous value (0) which is not correct. So,
147 inserting the value UNKNOWN makes much more sense here.
148 The next two parameters are the minimum and maximum value, respec‐
150 tively. If the variable to be stored has predictable maximum and mini‐
151 mum values, this should be specified here. Any update value falling out
152 of this range will be stored as UNKNOWN.
153 The next line declares a round robin archive (RRA). The syntax for
155 declaring an RRA is
156 RRA:CF:xff:step:rows
158 RRA is the keyword to declare RRAs. The consolidation function (CF) can
160 be AVERAGE, MINIMUM, MAXIMUM, and LAST. The concept of the consolidated
161 data point (CDP) comes into the picture here. A CDP is CFed (averaged,
162 maximum/minimum value or last value) from step number of PDPs. This RRA
163 will hold rows CDPs.
164 Lets have a look at the example above. For the first RRA, 12 (steps)
166 PDPs (DS variables) are AVERAGEed (CF) to form one CDP. 24 (rows) of
167 theses CDPs are archived. Each PDP occurs at 300 seconds. 12 PDPs rep‐
168 resent 12 times 300 seconds which is 1 hour. It means 1 CDP (which is
169 equal to 12 PDPs) represents data worth 1 hour. 24 such CDPs represent
170 1 day (1 hour times 24 CDPs). This means, this RRA is an archive for
171 one day. After 24 CDPs, CDP number 25 will replace the 1st CDP. The
172 second RRA saves 31 CDPs; each CPD represents an AVERAGE value for a
173 day (288 PDPs, each covering 300 seconds = 24 hours). Therefore this
174 RRA is an archive for one month. A single database can have many RRAs.
175 If there are multiple DSs, each individual RRA will save data for all
176 the DSs in the database. For example, if a database has 3 DSs and
177 daily, weekly, monthly, and yearly RRAs are declared, then each RRA
178 will hold data from all 3 data sources.
179 Graphical Magic
181 Another important feature of RRDtool is its ability to create graphs.
183 The "graph" command uses the "fetch" command internally to retrieve
184 values from the database. With the retrieved values it draws graphs as
185 defined by the parameters supplied on the command line. A single graph
186 can show different DS (Data Sources) from a database. It is also possi‐
187 ble to show the values from more than one database in a single graph.
188 Often, it is necessary to perform some math on the values retrieved
189 from the database before plotting them. For example, in SNMP replies,
190 memory consumption values are usually specified in KBytes and traffic
191 flow on interfaces is specified in Bytes. Graphs for these values will
192 be more meaningful if values are represented in MBytes and mbps. The
193 RRDtool graph command allows to define such conversions. Apart from
194 mathematical calculations, it is also possible to perform logical oper‐
195 ations such as greater than, less than, and if/then/else. If a database
196 contains more than one RRA archive, then a question may arise - how
197 does RRDtool decide which RRA archive to use for retrieving the values?
198 RRDtool looks at several things when making its choice. First it makes
199 sure that the RRA covers as much of the graphing time frame as possi‐
200 ble. Second it looks at the resolution of the RRA compared to the reso‐
201 lution of the graph. It tries to find one which has the same or higher
202 better resolution. With the "-r" option you can force RRDtool to assume
203 a different resolution than the one calculated from the pixel width of
204 the graph.
205 Values of different variables can be presented in 5 different shapes in
207 a graph - AREA, LINE1, LINE2, LINE3, and STACK. AREA is represented by
208 a solid colored area with values as the boundary of this area.
209 LINE1/2/3 (increasing width) are just plain lines representing the val‐
210 ues. STACK is also an area but it is "stack"ed on top AREA or
211 LINE1/2/3. Another important thing to note is that variables are plot‐
212 ted in the order they are defined in the graph command. Therefore care
213 must be taken to define STACK only after defining AREA/LINE. It is also
214 possible to put formatted comments within the graph. Detailed instruc‐
215 tions can be found in the graph manual.
216 Wrapping RRDtool within Shell/Perl script
218 After understanding RRDtool it is now a time to actually use RRDtool in
220 scripts. Tasks involved in network management are data collection, data
221 storage, and data retrieval. In the following example, the previously
222 created target.rrd database is used. Data collection and data storage
223 is done using Shell scripts. Data retrieval and report generation is
224 done using Perl scripts. These scripts are shown below:
225 Shell script (collects data, updates database)
227 #!/bin/sh
229 a=0
230 while [ "$a" == 0 ]; do
231 snmpwalk -c public 192.168.1.250 hrSWRunPerfMem > snmp_reply
232 total_mem=`awk 'BEGIN {tot_mem=0}
233 { if ($NF == "KBytes")
234 {tot_mem=tot_mem+$(NF-1)}
235 }
236 END {print tot_mem}' snmp_reply`
237 # I can use N as a replacement for the current time
238 rrdtool update target.rrd N:$total_mem
239 # sleep until the next 300 seconds are full
240 perl -e 'sleep 300 - time % 300'
241 done # end of while loop
242 Perl script (retrieves data from database and generates graphs and sta‐
244 tistics)
245 #!/usr/bin/perl -w
247 # This script fetches data from target.rrd, creates a graph of memory
248 # consumption on the target (Dual P3 Processor 1 GHz, 656 MB RAM)
249 # call the RRD perl module
251 use lib qw( /usr/local/rrdtool-1.0.41/lib/perl ../lib/perl );
252 use RRDs;
253 my $cur_time = time(); # set current time
254 my $end_time = $cur_time - 86400; # set end time to 24 hours ago
255 my $start_time = $end_time - 2592000; # set start 30 days in the past
256 # fetch average values from the RRD database between start and end time
258 my ($start,$step,$ds_names,$data) =
259 RRDs::fetch("target.rrd", "AVERAGE",
260 "-r", "600", "-s", "$start_time", "-e", "$end_time");
261 # save fetched values in a 2-dimensional array
262 my $rows = 0;
263 my $columns = 0;
264 my $time_variable = $start;
265 foreach $line (@$data) {
266 $vals[$rows][$columns] = $time_variable;
267 $time_variable = $time_variable + $step;
268 foreach $val (@$line) {
269 $vals[$rows][++$columns] = $val;}
270 $rows++;
271 $columns = 0;
272 }
273 my $tot_time = 0;
274 my $count = 0;
275 # save the values from the 2-dimensional into a 1-dimensional array
276 for $i ( 0 .. $#vals ) {
277 $tot_mem[$count] = $vals[$i][1];
278 $count++;
279 }
280 my $tot_mem_sum = 0;
281 # calculate the total of all values
282 for $i ( 0 .. ($count-1) ) {
283 $tot_mem_sum = $tot_mem_sum + $tot_mem[$i];
284 }
285 # calculate the average of the array
286 my $tot_mem_ave = $tot_mem_sum/($count);
287 # create the graph
288 RRDs::graph ("/images/mem_$count.png", \
289 "--title= Memory Usage", \
290 "--vertical-label=Memory Consumption (MB)", \
291 "--start=$start_time", \
292 "--end=$end_time", \
293 "--color=BACK#CCCCCC", \
294 "--color=CANVAS#CCFFFF", \
295 "--color=SHADEB#9999CC", \
296 "--height=125", \
297 "--upper-limit=656", \
298 "--lower-limit=0", \
299 "--rigid", \
300 "--base=1024", \
301 "DEF:tot_mem=target.rrd:mem:AVERAGE", \
302 "CDEF:tot_mem_cor=tot_mem,0,671744,LIMIT,UN,0,tot_mem,IF,1024,/",\
303 "CDEF:machine_mem=tot_mem,656,+,tot_mem,-",\
304 "COMMENT:Memory Consumption between $start_time",\
305 "COMMENT: and $end_time ",\
306 "HRULE:656#000000:Maximum Available Memory - 656 MB",\
307 "AREA:machine_mem#CCFFFF:Memory Unused", \
308 "AREA:tot_mem_cor#6699CC:Total memory consumed in MB");
309 my $err=RRDs::error;
310 if ($err) {print "problem generating the graph: $err\n";}
311 # print the output
312 print "Average memory consumption is ";
313 printf "%5.2f",$tot_mem_ave/1024;
314 print " MB. Graphical representation can be found at /images/mem_$count.png.";
315
317 Ketan Patel <k2pattu@yahoo.com>
3181.2.27 2008-02-17 RRD-BEGINNERS(1)