1RRD-BEGINNERS(1)                    rrdtool                   RRD-BEGINNERS(1)
2
3
4

NAME

6       rrd-beginners - RRDtool Beginners' Guide
7

SYNOPSIS

9       Helping new RRDtool users to understand the basics of RRDtool
10

DESCRIPTION

12       This manual is an attempt to assist beginners in understanding the
13       concepts of RRDtool. It sheds a light on differences between RRDtool
14       and other databases. With help of an example, it explains the structure
15       of RRDtool database. This is followed by an overview of the "graph"
16       feature of RRDtool.  At the end, it has sample scripts that illustrate
17       the usage/wrapping of RRDtool within Shell or Perl scripts.
18
19   What makes RRDtool so special?
20       RRDtool is GNU licensed software developed by Tobias Oetiker, a system
21       manager at the Swiss Federal Institute of Technology. Though it is a
22       database, there are distinct differences between RRDtool databases and
23       other databases as listed below:
24
25       ·   RRDtool stores data; that makes it a back-end tool. The RRDtool
26           command set allows one to create graphs; that makes it a front-end
27           tool as well. Other databases just store data and can not create
28           graphs.
29
30       ·   In case of linear databases, new data gets appended at the bottom
31           of the database table. Thus its size keeps on increasing, whereas
32           the size of an RRDtool database is determined at creation time.
33           Imagine an RRDtool database as the perimeter of a circle. Data is
34           added along the perimeter. When new data reaches the starting
35           point, it overwrites existing data. This way, the size of an
36           RRDtool database always remains constant. The name "Round Robin"
37           stems from this behavior.
38
39       ·   Other databases store the values as supplied. RRDtool can be
40           configured to calculate the rate of change from the previous to the
41           current value and store this information instead.
42
43       ·   Other databases get updated when values are supplied. The RRDtool
44           database is structured in such a way that it needs data at
45           predefined time intervals. If it does not get a new value during
46           the interval, it stores an UNKNOWN value for that interval. So,
47           when using the RRDtool database, it is imperative to use scripts
48           that run at regular intervals to ensure a constant data flow to
49           update the RRDtool database.
50
51       RRDtool is designed to store time series of data. With every data
52       update, an associated time stamp is stored. Time is always expressed in
53       seconds passed since epoch (01-01-1970). RRDtool can be installed on
54       Unix as well as Windows. It comes with a command set to carry out
55       various operations on RRD databases. This command set can be accessed
56       from the command line, as well as from Shell or Perl scripts. The
57       scripts act as wrappers for accessing data stored in RRDtool databases.
58
59   Understanding by an example
60       The structure of an RRD database is different than other linear
61       databases.  Other databases define tables with columns, and many other
62       parameters. These definitions sometimes are very complex, especially in
63       large databases.  RRDtool databases are primarily used for monitoring
64       purposes and hence are very simple in structure. The parameters that
65       need to be defined are variables that hold values and archives of those
66       values. Being time sensitive, a couple of time related parameters are
67       also defined. Because of its structure, the definition of an RRDtool
68       database also includes a provision to specify specific actions to take
69       in the absence of update values. Data Source (DS), heartbeat, Date
70       Source Type (DST), Round Robin Archive (RRA), and Consolidation
71       Function (CF) are some of the terminologies related to RRDtool
72       databases.
73
74       The structure of a database and the terminology associated with it can
75       be best explained with an example.
76
77        rrdtool create target.rrd \
78                --start 1023654125 \
79                --step 300 \
80                DS:mem:GAUGE:600:0:671744 \
81                RRA:AVERAGE:0.5:12:24 \
82                RRA:AVERAGE:0.5:288:31
83
84       This example creates a database named target.rrd. Start time
85       (1'023'654'125) is specified in total number of seconds since epoch
86       (time in seconds since 01-01-1970). While updating the database, the
87       update time is also specified.  This update time MUST be larger (later)
88       than start time and MUST be in seconds since epoch.
89
90       The step of 300 seconds indicates that the database expects new values
91       every 300 seconds. The wrapper script should be scheduled to run every
92       step seconds so that it updates the database every step seconds.
93
94       DS (Data Source) is the actual variable which relates to the parameter
95       on the device that is monitored. Its syntax is
96
97        DS:variable_name:DST:heartbeat:min:max
98
99       DS is a key word. "variable_name" is a name under which the parameter
100       is saved in the database. There can be as many DSs in a database as
101       needed. After every step interval, a new value of DS is supplied to
102       update the database.  This value is also called Primary Data Point
103       (PDP). In our example mentioned above, a new PDP is generated every 300
104       seconds.
105
106       Note, that if you do NOT supply new data points exactly every 300
107       seconds, this is not a problem, RRDtool will interpolate the data
108       accordingly.
109
110       DST (Data Source Type) defines the type of the DS. It can be COUNTER,
111       DERIVE, ABSOLUTE, GAUGE. A DS declared as COUNTER will save the rate of
112       change of the value over a step period. This assumes that the value is
113       always increasing (the difference between the current and the previous
114       value is greater than 0). Traffic counters on a router are an ideal
115       candidate for using COUNTER as DST. DERIVE is the same as COUNTER, but
116       it allows negative values as well. If you want to see the rate of
117       change in free disk space on your server, then you might want to use
118       the DERIVE data type. ABSOLUTE also saves the rate of change, but it
119       assumes that the previous value is set to 0. The difference between the
120       current and the previous value is always equal to the current value.
121       Thus it just stores the current value divided by the step interval (300
122       seconds in our example). GAUGE does not save the rate of change. It
123       saves the actual value itself. There are no divisions or calculations.
124       Memory consumption in a server is a typical example of gauge. The
125       difference between the different types DSTs can be explained better
126       with the following example:
127
128        Values       = 300, 600, 900, 1200
129        Step         = 300 seconds
130        COUNTER DS   =    1,  1,   1,    1
131        DERIVE DS    =    1,  1,   1,    1
132        ABSOLUTE DS  =    1,  2,   3,    4
133        GAUGE DS     = 300, 600, 900, 1200
134
135       The next parameter is heartbeat. In our example, heartbeat is 600
136       seconds. If the database does not get a new PDP within 300 seconds, it
137       will wait for another 300 seconds (total 600 seconds).  If it doesn't
138       receive any PDP within 600 seconds, it will save an UNKNOWN value into
139       the database. This UNKNOWN value is a special feature of RRDtool - it
140       is much better than to assume a missing value was 0 (zero) or any other
141       number which might also be a valid data value.  For example, the
142       traffic flow counter on a router keeps increasing. Lets say, a value is
143       missed for an interval and 0 is stored instead of UNKNOWN. Now when the
144       next value becomes available, it will calculate the difference between
145       the current value and the previous value (0) which is not correct. So,
146       inserting the value UNKNOWN makes much more sense here.
147
148       The next two parameters are the minimum and maximum value,
149       respectively. If the variable to be stored has predictable maximum and
150       minimum values, this should be specified here. Any update value falling
151       out of this range will be stored as UNKNOWN.
152
153       The next line declares a round robin archive (RRA). The syntax for
154       declaring an RRA is
155
156        RRA:CF:xff:step:rows
157
158       RRA is the keyword to declare RRAs. The consolidation function (CF) can
159       be AVERAGE, MINIMUM, MAXIMUM, and LAST. The concept of the consolidated
160       data point (CDP) comes into the picture here. A CDP is CFed (averaged,
161       maximum/minimum value or last value) from step number of PDPs. This RRA
162       will hold rows CDPs.
163
164       Lets have a look at the example above. For the first RRA, 12 (steps)
165       PDPs (DS variables) are AVERAGEed (CF) to form one CDP. 24 (rows) of
166       theses CDPs are archived. Each PDP occurs at 300 seconds. 12 PDPs
167       represent 12 times 300 seconds which is 1 hour. It means 1 CDP (which
168       is equal to 12 PDPs) represents data worth 1 hour. 24 such CDPs
169       represent 1 day (1 hour times 24 CDPs). This means, this RRA is an
170       archive for one day. After 24 CDPs, CDP number 25 will replace the 1st
171       CDP. The second RRA saves 31 CDPs; each CPD represents an AVERAGE value
172       for a day (288 PDPs, each covering 300 seconds = 24 hours). Therefore
173       this RRA is an archive for one month. A single database can have many
174       RRAs. If there are multiple DSs, each individual RRA will save data for
175       all the DSs in the database. For example, if a database has 3 DSs and
176       daily, weekly, monthly, and yearly RRAs are declared, then each RRA
177       will hold data from all 3 data sources.
178
179   Graphical Magic
180       Another important feature of RRDtool is its ability to create graphs.
181       The "graph" command uses the "fetch" command internally to retrieve
182       values from the database. With the retrieved values it draws graphs as
183       defined by the parameters supplied on the command line. A single graph
184       can show different DS (Data Sources) from a database. It is also
185       possible to show the values from more than one database in a single
186       graph. Often, it is necessary to perform some math on the values
187       retrieved from the database before plotting them. For example, in SNMP
188       replies, memory consumption values are usually specified in KBytes and
189       traffic flow on interfaces is specified in Bytes. Graphs for these
190       values will be more meaningful if values are represented in MBytes and
191       mbps. The RRDtool graph command allows one to define such conversions.
192       Apart from mathematical calculations, it is also possible to perform
193       logical operations such as greater than, less than, and if/then/else.
194       If a database contains more than one RRA archive, then a question may
195       arise - how does RRDtool decide which RRA archive to use for retrieving
196       the values? RRDtool looks at several things when making its choice.
197       First it makes sure that the RRA covers as much of the graphing time
198       frame as possible. Second it looks at the resolution of the RRA
199       compared to the resolution of the graph. It tries to find one which has
200       the same or higher better resolution. With the "-r" option you can
201       force RRDtool to assume a different resolution than the one calculated
202       from the pixel width of the graph.
203
204       Values of different variables can be presented in 5 different shapes in
205       a graph - AREA, LINE1, LINE2, LINE3, and STACK. AREA is represented by
206       a solid colored area with values as the boundary of this area.
207       LINE1/2/3 (increasing width) are just plain lines representing the
208       values. STACK is also an area but it is "stack"ed on top AREA or
209       LINE1/2/3. Another important thing to note is that variables are
210       plotted in the order they are defined in the graph command. Therefore
211       care must be taken to define STACK only after defining AREA/LINE. It is
212       also possible to put formatted comments within the graph.  Detailed
213       instructions can be found in the graph manual.
214
215   Wrapping RRDtool within Shell/Perl script
216       After understanding RRDtool it is now a time to actually use RRDtool in
217       scripts. Tasks involved in network management are data collection, data
218       storage, and data retrieval. In the following example, the previously
219       created target.rrd database is used. Data collection and data storage
220       is done using Shell scripts. Data retrieval and report generation is
221       done using Perl scripts. These scripts are shown below:
222
223       Shell script (collects data, updates database)
224
225        #!/bin/sh
226        a=0
227        while [ "$a" == 0 ]; do
228        snmpwalk -c public 192.168.1.250 hrSWRunPerfMem > snmp_reply
229            total_mem=`awk 'BEGIN {tot_mem=0}
230                                  { if ($NF == "KBytes")
231                                    {tot_mem=tot_mem+$(NF-1)}
232                                  }
233                            END {print tot_mem}' snmp_reply`
234            # I can use N as a replacement for the current time
235            rrdtool update target.rrd N:$total_mem
236            # sleep until the next 300 seconds are full
237            perl -e 'sleep 300 - time % 300'
238        done # end of while loop
239
240       Perl script (retrieves data from database and generates graphs and
241       statistics)
242
243        #!/usr/bin/perl -w
244        # This script fetches data from target.rrd, creates a graph of memory
245        # consumption on the target (Dual P3 Processor 1 GHz, 656 MB RAM)
246
247        # call the RRD perl module
248        use lib qw( /usr/local/rrdtool-1.0.41/lib/perl ../lib/perl );
249        use RRDs;
250        my $cur_time = time();                # set current time
251        my $end_time = $cur_time - 86400;     # set end time to 24 hours ago
252        my $start_time = $end_time - 2592000; # set start 30 days in the past
253
254        # fetch average values from the RRD database between start and end time
255        my ($start,$step,$ds_names,$data) =
256            RRDs::fetch("target.rrd", "AVERAGE",
257                        "-r", "600", "-s", "$start_time", "-e", "$end_time");
258        # save fetched values in a 2-dimensional array
259        my $rows = 0;
260        my $columns = 0;
261        my $time_variable = $start;
262        foreach $line (@$data) {
263          $vals[$rows][$columns] = $time_variable;
264          $time_variable = $time_variable + $step;
265          foreach $val (@$line) {
266                  $vals[$rows][++$columns] = $val;}
267          $rows++;
268          $columns = 0;
269        }
270        my $tot_time = 0;
271        my $count = 0;
272        # save the values from the 2-dimensional into a 1-dimensional array
273        for $i ( 0 .. $#vals ) {
274            $tot_mem[$count] = $vals[$i][1];
275            $count++;
276        }
277        my $tot_mem_sum = 0;
278        # calculate the total of all values
279        for $i ( 0 .. ($count-1) ) {
280            $tot_mem_sum = $tot_mem_sum + $tot_mem[$i];
281        }
282        # calculate the average of the array
283        my $tot_mem_ave = $tot_mem_sum/($count);
284        # create the graph
285        RRDs::graph ("/images/mem_$count.png",
286                    "--title= Memory Usage",
287                    "--vertical-label=Memory Consumption (MB)",
288                    "--start=$start_time",
289                    "--end=$end_time",
290                    "--color=BACK#CCCCCC",
291                    "--color=CANVAS#CCFFFF",
292                    "--color=SHADEB#9999CC",
293                    "--height=125",
294                    "--upper-limit=656",
295                    "--lower-limit=0",
296                    "--rigid",
297                    "--base=1024",
298                    "DEF:tot_mem=target.rrd:mem:AVERAGE",
299                    "CDEF:tot_mem_cor=tot_mem,0,671744,LIMIT,UN,0,tot_mem,IF,1024,/",
300                    "CDEF:machine_mem=tot_mem,656,+,tot_mem,-",
301                    "COMMENT:Memory Consumption between $start_time",
302                    "COMMENT:    and $end_time                     ",
303                    "HRULE:656#000000:Maximum Available Memory - 656 MB",
304                    "AREA:machine_mem#CCFFFF:Memory Unused",
305                    "AREA:tot_mem_cor#6699CC:Total memory consumed in MB");
306        my $err=RRDs::error;
307        if ($err) {print "problem generating the graph: $err\n";}
308        # print the output
309        print "Average memory consumption is ";
310        printf "%5.2f",$tot_mem_ave/1024;
311        print " MB. Graphical representation can be found at /images/mem_$count.png.";
312

AUTHOR

314       Ketan Patel <k2pattu@yahoo.com>
315
316
317
3181.7.1                             2019-02-04                  RRD-BEGINNERS(1)
Impressum