The simplicity of rrdtool as a database

rrd is the sqlite of time series database it does two things well : update the archive (which cacti, nagios, munin > ...) knows fairly well how to update, but it also is an archive format.

When you are on a distant server, you may not have access to a web browser, nor do you want to install Xserver (or wayland) locally to see the data.

Here is your savior : pkg install gnuplot-lite. A version of gnuplot aiming at graphing plots on the terminal in ASCII.

First, let's assume we work with munin wich by convention put only one data source per archive, and name it « 42 ». It is a wise choice to avoid confusion and have the memory of where data source are stored at the price of having a lot of rrd files.

Hence, by convention our rrd are named host.group_plugin_name_of_datasource.rrd.

To extract the data store we need the points
LC_ALL=C rrdtool fetch  badass.home-cpu-system-d.rrd AVERAGE
will extract the points without LANG=fr changing your decimal separator.
If there is one command to remember for plotting with gnuplot: this is the one. If you remove the « : » you have the exact expected input from gnuplot for a plot.
$LC_ALL=C rrdtool fetch pierrot.home-cpu-system-d.rrd AVERAGE |\
    perl -ane 'm!^(\d+): ([\d\+\-\.e]+)$! and print "$1 $2\n";' \
    | head
1718192400 1.7250666667e+00
1718192700 1.6645333333e+00
1718193000 1.6012666667e+00
1718193300 1.6555333333e+00
1718193600 1.6632000000e+00
1718193900 1.8789333333e+00
1718194200 1.6220000000e+00
1718194500 1.7081333333e+00
1718194800 1.6512000000e+00
1718195100 1.7545333333e+00
Hence with gnuplot dumb output :
$LC_ALL=C rrdtool fetch pierrot.home-cpu-system-d.rrd  AVERAGE |\
	perl -ane 'm!^(\d+): ([\d\+\-\.e]+)$! and print "$1 $2\n";'|\
    gnuplot -p -e "set xdata time;set terminal dumb 60 40 ;
    	set timefmt \"%s\";set xtics rotate by 45 right; 
        set format x \"%m/%d/%Y %H:%M\"; 
        plot \"-\" using 1:2 with l lines \"-\""

                                                            
  10 +--------------------------------------------------+   
     |   +   +   +   +   +   +  +   +   +   +   +   +   |   
     |                            "-" using 1:2 +.....+ |   
     |                                                  |   
   9 |-+                           +                  +-|   
     |                             :                    |   
     |                             :                    |   
     |                             :                    |   
   8 |-+                           :                  +-|   
     |                             :                    |   
     |                             :                    |   
     |                             :                    |   
   7 |-+                           :                  +-|   
     |                             :                    |   
     |                             :                    |   
     |                             :           +        |   
   6 |-+                           :           :      +-|   
     |                             :           :        |   
     |                             :           :        |   
     |                            ::           :        |   
   5 |-+                          ::           :      +-|   
     |                            ::           :        |   
     |                            ::           :        |   
     |                            ::           :        |   
   4 |-+                          ::+          ::     +-|   
     |                            ::+          ::       |   
     |                            :::          ::       |   
     |                            :::          ::       |   
   3 |-+                          :::          ::     +-|   
     |                            :::         :::       |   
     |                            :+:         +::       |   
     |                            :::         :++ +  +  |   
   2 |-+ ++++++  +++      +++  + +:::++ + + + ::: +++:+-|   
     |  ++++++++++++++++++++++++++++++++++++++++++:++   |   
     |   +    +  ++ ++++ ++ +    +    +  ++ +     ++    |   
     |   +   +   +   +   +   +  +   +   +   +   +   +   |   
   1 +--------------------------------------------------+   
 06/106/106/106/06/106/106/106/106/106/106/13/2024 12:00    
                                                 
                                               
This means, we can now write a script that can both be used as a terminal plotter of rrd archive, but also as a gallery viewer : It seems fun and all, but what about looking at the value of two distinct data source on the same graph to see if they are correlated.

Let's say I want to compare my usage CPU and the sum of my network interfaces as a ratio of their max to see if they are related ? How complex is it ?

Well actually rrdtool export (man rrdexport) provides a tool for exactly this.

It's a FORTH based pipeline operator on views of time series here is an example.

Let's say we want to compare our CPU usage and our incoming + outgoing network trafic. It needs 3 operations :
  • one additions of the network output and input
  • two normalization by 1 so that we can compare on the same graph
LC_ALL=C rrdtool xport -t  -e now \
    DEF:x=pierrot.home-ibm_acpi-CPU-g.rrd:42:AVERAGE  \ # we define x as average of CPU temp
    VDEF:y=x,MAXIMUM \                                  # y = max(x) (FORTH X,MAXIMUM)
    VDEF:l=x,MINIMUM \									# l=min(x)
    CDEF:z=x,l,-,y,l,-,/                                # z=(x-minx)/(maxx-minx)
    DEF:a=pierrot.home-if_wlan0-obytes-d.rrd:42:AVERAGE \ # a = incoming traffic
    DEF:b=pierrot.home-if_wlan0-rbytes-d.rrd:42:AVERAGE \ # b = receiveid traffic
    CDEF:c=a,b,+ \                                      # CDEF for adding rank by rank operation in FORTH
    VDEF:d=c,MAXIMUM \                                  # VDEF for the whole series (ex percentile)
    CDEF:e=c,d,/ \                                      # normalization
    XPORT:z:ratio \                                     # export ratio z
    XPORT:e:ratio                                       # export ratio e
    
 
If you export it as json you just need to pipe jq to get a gnuplot default input :
 jq -j -c -r '.data[] |  .[0] , " " , .[1], " ", .[2] , "\n"' > temp.csv
 
You take the array defined at the key for data. The -t option of rrdtool xport makes sure data[][0] is the time and the following one, the one specified in the xport directive.

Then, the reminder is the gnuplot that gives the following result :
And now we can make a nice CLI tool to compare two rrd from the command line with output in ascii art Giving the following result
X=80 Y=40 ./hf.sh pierrot.home-ibm_acpi-CPU-g.rrd pierrot.home-hddtemp_smartctl-ada0-g.rrd 

                                                                                
    1 +---------------------------------------------------------------------+   
      |     +     +     +    +     +     +    ##     +     #   ##     +     |   
      |                             pierrot.ho##-ibm_acpi-C#U-g##rd +.....+ |   
      |                    pierrot.home-hddtem##smartctl-ad#0-g##rd ####### |   
      |                                       ##           #   ##           |   
      |                                      ###           #   ##           |   
      |                                      ###           #   ##           |   
  0.8 |-+                                    ###           #   ##         +-|   
      |                                      ###           #   ##  +        |   
      |                      ###############################   ##  :        |   
      |                      #                                 :   :        |   
      |    +                 #                                 +   :        |   
      |    :                 #                                 :   :        |   
      |    :                 #                                 :   :        |   
  0.6 |-+  :                #        +       +    +          + :   : +    +-|   
      |    :                #        :       :    :          : :   : :      |   
      |    :                #        :       :   +:     +    :++   : :      |   
      |    :        +       #        :     ++: + ::    +:  ++:::   + :      |   
      |    :        :    ## #        ::    ::: : ::    ::  :::::   :::      |   
      |    :        :    ## #       +::++++:+: :+++ +++:+++:::+ : +++:      |   
      |    :        :    ## #       ::::::::::::::: ::::::::::: : ::::      |   
      |    :        :    ## #       ::::::::::::::::::::::::::: : ::::      |   
  0.4 |-+  :        + +  ## #       :++++:++++++++++++:++++++++ +++:++++  +-|   
      |    :       :: : # ##        ::   :  : :   ::: : ::::::  ::::  ::    |   
      |    :       :: : # ##        ::   :  : :   ::  : ::::::  :::+  ::    |   
      |    :++++  +:+ + # ##        +:   +  + +   ++  + ++++:+  +:+   ++    |   
      |    :::::  ::::: # ##        ::                      :    :          |   
      |   ############### ##        :+                      +    :          |   
      |   +:+:++++##:++++++:        +                            :          |   
  0.2 |-+ ::::: ::##: :::   :       :                            :        +-|   
      |   ::::: + ##: + +   +       :                            +          |   
      |   :: ::   ##:   :   :       :                                       |   
      |   :: ::   # :   :   :       :                                       |   
      |   ++ ++   # +   +   :++++++:                                        |   
      |           #         ::::::::                                        |   
      |     +     #     +   :+::::::     +     +     +     +    +     +     |   
    0 +---------------------------------------------------------------------+   
  06/12/06/1206/12/06/12/06/13/06/13/06/13/06/1306/13/06/13/06/13/2024 12:00    
                                                                                

As you noticed in my example : rrdtool xport is giving you the basic of treating/transforming datasource in rpn, and graphing is just a few steps away from exporting and it gives the following results :
At the end of the day, there is a great frustration using rrd : it is an amazing under used, uncorrectly documented tool. Most of the power of rrd lies in obscure features such as xport that requires a better explanation of how the forth works, like for instance how we can build a custom aggregation function (recoding a moving median as an example would be nice). However, I still find it a fun tool to toy with.

No comments: