Linux Server Monitoring with Munin

One problem with modern continuous integration systems (i.e. Hudson, Jenkins, et al.) is the maintenance and observation of all the build agents. Normally it is a tedious task to get all the needed information from all agents (Hard disk space, swapping activities, network traffic, etc.). One possibility to collect all the information is the installation of a resource monitoring tool that can help analyze resource trends and problems.

Munin

Munin is a tool for collecting performance data from different servers and displaying it in a web interface. It has a master/node architecture. The nodes implement an interface to allow cyclically collecting data from the nodes. The master polls this data at regular intervals, stores the data in RRD (Round Robin Database, http://oss.oetiker.ch/rrdtool/), and updates the graphs.

Overview

The Development of Munin started in 2002 under the name LRRD.
Version 1.0 of the tool was released in 2004. The tool is written in perl and released under the GNU Public License v2. One of the main design goals was the ease of creating plugins.

Munin on Ubuntu

The installation of munin on Ubuntu (Version 12.04) is very simple.
The node can be installed with:

apt-get install munin-node

and the master with:

apt-get install munin-server

Before we can get started we have to configure the nodes and the master.

Configuring the master

The main configuration of the master can be found in munin.conf. The file is usually located in the directory /etc/munin. The programs using it are munin-update, munin-graph, munin-limits and munin-html. They are called from munin-cron, which is usually triggered every 5 minutes.
There are two sections you need to look at in the munin.conf file.
The first section is ‘Server Variables’. The variables are responsible for data, logfiles and html file location.

dbdir /var/lib/munin
htmldir /var/www/munin
logdir /var/log/munin

The htmldir has to be accessible by your webserver. It is the location where the output for the webserver is stored.

The second is the configuration of the nodes.

[testserver1]
address 192.168.0.180
use_node_name yes

[testserver2]
address 192.168.0.181
use_node_name yes

# more servers…

Every node visible on the web frontend has to be configured here.

Configuring the node

The configuration of the node is also simple. Edit the file

/etc/munin/munin-node.conf

In order to grant access from the munin-master you have to explicitly enable (i.e. allow) the ip-address of the master.

allow ^127\.0\.0\.1$

# add master address
allow ^192\.168\.0\.170$

# Which port to bind to;
port 4949

Remember to open port 4949 in iptables so the munin node can receive connections from the munin master.

The node can then be started with:

sudo service munin-node start

If everything is setup correctly you should see the munin output on your browser.

(Example from http://munin.ping.uio.no/)

Munin_1
Overview Tab:
All configured nodes are listed here. The yellow text shows a warning alert
and a red text stands for a critical alert.

Munin_21
The details of a node display the measurements per day and per week.
(resolution per day: 5 min)

Node Plugins

Munin offers many predefined plugins. These are located in the directory

/usr/share/munin/plugins.

In order to enable them you have to create a link from the directory /etc/munin/plugins/ to /usr/share/munin/plugins. Some useful plugins are already activated by default. (for example diskstats, memory, cpu, etc.)

You can test the output of plugins on the node with telnet:


user@testserver1:~$ telnet 127.0.0.1 4949
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
# munin node at testserver1
fetch df
_dev_sda1.value 2.79539497746116
_dev.value 0.000202161102182329
_run.value 0.104154611734753
_run_lock.value 0
_run_shm.value 0.0142898833663745
cgroup.value 0
.

Writing your own plugins

The munin framework is written in Perl. The plugins however can be written in any language as long as you implement the interface. Writing a plugin is quite simple. For demonstration purposes let us write a plugin that monitors the number of files in the directory ‘/test’ and store it under ‘testmunin.sh’

#!/bin/bash
case $1 in
config)
cat <<'EOM'
graph_title My First Munin Plugin
graph_vlabel myfiles
myfiles.label files
EOM
exit 0;;
esac
echo -n "myfiles.value"
ls /test |wc -l

Calling the script directly gives the following output:

./testmunin.sh config
graph_title My First Munin Plugin
graph_vlabel myfiles
myfiles.label files

And without parameter

./testmunin.sh
myfiles.value 4

Enabling the script for the node can be done with the following steps:
Create a link in /etc/munin/plugins

ln -s /pathToTheScript/testmunin.sh test

Restart the munin-node

sudo /etc/init.d/munin-node restart

and test the output of the node with telnet:

telnet 192.168.0.180 4949
Trying 192.168.0.180...
Connected to 192.168.0.180.
Escape character is '^]'.
# munin node at mypc
fetch test
myfiles.value 4

Now test the output on the master:

sudo munin-run test config
graph_title My First Munin Plugin
graph_vlabel myfiles
myfiles.label files

After a while you should see the plugin on the web server.

Munin_3

Setup of an alert mail

Munin allows sending alert mails for two alert levels. These are warning and critical. The setup of an alert mail is rather easy. As a prerequisite you need to have a MTA (Mail Transfer Agent) installed and running on the master (like Postfix).

First of all, we define the warning level for our example. Let us assume that more than 6 files in the directory should issue a critical alert.
On the master side edit /etc/munin/munin.conf and add the following line to define the warning level:

[testserver1]
address 192.168.0.180
use_node_name yes
test.myfiles.critical 6

The mail setup is also straight forward: Add following lines to /etc/munin/munin.conf.

contacts me
contact.me.command mail -s "Munin notification ${var:host}" myaccount@example.com

If the level (the number of files in /test) reaches the critical level we get following mail:

localdomain :: testserver1.localdomain :: My First Munin Plugin
CRITICALs: files is 7 (outside range [:6]).

Summary

Munin is a simple to install yet powerful tool to monitor all your servers via a web interface. Its simplicity in installation and its ease in creating custom made plugins make it an ideal tool. It is easy to write plugins in any language you are comfortable with.
In addition you can use other Munin-node implementations on embedded systems.

For more information have a look at:
OpenWRT
Muninlite

There is also a Windows node installation.

The official documentation can be found here: munin-monitoring.org/wiki

This entry was posted in Linux and tagged , , . Bookmark the permalink.

One Response to Linux Server Monitoring with Munin

  1. Casey says:

    Munin is one of the best tool for monitoring server. Its very useful among server admins and net work engineers.

Leave a Reply

Your email address will not be published. Required fields are marked *