Difference between revisions of "Website:Monitoring services"

From WormBaseWiki
Jump to navigationJump to search
Line 148: Line 148:
  
  
# Set it up to start automatically.
+
# Set it up to start automatically.
 
  sudo update-rc.d c-fcgi defaults
 
  sudo update-rc.d c-fcgi defaults
 
  sudo /etc/init.d/c-fcgi start
 
  sudo /etc/init.d/c-fcgi start

Revision as of 17:55, 21 December 2010

Overview

Load-balancing and automatic failover mechanisms are essential components of a high-availability system. Equally important is efficient detection of failing services and servers that may not fall under the auspices of the failover protocol. This may include monitoring the size of log files, the amount of disk space available, memory race conditions, and the availaibility of secondary services.

Install nagios

sudo adduser nagios
sudo adduser nagios
sudo groupadd nagioscmd
sudo usermod -G nagioscmd nagios
sudo usermod -G nagioscmd nobody
cd /usr/local/wormbase/build
mkdir nagios ; cd nagios
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.3.tar.gz
tar xzf nagios-3.2.3.tar.gz 
cd nagios-3.2.3
cd /usr/local/wormbase/services
mkdir nagios-3.2.3
sudo chown -R nagios:nagios nagios-3.2.3
ln -s nagios-3.2.3 nagios
cd build/nagios/nagios-3.2.3
./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \
       --with-nagios-user=nagios \
       --with-nagios-group=nagios \
       --with-command-group=nagioscmd
make all
sudo make install
sudo make install-init
sudo make install-commandmode
sudo make install-config
cd /usr/local/wormbase/build/nagios
wet http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz
tar xzf nagios-plugins*
cd nagios-plugins-1.4.15
 ./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \
       --with-nagios-user=nagios \
       --with-nagios-group=nagios
make
sudo make install

Configure nagios

In /usr/local/wormbase/services/nagios/etc/cgi.cfg you should turn use_authentication on (it should be by default):
use_authentication=1

Add authentication:

htpasswd -c /usr/local/wormbase/services/nginx/conf/htpasswd todd

Configure nginx

Add the following stanza to /usr/local/wormbase/services/nginx/

#############################################                                                                                     
#                                                                                                                                 
#  nagios:8080                                                                                                                    
#                                                                                                                                 
#############################################                                                                                     
server {
      listen   8080;
      server_name  dev.wormbase.org;
      access_log  /usr/local/wormbase/logs/nagios-acccess.log;
      location / {
            root   /usr/local/wormbase/services/nagios/share;
            index  index.html;
            rewrite      ^/nagios/images/(.*)\.png /images/$1.png break;
            auth_basic   "Restricted";
            auth_basic_user_file       conf/htpasswd;
       }
      location ~ \.cgi$ {
            root       /usr/local/wormbase/services/nagios/sbin;
            rewrite    ^/nagios/cgi-bin/(.*)\.cgi /$1.cgi break;
            include /usr/local/wormbase/services/nginx/conf/fastcgi_params;
            fastcgi_pass   127.0.0.1:49233;
            fastcgi_param  SCRIPT_FILENAME  /usr/local/nagios$fastcgi_script_name;  # same path as above                         
            auth_basic         "Restricted";
            auth_basic_user_file       conf/htpasswd;
            fastcgi_param  AUTH_USER          $remote_user;
            fastcgi_param  REMOTE_USER   $remote_user;
      }
}

Build spawn-fcgi

cd /usr/local/wormbase/build
wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
tar xvjf lighttpd-1.4.18.tar.bz2
cd lighttpd-1.4.18/
./configure
make
sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi

And the fcgi wrapper ...

cd /usr/local/wormbase/build
wget http://www.fastcgi.com/dist/fcgi.tar.gz
tar xzf fcgi-2.4.0.tar.gz 
cd fcgi-2.4.0
./configure
make
sudo make install
wget http://github.com/gnosek/fcgiwrap/tarball/master
tar xzf gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec.tar.gz
cd gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec/
sudo apt-get install autoconf autotools-dev
autoreconf -i
./configure
make
sudo mv fcgiwrap /usr/bin/
sudo chown root:root /usr/bin/fcgiwrap


Create a wrapper script to let the spawn-fcgi binary run a fcgiwrap instance and make it executable.

sudo emacs /usr/bin/c-fcgi.sh
 #!/bin/sh
/usr/bin/spawn-fcgi -f /usr/bin/fcgiwrap -a 127.0.0.1 -p 49233 -P /var/run/fastcgi-c.pid -u www-data -g www-data

Create an init script...

#!/bin/bash
C_SCRIPT=/usr/bin/c-fcgi 
RETVAL=0
case "$1" in

start) echo "Starting fastcgi" $C_SCRIPT RETVAL=$?

 ;;

stop) echo "Stopping fastcgi" killall -9 fcgiwrap RETVAL=$?

 ;;

restart) echo "Restarting fastcgi" killall -9 fcgiwrap $C_SCRIPT RETVAL=$?

 ;;

*) echo "Usage: c-fastcgi {start|stop|restart}" exit 1

 ;;
esac
exit $RETVAL


  1. Set it up to start automatically.
sudo update-rc.d c-fcgi defaults
sudo /etc/init.d/c-fcgi start

Installing monit

Monit is simple to install.

todd> cd ~/build
todd> tar xzf ../src/monit*.tar.gz
todd> cd monit*
todd> ./configure
todd> make
todd> sudo make install

Configuring monit

monit uses a fun free-text configuration file-format. Multiple instances of monit can be launched, each pointing to its own configuration file. By fun, I mean that it is much more fun that writing init scripts.

monit configuration files live at:

/usr/local/wb/admin/conf/monit

Test the monit configuration file

todd> monit -t -c /path/to/monitrc

Starting and stopping monit

Start monit by:

todd> monit -c /path/to/monitrc

Stop monit by:

todd> monit -c /path/to/monitrc quit

Configuring the system to run monit under init

We configure monit to run under init. To do this edit /etc/inittab with entries for each of the WormBase monitrc files:

 todd> sudo nano /etc/inittab
 // Add the following line as an example
mo:2345:respawn:/usr/local/bin/monit -Ic /home/todd/monitrc

Force init to re-read the configuration file by

todd> sudo /sbin/telinit q

What WormBase monitors

logs

Nothing can bring down a server faster than logs that eat up all available disk space or that grow to behemoth proportions. This is particularly true of the sgifaceserver log serverlog.wrm, a file that grows so fast it makes my head spin. If this file hits 2GB in size, sgifaceserver will crash.

sgifaceserver

blat

The Squid Cache on fe.wormbase.org

The Cache Manager CGI

The Cache Manager CGI is only available from fe.wormbase.org:81. Note you must be "on localhost" in order to use this statistics viewer. You can access this CGI by tunneling port 81 traffic to fe.wormbase.org and then accessing the following URL:

http://localhost:81/squid/cachemgr.cgi
CacheHost   : fe.wormbase.org
CachePort   : 80
Managername : [LSs canine]
Password    : [none required]

RRD Tool / real-time graphical analysis

You can also monitor squid performance in real-time graphically using RRDTools. Again, these tools are located behind the firewall on port fe.wormbase.org:81.

1 day statistics:
http://localhost:81/squid-monitor/1day.cgi
1 week statistics:
http://localhost:81/squid-monitor/1week.cgi


httpd

Select IP addresses can monitor httpd at:

  http://machine_to_monitor/status

I tend to disable this except when I am troubleshooting server load since it causes apache to unnecessarily keep track of lots of things.


Select IP addresses can also use my Apache::VMonitor script:

  -- IN PROGRESS - it currently only works on apache2 --