Website:Monitoring services

From WormBaseWiki
Jump to navigationJump to search

Overview

Load-balancing and automatic failover mechanisms are essential components of a high-availability system. Equally important is efficient detection of failing services and servers that may not fall under the auspices of the failover protocol. This may include monitoring the size of log files, the amount of disk space available, memory race conditions, and the availaibility of secondary services.

Install nagios

sudo adduser nagios
sudo adduser nagios
sudo groupadd nagioscmd
sudo usermod -G nagioscmd nagios
sudo usermod -G nagioscmd nobody
cd /usr/local/wormbase/build
mkdir nagios ; cd nagios
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.3.tar.gz
tar xzf nagios-3.2.3.tar.gz 
cd nagios-3.2.3
cd /usr/local/wormbase/services
mkdir nagios-3.2.3
sudo chown -R nagios:nagios nagios-3.2.3
ln -s nagios-3.2.3 nagios
cd build/nagios/nagios-3.2.3
./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \
       --with-nagios-user=nagios \
       --with-nagios-group=nagios \
       --with-command-group=nagioscmd
make all
sudo make install
sudo make install-init
sudo make install-commandmode
sudo make install-config
cd /usr/local/wormbase/build/nagios
wet http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz
tar xzf nagios-plugins*
cd nagios-plugins-1.4.15
 ./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \
       --with-nagios-user=nagios \
       --with-nagios-group=nagios
make
sudo make install

Configure nagios

In /usr/local/wormbase/services/nagios/etc/cgi.cfg you should turn use_authentication on (it should be by default):
use_authentication=1

Add authentication:

htpasswd -c /usr/local/wormbase/services/nginx/conf/htpasswd todd

Configure nginx

Add the following stanza to /usr/local/wormbase/services/nginx/

#############################################                                                                                     
#                                                                                                                                 
#  nagios:8080                                                                                                                    
#                                                                                                                                 
#############################################                                                                                     
server {
      listen   8080;
      server_name  dev.wormbase.org;
      access_log  /usr/local/wormbase/logs/nagios-acccess.log;
      location / {
            root   /usr/local/wormbase/services/nagios/share;
            index index.php;                                                                                                                                                                             
#            rewrite      ^/nagios/images/(.*)\.png /images/$1.png break;                                                                                                                                                 
#             auth_basic          "Restricted";                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
#             auth_basic_user_file      conf/htpasswd;                                                                                                                                                                    
       }
       # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000                                                                                                                                              
       #                                                                                                                                                                                                                 
       location ~ \.php$ {                                                                                                                                                                                               
            root   /usr/local/wormbase/services/nagios/share;
           fastcgi_pass   127.0.0.1:9000;
           fastcgi_index  index.php;
           fastcgi_param  SCRIPT_FILENAME  /usr/local/wormbase/services/nagios/share$fastcgi_script_name;
            include /usr/local/wormbase/services/nginx/conf/fastcgi_params;
#            include        fastcgi_params;                                                                                                                                                                               
       }
      location ~ \.cgi$ {
            root       /usr/local/wormbase/services/nagios/sbin;
            rewrite    ^/nagios/cgi-bin/(.*)\.cgi /$1.cgi break;
            include /usr/local/wormbase/services/nginx/conf/fastcgi_params;
            fastcgi_pass   127.0.0.1:49233;
            fastcgi_param  SCRIPT_FILENAME  /usr/local/wormbase/services/nagios/sbin$fastcgi_script_name;  # same path as above                                                                                               
#            auth_basic         "Restricted";                                                                                                                                                                             
#            auth_basic_user_file       conf/htpasswd;                                                                                                                                                                    
            fastcgi_param  AUTH_USER     $remote_user;
            fastcgi_param  REMOTE_USER   $remote_user;
      }
}

Build spawn-fcgi

cd /usr/local/wormbase/build
wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
tar xvjf lighttpd-1.4.18.tar.bz2
cd lighttpd-1.4.18/
./configure
make
sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi

And the fcgi wrapper ...

cd /usr/local/wormbase/build
wget http://www.fastcgi.com/dist/fcgi.tar.gz
tar xzf fcgi-2.4.0.tar.gz 
cd fcgi-2.4.0
./configure
make
sudo make install
wget http://github.com/gnosek/fcgiwrap/tarball/master
tar xzf gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec.tar.gz
cd gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec/
sudo apt-get install autoconf autotools-dev
autoreconf -i
./configure
make
sudo mv fcgiwrap /usr/bin/
sudo chown root:root /usr/bin/fcgiwrap


Add a wrapper script:

#!/bin/sh                                                                                                                                                                                                                 
/usr/bin/spawn-fcgi -f /usr/bin/fcgiwrap -a 127.0.0.1 -p 49233 -P /var/run/fastcgi-c.pid -u www-data -g www-data

Add an init script at:

 #!/bin/bash                                                                                                                                                                                                               
 C_SCRIPT=/usr/bin/c-fcgi.sh
 RETVAL=0
 case "$1" in
     start)
        echo "Starting fastcgi"
             $C_SCRIPT
                RETVAL=$?
  ;;
  stop)
        echo "Stopping fastcgi"
             killall -9 fcgiwrap
                     RETVAL=$?
  ;;
  restart)
        echo "Restarting fastcgi"
             killall -9 fcgiwrap
                     $C_SCRIPT
                        RETVAL=$?
  ;;
  *)
        echo "Usage: c-fastcgi {start|stop|restart}"
             exit 1
  ;;
esac
exit $RETVAL

Set FCGI to start up automatically:

sudo update-rc.d c-fcgi defaults
sudo /etc/init.d/c-fcgi start

Install a fastcgi-enabled php package

sudo apt-get install php5-cgi

Append the following to: /etc/php5/cgi/php.ini:

/etc/php5/cgi/php.ini

Create a php5-fcgi init file

sudo emacs /etc/init.d/php-fastcgi
#! /bin/sh
### BEGIN INIT INFO
# Provides:          php-fastcgi
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start and stop php-cgi in external FASTCGI mode
# Description:       Start and stop php-cgi in external FASTCGI mode
### END INIT INFO

# Author: Kurt Zankl <kz@xon.uni.cc>

# Do NOT "set -e"

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="php-cgi in external FASTCGI mode"
NAME=php-fastcgi
DAEMON=/usr/bin/php-cgi
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

# If the daemon is not enabled, give the user a warning and then exit,
# unless we are stopping the daemon
if [ "$START" != "yes" -a "$1" != "stop" ]; then
        log_warning_msg "To enable $NAME, edit /etc/default/$NAME and set START=yes"
        exit 0
fi

# Process configuration
export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS
DAEMON_ARGS="-q -b $FCGI_HOST:$FCGI_PORT"


do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --background --make-pidfile --chuid $EXEC_AS_USER --startas $DAEMON -- $DAEMON_ARGS || return 2
}

do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE > /dev/null # --name $DAEMON
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}

case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  restart|force-reload)
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  *)
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac


Set deafults:

sudo emacs /etc/defaults/php-fastcgi

#
# Settings for php-cgi in external FASTCGI Mode
#

# Should php-fastcgi run automatically on startup? (default: no)

START=yes

# Which user runs PHP? (default: www-data)

EXEC_AS_USER=www-data

# Host and TCP port for FASTCGI-Listener (default: localhost:9000)

FCGI_HOST=localhost
FCGI_PORT=9000

# Environment variables, which are processed by PHP

PHP_FCGI_CHILDREN=5
PHP_FCGI_MAX_REQUESTS=1000

To have it start at launch:

sudo update-rc.d php-fastcgi defaults

What WormBase monitors

logs

Nothing can bring down a server faster than logs that eat up all available disk space or that grow to behemoth proportions. This is particularly true of the sgifaceserver log serverlog.wrm, a file that grows so fast it makes my head spin. If this file hits 2GB in size, sgifaceserver will crash.

sgifaceserver

blat

The Squid Cache on fe.wormbase.org

The Cache Manager CGI

The Cache Manager CGI is only available from fe.wormbase.org:81. Note you must be "on localhost" in order to use this statistics viewer. You can access this CGI by tunneling port 81 traffic to fe.wormbase.org and then accessing the following URL:

http://localhost:81/squid/cachemgr.cgi
CacheHost   : fe.wormbase.org
CachePort   : 80
Managername : [LSs canine]
Password    : [none required]

RRD Tool / real-time graphical analysis

You can also monitor squid performance in real-time graphically using RRDTools. Again, these tools are located behind the firewall on port fe.wormbase.org:81.

1 day statistics:
http://localhost:81/squid-monitor/1day.cgi
1 week statistics:
http://localhost:81/squid-monitor/1week.cgi


httpd

Select IP addresses can monitor httpd at:

  http://machine_to_monitor/status

I tend to disable this except when I am troubleshooting server load since it causes apache to unnecessarily keep track of lots of things.


Select IP addresses can also use my Apache::VMonitor script:

  -- IN PROGRESS - it currently only works on apache2 --