Difference between revisions of "Website:Monitoring services"
m (Protected "Website:Monitoring services" ([edit=sysop] (indefinite) [move=sysop] (indefinite))) |
|||
(3 intermediate revisions by the same user not shown) | |||
Line 161: | Line 161: | ||
sudo /etc/init.d/c-fcgi start | sudo /etc/init.d/c-fcgi start | ||
− | + | ==Install a fastcgi-enabled php package == | |
sudo apt-get install php5-cgi | sudo apt-get install php5-cgi | ||
Line 299: | Line 299: | ||
Set deafults: | Set deafults: | ||
− | sudo | + | sudo emacs /etc/defaults/php-fastcgi |
<pre> | <pre> | ||
Line 328: | Line 328: | ||
sudo update-rc.d php-fastcgi defaults | sudo update-rc.d php-fastcgi defaults | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
= What WormBase monitors = | = What WormBase monitors = |
Latest revision as of 19:29, 1 August 2013
Contents
Overview
Load-balancing and automatic failover mechanisms are essential components of a high-availability system. Equally important is efficient detection of failing services and servers that may not fall under the auspices of the failover protocol. This may include monitoring the size of log files, the amount of disk space available, memory race conditions, and the availaibility of secondary services.
Install nagios
sudo adduser nagios sudo adduser nagios sudo groupadd nagioscmd sudo usermod -G nagioscmd nagios sudo usermod -G nagioscmd nobody cd /usr/local/wormbase/build mkdir nagios ; cd nagios wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.3.tar.gz tar xzf nagios-3.2.3.tar.gz cd nagios-3.2.3 cd /usr/local/wormbase/services mkdir nagios-3.2.3 sudo chown -R nagios:nagios nagios-3.2.3 ln -s nagios-3.2.3 nagios cd build/nagios/nagios-3.2.3 ./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \ --with-nagios-user=nagios \ --with-nagios-group=nagios \ --with-command-group=nagioscmd make all sudo make install sudo make install-init sudo make install-commandmode sudo make install-config
cd /usr/local/wormbase/build/nagios wet http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz tar xzf nagios-plugins* cd nagios-plugins-1.4.15 ./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \ --with-nagios-user=nagios \ --with-nagios-group=nagios make sudo make install
Configure nagios
In /usr/local/wormbase/services/nagios/etc/cgi.cfg you should turn use_authentication on (it should be by default):
use_authentication=1
Add authentication:
htpasswd -c /usr/local/wormbase/services/nginx/conf/htpasswd todd
Configure nginx
Add the following stanza to /usr/local/wormbase/services/nginx/
############################################# # # nagios:8080 # ############################################# server { listen 8080; server_name dev.wormbase.org; access_log /usr/local/wormbase/logs/nagios-acccess.log; location / { root /usr/local/wormbase/services/nagios/share; index index.php; # rewrite ^/nagios/images/(.*)\.png /images/$1.png break; # auth_basic "Restricted"; # auth_basic_user_file conf/htpasswd; } # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # location ~ \.php$ { root /usr/local/wormbase/services/nagios/share; fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /usr/local/wormbase/services/nagios/share$fastcgi_script_name; include /usr/local/wormbase/services/nginx/conf/fastcgi_params; # include fastcgi_params; } location ~ \.cgi$ { root /usr/local/wormbase/services/nagios/sbin; rewrite ^/nagios/cgi-bin/(.*)\.cgi /$1.cgi break; include /usr/local/wormbase/services/nginx/conf/fastcgi_params; fastcgi_pass 127.0.0.1:49233; fastcgi_param SCRIPT_FILENAME /usr/local/wormbase/services/nagios/sbin$fastcgi_script_name; # same path as above # auth_basic "Restricted"; # auth_basic_user_file conf/htpasswd; fastcgi_param AUTH_USER $remote_user; fastcgi_param REMOTE_USER $remote_user; } }
Build spawn-fcgi
cd /usr/local/wormbase/build wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2 tar xvjf lighttpd-1.4.18.tar.bz2 cd lighttpd-1.4.18/ ./configure make sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi
And the fcgi wrapper ...
cd /usr/local/wormbase/build wget http://www.fastcgi.com/dist/fcgi.tar.gz tar xzf fcgi-2.4.0.tar.gz cd fcgi-2.4.0 ./configure make sudo make install
wget http://github.com/gnosek/fcgiwrap/tarball/master tar xzf gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec.tar.gz cd gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec/ sudo apt-get install autoconf autotools-dev autoreconf -i ./configure make sudo mv fcgiwrap /usr/bin/ sudo chown root:root /usr/bin/fcgiwrap
Add a wrapper script:
#!/bin/sh /usr/bin/spawn-fcgi -f /usr/bin/fcgiwrap -a 127.0.0.1 -p 49233 -P /var/run/fastcgi-c.pid -u www-data -g www-data
Add an init script at:
#!/bin/bash C_SCRIPT=/usr/bin/c-fcgi.sh RETVAL=0 case "$1" in start) echo "Starting fastcgi" $C_SCRIPT RETVAL=$? ;; stop) echo "Stopping fastcgi" killall -9 fcgiwrap RETVAL=$? ;; restart) echo "Restarting fastcgi" killall -9 fcgiwrap $C_SCRIPT RETVAL=$? ;; *) echo "Usage: c-fastcgi {start|stop|restart}" exit 1 ;; esac exit $RETVAL
Set FCGI to start up automatically:
sudo update-rc.d c-fcgi defaults sudo /etc/init.d/c-fcgi start
Install a fastcgi-enabled php package
sudo apt-get install php5-cgi
Append the following to: /etc/php5/cgi/php.ini:
/etc/php5/cgi/php.ini
Create a php5-fcgi init file
sudo emacs /etc/init.d/php-fastcgi
#! /bin/sh ### BEGIN INIT INFO # Provides: php-fastcgi # Required-Start: $all # Required-Stop: $all # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start and stop php-cgi in external FASTCGI mode # Description: Start and stop php-cgi in external FASTCGI mode ### END INIT INFO # Author: Kurt Zankl <kz@xon.uni.cc> # Do NOT "set -e" PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="php-cgi in external FASTCGI mode" NAME=php-fastcgi DAEMON=/usr/bin/php-cgi PIDFILE=/var/run/$NAME.pid SCRIPTNAME=/etc/init.d/$NAME # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 # Read configuration variable file if it is present [ -r /etc/default/$NAME ] && . /etc/default/$NAME # Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh # Define LSB log_* functions. # Depend on lsb-base (>= 3.0-6) to ensure that this file is present. . /lib/lsb/init-functions # If the daemon is not enabled, give the user a warning and then exit, # unless we are stopping the daemon if [ "$START" != "yes" -a "$1" != "stop" ]; then log_warning_msg "To enable $NAME, edit /etc/default/$NAME and set START=yes" exit 0 fi # Process configuration export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS DAEMON_ARGS="-q -b $FCGI_HOST:$FCGI_PORT" do_start() { # Return # 0 if daemon has been started # 1 if daemon was already running # 2 if daemon could not be started start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null || return 1 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --background --make-pidfile --chuid $EXEC_AS_USER --startas $DAEMON -- $DAEMON_ARGS || return 2 } do_stop() { # Return # 0 if daemon has been stopped # 1 if daemon was already stopped # 2 if daemon could not be stopped # other if a failure occurred start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE > /dev/null # --name $DAEMON RETVAL="$?" [ "$RETVAL" = 2 ] && return 2 # Wait for children to finish too if this is a daemon that forks # and if the daemon is only ever run from this initscript. # If the above conditions are not satisfied then add some other code # that waits for the process to drop all resources that could be # needed by services started subsequently. A last resort is to # sleep for some time. start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON [ "$?" = 2 ] && return 2 # Many daemons don't delete their pidfiles when they exit. rm -f $PIDFILE return "$RETVAL" } case "$1" in start) [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME" do_start case "$?" in 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;; 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;; esac ;; stop) [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME" do_stop case "$?" in 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;; 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;; esac ;; restart|force-reload) log_daemon_msg "Restarting $DESC" "$NAME" do_stop case "$?" in 0|1) do_start case "$?" in 0) log_end_msg 0 ;; 1) log_end_msg 1 ;; # Old process is still running *) log_end_msg 1 ;; # Failed to start esac ;; *) # Failed to stop log_end_msg 1 ;; esac ;; *) echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2 exit 3 ;; esac
Set deafults:
sudo emacs /etc/defaults/php-fastcgi
# # Settings for php-cgi in external FASTCGI Mode # # Should php-fastcgi run automatically on startup? (default: no) START=yes # Which user runs PHP? (default: www-data) EXEC_AS_USER=www-data # Host and TCP port for FASTCGI-Listener (default: localhost:9000) FCGI_HOST=localhost FCGI_PORT=9000 # Environment variables, which are processed by PHP PHP_FCGI_CHILDREN=5 PHP_FCGI_MAX_REQUESTS=1000
To have it start at launch:
sudo update-rc.d php-fastcgi defaults
What WormBase monitors
logs
Nothing can bring down a server faster than logs that eat up all available disk space or that grow to behemoth proportions. This is particularly true of the sgifaceserver log serverlog.wrm, a file that grows so fast it makes my head spin. If this file hits 2GB in size, sgifaceserver will crash.
sgifaceserver
blat
The Squid Cache on fe.wormbase.org
The Cache Manager CGI
The Cache Manager CGI is only available from fe.wormbase.org:81. Note you must be "on localhost" in order to use this statistics viewer. You can access this CGI by tunneling port 81 traffic to fe.wormbase.org and then accessing the following URL:
http://localhost:81/squid/cachemgr.cgi
CacheHost  : fe.wormbase.org CachePort  : 80 Managername : [LSs canine] Password  : [none required]
RRD Tool / real-time graphical analysis
You can also monitor squid performance in real-time graphically using RRDTools. Again, these tools are located behind the firewall on port fe.wormbase.org:81.
1 day statistics: http://localhost:81/squid-monitor/1day.cgi
1 week statistics: http://localhost:81/squid-monitor/1week.cgi
httpd
Select IP addresses can monitor httpd at:
http://machine_to_monitor/status
I tend to disable this except when I am troubleshooting server load since it causes apache to unnecessarily keep track of lots of things.
Select IP addresses can also use my Apache::VMonitor script:
-- IN PROGRESS - it currently only works on apache2 --