Difference between revisions of "Website:Monitoring services"
(→nagios) |
|||
Line 112: | Line 112: | ||
− | + | Create a wrapper script to let the spawn-fcgi binary run a fcgiwrap instance and make it executable. | |
+ | |||
+ | sudo emacs /usr/bin/c-fcgi.sh | ||
+ | #!/bin/sh | ||
+ | /usr/bin/spawn-fcgi -f /usr/bin/fcgiwrap -a 127.0.0.1 -p 49233 -P /var/run/fastcgi-c.pid -u www-data -g www-data | ||
+ | |||
+ | Create an init script... | ||
+ | |||
+ | #!/bin/bash | ||
+ | C_SCRIPT=/usr/bin/c-fcgi | ||
+ | RETVAL=0 | ||
+ | case "$1" in | ||
+ | start) | ||
+ | echo "Starting fastcgi" | ||
+ | $C_SCRIPT | ||
+ | RETVAL=$? | ||
+ | ;; | ||
+ | stop) | ||
+ | echo "Stopping fastcgi" | ||
+ | killall -9 fcgiwrap | ||
+ | RETVAL=$? | ||
+ | ;; | ||
+ | restart) | ||
+ | echo "Restarting fastcgi" | ||
+ | killall -9 fcgiwrap | ||
+ | $C_SCRIPT | ||
+ | RETVAL=$? | ||
+ | ;; | ||
+ | *) | ||
+ | echo "Usage: c-fastcgi {start|stop|restart}" | ||
+ | exit 1 | ||
+ | ;; | ||
+ | esac | ||
+ | exit $RETVAL | ||
+ | |||
+ | |||
+ | # Set it up to start automatically. | ||
+ | sudo update-rc.d c-fcgi defaults | ||
+ | sudo /etc/init.d/c-fcgi start | ||
= Installing monit = | = Installing monit = |
Revision as of 17:55, 21 December 2010
Contents
Overview
Load-balancing and automatic failover mechanisms are essential components of a high-availability system. Equally important is efficient detection of failing services and servers that may not fall under the auspices of the failover protocol. This may include monitoring the size of log files, the amount of disk space available, memory race conditions, and the availaibility of secondary services.
Install nagios
sudo adduser nagios sudo adduser nagios sudo groupadd nagioscmd sudo usermod -G nagioscmd nagios sudo usermod -G nagioscmd nobody cd /usr/local/wormbase/build mkdir nagios ; cd nagios wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.3.tar.gz tar xzf nagios-3.2.3.tar.gz cd nagios-3.2.3 cd /usr/local/wormbase/services mkdir nagios-3.2.3 sudo chown -R nagios:nagios nagios-3.2.3 ln -s nagios-3.2.3 nagios cd build/nagios/nagios-3.2.3 ./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \ --with-nagios-user=nagios \ --with-nagios-group=nagios \ --with-command-group=nagioscmd make all sudo make install sudo make install-init sudo make install-commandmode sudo make install-config
cd /usr/local/wormbase/build/nagios wet http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz tar xzf nagios-plugins* cd nagios-plugins-1.4.15 ./configure --prefix=/usr/local/wormbase/services/nagios-3.2.3 \ --with-nagios-user=nagios \ --with-nagios-group=nagios make sudo make install
Configure nagios
In /usr/local/wormbase/services/nagios/etc/cgi.cfg you should turn use_authentication on (it should be by default):
use_authentication=1
Add authentication:
htpasswd -c /usr/local/wormbase/services/nginx/conf/htpasswd todd
Configure nginx
Add the following stanza to /usr/local/wormbase/services/nginx/
############################################# # # nagios:8080 # ############################################# server { listen 8080; server_name dev.wormbase.org; access_log /usr/local/wormbase/logs/nagios-acccess.log; location / { root /usr/local/wormbase/services/nagios/share; index index.html; rewrite ^/nagios/images/(.*)\.png /images/$1.png break; auth_basic "Restricted"; auth_basic_user_file conf/htpasswd; } location ~ \.cgi$ { root /usr/local/wormbase/services/nagios/sbin; rewrite ^/nagios/cgi-bin/(.*)\.cgi /$1.cgi break; include /usr/local/wormbase/services/nginx/conf/fastcgi_params; fastcgi_pass 127.0.0.1:49233; fastcgi_param SCRIPT_FILENAME /usr/local/nagios$fastcgi_script_name; # same path as above auth_basic "Restricted"; auth_basic_user_file conf/htpasswd; fastcgi_param AUTH_USER $remote_user; fastcgi_param REMOTE_USER $remote_user; } }
Build spawn-fcgi
cd /usr/local/wormbase/build wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2 tar xvjf lighttpd-1.4.18.tar.bz2 cd lighttpd-1.4.18/ ./configure make sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi
And the fcgi wrapper ...
cd /usr/local/wormbase/build wget http://www.fastcgi.com/dist/fcgi.tar.gz tar xzf fcgi-2.4.0.tar.gz cd fcgi-2.4.0 ./configure make sudo make install
wget http://github.com/gnosek/fcgiwrap/tarball/master tar xzf gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec.tar.gz cd gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec/ sudo apt-get install autoconf autotools-dev autoreconf -i ./configure make sudo mv fcgiwrap /usr/bin/ sudo chown root:root /usr/bin/fcgiwrap
Create a wrapper script to let the spawn-fcgi binary run a fcgiwrap instance and make it executable.
sudo emacs /usr/bin/c-fcgi.sh #!/bin/sh /usr/bin/spawn-fcgi -f /usr/bin/fcgiwrap -a 127.0.0.1 -p 49233 -P /var/run/fastcgi-c.pid -u www-data -g www-data
Create an init script...
#!/bin/bash C_SCRIPT=/usr/bin/c-fcgi RETVAL=0 case "$1" in
start) echo "Starting fastcgi" $C_SCRIPT RETVAL=$?
;;
stop) echo "Stopping fastcgi" killall -9 fcgiwrap RETVAL=$?
;;
restart) echo "Restarting fastcgi" killall -9 fcgiwrap $C_SCRIPT RETVAL=$?
;;
*) echo "Usage: c-fastcgi {start|stop|restart}" exit 1
;; esac exit $RETVAL
# Set it up to start automatically. sudo update-rc.d c-fcgi defaults sudo /etc/init.d/c-fcgi start
Installing monit
Monit is simple to install.
todd> cd ~/build todd> tar xzf ../src/monit*.tar.gz todd> cd monit* todd> ./configure todd> make todd> sudo make install
Configuring monit
monit uses a fun free-text configuration file-format. Multiple instances of monit can be launched, each pointing to its own configuration file. By fun, I mean that it is much more fun that writing init scripts.
monit configuration files live at:
/usr/local/wb/admin/conf/monit
Test the monit configuration file
todd> monit -t -c /path/to/monitrc
Starting and stopping monit
Start monit by:
todd> monit -c /path/to/monitrc
Stop monit by:
todd> monit -c /path/to/monitrc quit
Configuring the system to run monit under init
We configure monit to run under init. To do this edit /etc/inittab with entries for each of the WormBase monitrc files:
todd> sudo nano /etc/inittab
// Add the following line as an example mo:2345:respawn:/usr/local/bin/monit -Ic /home/todd/monitrc
Force init to re-read the configuration file by
todd> sudo /sbin/telinit q
What WormBase monitors
logs
Nothing can bring down a server faster than logs that eat up all available disk space or that grow to behemoth proportions. This is particularly true of the sgifaceserver log serverlog.wrm, a file that grows so fast it makes my head spin. If this file hits 2GB in size, sgifaceserver will crash.
sgifaceserver
blat
The Squid Cache on fe.wormbase.org
The Cache Manager CGI
The Cache Manager CGI is only available from fe.wormbase.org:81. Note you must be "on localhost" in order to use this statistics viewer. You can access this CGI by tunneling port 81 traffic to fe.wormbase.org and then accessing the following URL:
http://localhost:81/squid/cachemgr.cgi
CacheHost  : fe.wormbase.org CachePort  : 80 Managername : [LSs canine] Password  : [none required]
RRD Tool / real-time graphical analysis
You can also monitor squid performance in real-time graphically using RRDTools. Again, these tools are located behind the firewall on port fe.wormbase.org:81.
1 day statistics: http://localhost:81/squid-monitor/1day.cgi
1 week statistics: http://localhost:81/squid-monitor/1week.cgi
httpd
Select IP addresses can monitor httpd at:
http://machine_to_monitor/status
I tend to disable this except when I am troubleshooting server load since it causes apache to unnecessarily keep track of lots of things.
Select IP addresses can also use my Apache::VMonitor script:
-- IN PROGRESS - it currently only works on apache2 --