Load-balancing and automatic failover mechanisms are essential components of a high-availability system. Equally important is efficient detection of failing services and servers that may not fall under the auspices of the failover protocol. This may include monitoring the size of log files, the amount of disk space available, memory race conditions, and the availaibility of secondary services.
We use monit at WormBase to handle these tasks and more.
Monit is simple to install.
cd ~/build tar xzf ../src/monit*.tar.gz cd monit* ./configure make sudo make install
monit uses a fun free-text configuration file-format. Multiple instances of monit can be launched, each pointing to its own configuration file.