GBrowse Administration

Overview

GBrowse is one of the central -- and most heavily used -- components of the WormBase site. To meet this demand, we use a number of dedicated servers using GBrowse's built in master/slave configuration. This document describes how to build and configure these servers.

Hardware Topology

The GBrowse topology looks like this:

 Master
 ------------- -----------------------------
 |    gb1    |              \               \
 |           |        -------------    -------------   
 |* * * * * *| <----- |    gb2    |    |    gb2    |   Slaves
 -------------        -------------    -------------  

Where:

 * * * * * = relational databases running under mysql on gb1.

Node Configuration

Create groups

Both master and slave servers.

sudo addgroup wormbase
sudo addgroup gbrowse
# Add yourself to the groups
sudo usermod -a -G gbrowse, wormbase tharris

Create Directories

Both master and slave servers.

Create a directory for GBrowse. For now, we're maintaining GBrowse on dedicated hardware completely distinct from the rest of the web application. This makes updates significantly easier, although it increases maintenance (ie of boilerplate). We'll also keep gbrowse inside a wormbase/ directory should we wish to install other components on these servers at a later date.

Let $ROOT = /usr/local/wormbase

$ sudo mkdir -p ${ROOT}
$ sudo chgrp -R wormbase $ROOT
$ sudo chmod 2775 $ROOT

Debian preparation

Both master and slave servers.

$ sudo apt-get update
$ sudo apt-get install gcc

Install Perl 5.10.1 or greater

Both master and slave servers.

You'll need Perl version 5.10.1 or greater.

Debian:

$ sudo apt-get install perl

NOTE: This version of Perl on Debian Lenny doesn't ship with perl.h. You won't be able to build mod_perl without it!

Or from source:

bash> mkdir ~/website/perl
bash> cd ~/src
bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz
bash> cd ~/build
bash> tar xzf ../src.perl-5.10.1.tar.gz
bash> cd perl-5.10.1
bash> ./Configure -des
  // Or, to install in a local path:
bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1
bash> make
bash> make test
bash> make install

Be sure to set up your your system to preferentially use this newly installed Perl:

bash> export PATH=~/wormbase/perl/5.10.1/bin:${PATH}

Install Apache2 and mod_perl

Master server only - the slaves do not need apache/mod_perl

First, if you're running Debian, get rid of the installed apache. The default layout is just plain annoying.

$ sudo apt-get remove apache2

Assuming you have already fetched the source into ~/src:

# Build httpd 2.2.11
cd ~/src
tar xzf httpd-2.2.11.tar.gz
cd httpd-2.2.11
./configure --enable-mods-shared=all
make
sudo make install

# Install mod_perl 2-0.4
$ perl -MCPAN -e 'CPAN::install(mod_perl2)'

Fetch the appropriate values for @INC:
$ cd /usr/local/wormbase/gbrowse-current/extlib
$ perl -Mlocal::lib=./

cd ~/src
cd mod_perl-2.0.4
perl -I/usr/local/wormbase/website-classic/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/website-classic/extlib
// When prompted for the apxs path, enter: /usr/local/apache2/bin/apxs
make
sudo make install

If make fails, you may need to symlink libperl.so -> libperl.so.5.10.0 in /usr/lib

Configuration

• Set httpd to listen on your desired port. In the examples below, we assume port 8080.

# Unnecessary if running on the default port.
$ sudo perl -p -i -e's|Listen 80|Listen 8080|' /usr/local/apache2/conf/httpd.conf

Set up a virtual host on your port. Adjust PerlSwitches as appropriate.

<VirtualHost *:80>
    ErrorLog "/usr/local/wormbase/logs/error_log"
    CustomLog "/usr/local/wormbase/logs/access_log" common
  
# Support files: the wormbase css file for gbrowse, banner image, etc
Alias       "/support/     "/usr/local/wormbase/support-files/"

#  Include /usr/local/wormbase/gbrowse-current/conf/httpd.conf

Alias        "/gbrowse2/i/" "/usr/local/wormbase/gbrowse-current/tmp/images/"
Alias        "/gbrowse2"    "/usr/local/wormbase/gbrowse-current/html"
ScriptAlias  "/gb2"      "/usr/local/wormbase/gbrowse-current/cgi/gb2"

<Directory "/usr/local/wormbase/support-files">
  Options -Indexes -MultiViews +FollowSymLinks
  Order allow,deny
  Allow from all
</Directory>

<Directory "/usr/local/wormbase/gbrowse-current/tmp">
Allow from all
</Directory>

<Directory "/usr/local/wormbase/gbrowse-current/html">
  Options -Indexes -MultiViews +FollowSymLinks
  Order allow,deny                                              
  Allow from all
</Directory>

<Directory "/usr/local/wormbase/gbrowse-current/cgi/gb2">
  SetEnv PERL5LIB "/usr/local/wormbase/gbrowse-2.01/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/gbrowse-2.01/extlib/lib/pe$
  SetEnv GBROWSE_CONF   "/usr/local/wormbase/gbrowse-current/conf"
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

<IfModule mod_fcgid.c>
  Alias /fgb2 "/usr/local/wormbase/gbrowse-current/cgi/gb2"
  <Location /fgb2>
    SetHandler   fcgid-script
    Options      ExecCGI
  </Location>
  DefaultInitEnv GBROWSE_CONF /usr/local/wormbase/gbrowse-current/conf
  DefaultInitEnv PERL5LIB /usr/local/wormbase/gbrowse-2.01/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/gbrowse-2.01/extlib$
</IfModule>

<IfModule mod_fastcgi.c>
  Alias /fgb2 "/usr/local/wormbase/gbrowse-current/cgi/gb2"
  <Location /fgb2>
    SetHandler   fastcgi-script
    Options      ExecCGI
  </Location>
  FastCgiConfig -initial-env PERL5LIB=/usr/local/wormbase/gbrowse-2.01/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/gbrowse$
</IfModule>

<IfModule mod_perl.c>
   Alias /mgb2 "/usr/local/wormbase/gbrowse-current/cgi/gb2"
   PerlSwitches -I/usr/local/wormbase/gbrowse-2.01/extlib/lib/perl5/x86_64-linux-gnu-thread-multi -I/usr/local/wormbase/gbrowse-2.01/extlib/lib/p$
   <Location /mgb2>
     SetHandler perl-script
     PerlResponseHandler ModPerl::Registry
     PerlOptions +ParseHeaders
   </Location>
</IfModule>

</VirtualHost>

Enable virtual hosts in httpd.conf

$ sudo perl -p -i -e's|\#Include conf/extra/httpd-hosts.conf|Include conf/extra/httpd_hosts.conf|' \
     /usr/local/apache2/conf/httpd.conf

Set up httpd to run under inet.d

Remove /etc/init.d/apache2. It just confuses things.

Save the following file to /etc/init.d/httpd:

#!/bin/bash
#
# Startup script for the Apache Web Server
#
# chkconfig: - 85 15
# description: Apache is a World Wide Web server.  It is used to serve \
#              HTML files and CGI.
# processname: httpd
# pidfile: /usr/local/apache2/logs/httpd.pid
# config: /usr/local/apache2/conf/httpd.conf

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/local/apache2/bin/apachectl
httpd=/usr/local/apache2/bin/httpd
pid=/usr/local/apache2/logs/httpd.pid
prog=httpd
RETVAL=0

# The semantics of these two functions differ from the way apachectl does
# things -- attempting to start while running is a failure, and shutdown
# when not running is also a failure.  So we just do it the way init scripts
# are expected to behave here.
start() {
        echo -n $"Starting $prog: "
        daemon $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch /var/lock/subsys/httpd
        return $RETVAL
}
stop() {
        echo -n $"Stopping $prog: "
        killproc $httpd
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f /var/lock/subsys/httpd $pid
}
reload() {
        echo -n $"Reloading $prog: "
        killproc $httpd -HUP
        RETVAL=$?
        echo
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status $httpd
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  condrestart)
        if [ -f $pid ] ; then
                stop
                start
        fi
        ;;
  reload)
        reload
        ;;
  graceful|help|configtest|fullstatus)
        $apachectl $@
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $prog {start|stop|restart|condrestart|reload|status"
		echo $"|fullstatus|graceful|help|configtest}"
        exit 1
esac

exit $RETVAL

Set which runlevels httpd will run under:

chkconfig --add httpd
chkconfig --level 2345 httpd on
chkconfig --list

MySQL

For now, Master server only. The slaves will use the MySQL databases on the master.

Installation (>= 5.1)

We need a newer version of mysql than is available from the package manager. We'lll install the binary for x86_64.

For now, only the master server will run mysql. In the future, each dedicated slave could as well in order to meet growing demand.

$ groupadd mysql
$ useradd -g mysql mysql
$ usermod -a -G mysql tharris
$ cd /usr/local
$ sudo gunzip < /path/to/mysql-VERSION-OS.tar.gz | sudo tar xvf -
$ sudo ln -s full-path-to-mysql-VERSION-OS mysql
$ cd mysql
$ sudo chown -R mysql .
$ sudo chgrp -R mysql .
$ sudo scripts/mysql_install_db --user=mysql  // may be necessary to remove /etc/mysql/my.cnf first
$ sudo chown -R root .
$ sudo chown -R mysql data
$ sudo chmod 2775 data // I like to be able to write to my datadir

// Copy the configuration file
$ sudo cp support_files/my-medium.cnf /etc/my.cnf

// Set up mysql to start up automatically
$ sudo cp support_files/mysql.server /etc/init.d/.
$ cd /etc/rc3.d
$ sudo ln -s ../init.d/mysql.server S90mysql
$ cd ../rc5.d
$ sudo ln -s ../init.d/mysql.server S90mysql

// Start it up and set a root password.
$ sudo /etc/init.d/mysql.server start
$ mysqladmin -u root password 'PASSWORD'

Databases

MySQL databases will be mirrored over directly as part of the production update process. They should be located in /usr/local/mysql/data. Symlinks, using the symbolic name of the database, point to the current version:

 c_elegans -> c_elegans_WS211

Privileges

Be sure to grant privileges to both master and slave servers to the databases:

$ mysql -u root -p -e 'grant select on c_elegans.* to www-data@localhost'
$ mysql -u root -p -e 'grant select on c_elegans.* to www-data@Wb-www2.oicr.on.ca'

Install Requisite Perl Modules

Both master and slave servers.

All libraries specific for GBrowse are maintained in a local directory, specific to the version of gbrowse being run. This means that prerequisites need to be installed for each version of Gbrowse.

$ cd /usr/local/wormbase
$ mkdir gbrowse-2.01  // eg
$ ln -s gbrowse-2.01 gbrowse-current
$ mkdir gbrowse-current/extlib

Install local::lib to make things easier:

$ sudo perl -MCPAN -e 'CPAN::install(local::lib)'
$ cd /usr/local/wormbase/gbrowse-current/extlib

To install modules to this path:

$ perl -Mlocal::lib=.
$ eval $(perl -Mlocal::lib=--self-contained,./)
$ sudo rm -rf ~/.cpan
$ perl -MCPAN -e shell

// You need libexpat first. On debian:
$ sudo apt-get install expat libexpat1 libexpat1-dev

cpan> install Bio::Perl        // may be necessary to install from source
cpan> install Bio::Graphics    // may be necessary to install from source

GD can be problematic:

$ sudo apt-get install libgd2-noxpm libgd2-noxpm-dev
cpan> install GD GD::SVG

Install GBrowse2

$ perl ./Build.PL --install_base /usr/local/wormbase/gbrowse-current
$ perl ./Build --reconfig
Directory for GBrowse's config and support files? /usr/local/wormbase/gbrowse-current/conf
Directory for GBrowse's static images & HTML files? /usr/local/wormbase/gbrowse-current/html
Directory for GBrowse's temporary data /usr/local/wormbase/gbrowse-current/tmp
Directory for GBrowse's example databases /usr/local/wormbase/gbrowse-current/databases
Directory for GBrowse's CGI script executables? /usr/local/wormbase/gbrowse-current/cgi/gb2  
Internet port to run demo web site on (for demo)? [8000] 
Apache loadable module directory (for demo)? [/usr/local/apache2/modules] 
User account under which Apache daemon runs? [www-data] 
Automatically update Apache config files to run GBrowse? [n] 
Automatically update system config files to run gbrowse-slave? [n] y

To fetch suggested apache configuration:

./Build apache_conf

Configuration

Support files

The WormBase-specific installation requires a number of support files: css and an image or two. We keep these files -- and all configuration files -- outside of the gbrowse installation path. This makes upgrades much, much easier.

Support files are maintained in WormBase CVS. The directory looks like:

bash> cd /usr/local/wormbase/gbrowse-support-files
bash> ls -1
  conf/
  banner-image.jpg
  wormbase-gbrowse.css

bash> ls -1 conf/
 GBrowse.conf    // The main gbrowse config file, #included by the primary GBrowse.conf file.
 c_elegans.conf
 c_remanei.conf
 etc...

See the Apache configuration section for details.