Difference between revisions of "Wormmart administration"

From WormBaseWiki
Jump to navigationJump to search
Line 68: Line 68:
  
 
  $ sudo apt-get remove apache2
 
  $ sudo apt-get remove apache2
 
  
 
  # Build httpd 2.2.15
 
  # Build httpd 2.2.15

Revision as of 13:58, 12 April 2010

Overview

WormMart is WormBase's implementation of the BioMart data warehouse.

Basic Server Configuration

Create groups

sudo addgroup wormbase
sudo addgroup biomart
# Add yourself to the groups
sudo usermod -a -G biomart, wormbase acabunoc

Create directories

Create a directory for GBrowse. For now, we're maintaining GBrowse on dedicated hardware completely distinct from the rest of the web application. This makes updates significantly easier, although it increases maintenance (ie boilerplate headers and footers are maintained in two places). We'll also keep gbrowse inside a wormbase/ directory should we wish to install other components on these servers at a later date.

$ sudo mkdir -p /usr/local/wormbase
$ sudo chgrp -R wormbase /usr/local/wormbase
$ sudo chmod 2775 /usr/local/wormbase

Install prerequisites (Debian)

// Update apt
$ sudo apt-get update
// Make tools
$ sudo apt-get install gcc make
// expat
$ sudo apt-get install expat libexpat1 libexpat1-dev
// GD support
$ sudo apt-get install libgd2-noxpm libgd2-noxpm-dev
// Graphviz, now apparently required by BioPerl
$ sudo apt-get install graphviz
// Curl
$ sudo apt-get install curl

Install Perl 5.10.1 or greater

You'll need Perl version 5.10.1 or greater.

Debian:

$ sudo apt-get install perl

NOTE: This version of Perl on Debian Lenny doesn't ship with perl.h. You won't be able to build mod_perl without it! You should probably build from source.

Building from source:

bash> cd ~/src
bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz
bash> cd ~/build
bash> tar xzf ../src/perl-5.10.1.tar.gz
bash> cd perl-5.10.1
bash> ./Configure -des
  // Or, to install in a local path:
bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1
bash> make
bash> make test
bash> sudo make install

Install Apache2

If you're running Debian, get rid of the installed apache. The default layout is just plain annoying.

$ sudo apt-get remove apache2
# Build httpd 2.2.15
cd ~/src
curl -O http://apache.raffsoftware.com/httpd/httpd-2.2.15.tar.gz
tar xzf httpd-2.2.15.tar.gz
cd httpd-2.2.15
./configure --enable-mods-shared=all
make
sudo make install

Apache Configuration

Edit the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) with the appropriate port, virtual host, and fcgi settings:

  • Set httpd to listen on your desired port. In the examples below, we assume port 8080.
# Edit "Listen 80" to read
Listen 8080

Enable virtual hosts by uncommenting out the following line:

#Include conf/extra/httpd_hosts.conf

Enable FastCGI module (installed later)

 LoadModule fastcgi_module modules/mod_fastcgi.so

Add FastCGI config from GBrowse (see the GBrowse section below for details). Note that we tweak the Location Aliases to place the fast CGI-enabled GBrowse under /db/gb2.

<IfModule mod_fcgid.c>
  Alias /db/gb2 "/usr/local/wormbase/gbrowse-current/cgi/gb2"
  <Location /db/gb2>
    SetHandler   fcgid-script
    Options      ExecCGI
  </Location>
  DefaultInitEnv GBROWSE_CONF /usr/local/wormbase/gbrowse-current/conf
  DefaultInitEnv PERL5LIB /usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multig:/usr/local/wormbase/extlib/lib/perl5:/usr/local/wormbase/gbrowse-current/lib/perl5:/usr/local/wormbase/gbrowse-support-files/lib:/usr/local/wormbase/gbrowse-support-files/lib/Bio:/usr/local/wormbase/gbrowse-support-files/lib/Bio/Graphics:/usr/local/wormbase/gbrowse-support-files/lib/Bio/Graphics/Glyph
</IfModule>

<IfModule mod_fastcgi.c>
  Alias /db/gb2 "/usr/local/wormbase/gbrowse-current/cgi/gb2"
  <Location /db/gb2>
    SetHandler   fastcgi-script
    Options      ExecCGI
  </Location>
  FastCgiConfig -initial-env PERL5LIB=/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multig:/usr/local/wormbase/extlib/lib/perl5:/usr/local/wormbase/gbrowse-current/lib/perl5:/usr/local/wormbase/gbrowse-support-files/lib:/usr/local/wormbase/gbrowse-support-files/lib/Bio:/usr/local/wormbase/gbrowse-support-files/lib/Bio/Graphics:/usr/local/wormbase/gbrowse-support-files/lib/Bio/Graphics/Glyph
</IfModule>


Set up a virtual host on your port by editing /usr/local/apache2/conf/extras/httpd-vhosts.conf:

<VirtualHost *:80>
    ErrorLog "/usr/local/wormbase/logs/error_log"
    CustomLog "/usr/local/wormbase/logs/access_log" common

    # Everything for GBrowse2 will be served from under a url of either
    #      /db/gb2  or /gbrowse2
    # This makes proxying from CSHL more convenient.
    # Note that /db is historical and required for mapping the old site to the new
    Alias        "/gbrowse2/i/" "/usr/local/wormbase/gbrowse-current/tmp/images/"

    # NOTE: this MUST be different from the ScriptAlias.
    Alias        "/gbrowse2"    "/usr/local/wormbase/gbrowse-current/html"
    ScriptAlias  "/db/gb2"      "/usr/local/wormbase/gbrowse-current/cgi/gb2"

    # Support files: the wormbase css file for gbrowse, banner image, etc
    Alias       "/gb2-support/     "/usr/local/wormbase/gbrowse-support-files/"

   <Directory "/usr/local/wormbase/gbrowse-support-files">
      Options -Indexes -MultiViews +FollowSymLinks
      Order allow,deny
      Allow from all
   </Directory>

   # The temporary GBrowse directory
   <Directory "/usr/local/wormbase/gbrowse-current/tmp">
      Allow from all
   </Directory>

   <Directory "/usr/local/wormbase/gbrowse-current/html">
      Options -Indexes -MultiViews +FollowSymLinks
      Order allow,deny                                              
      Allow from all
   </Directory>

   <Directory "/usr/local/wormbase/gbrowse-current/cgi/gb2">
      SetEnv PERL5LIB "/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/extlib/lib/perl5:/usr/local/wormbase/gbrowse-current/lib/perl5"
      SetEnv GBROWSE_CONF   "/usr/local/wormbase/gbrowse-current/conf"
   </Directory>
  </VirtualHost>

Set up httpd to run under inet.d

Only required for the master server.

Remove /etc/init.d/apache2. It just confuses things.

Save the following file to /etc/init.d/httpd:

#!/bin/bash
#
# Startup script for the Apache Web Server
#
# chkconfig: - 85 15
# description: Apache is a World Wide Web server.  It is used to serve \
#              HTML files and CGI.
# processname: httpd
# pidfile: /usr/local/apache2/logs/httpd.pid
# config: /usr/local/apache2/conf/httpd.conf

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/local/apache2/bin/apachectl
httpd=/usr/local/apache2/bin/httpd
pid=/usr/local/apache2/logs/httpd.pid
prog=httpd
RETVAL=0

# The semantics of these two functions differ from the way apachectl does
# things -- attempting to start while running is a failure, and shutdown
# when not running is also a failure.  So we just do it the way init scripts
# are expected to behave here.
start() {
        echo -n $"Starting $prog: "
        daemon $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch /var/lock/subsys/httpd
        return $RETVAL
}
stop() {
        echo -n $"Stopping $prog: "
        killproc $httpd
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f /var/lock/subsys/httpd $pid
}
reload() {
        echo -n $"Reloading $prog: "
        killproc $httpd -HUP
        RETVAL=$?
        echo
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status $httpd
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  condrestart)
        if [ -f $pid ] ; then
                stop
                start
        fi
        ;;
  reload)
        reload
        ;;
  graceful|help|configtest|fullstatus)
        $apachectl $@
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $prog {start|stop|restart|condrestart|reload|status"
		echo $"|fullstatus|graceful|help|configtest}"
        exit 1
esac

exit $RETVAL

Set which runlevels httpd will run under:

chkconfig --add httpd
chkconfig --level 2345 httpd on
chkconfig --list

Install Fast CGI

$ cd /usr/local/wormbase/build
$ wget http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz
$ tar xzf mod_fastcgi.*.tar.gz
$ cd mod_fastcgi*
$ cp Makefile.AP2 Makefile
$ make
$ sudo make install

MySQL

Each node in the renderfarm has its own MySQL databases.

Installation (>= 5.1)

We need a newer version of mysql than is available from the package manager. We'lll install the binary for x86_64.

$ sudo groupadd mysql
$ sudo useradd -g mysql mysql
$ sudo usermod -a -G mysql tharris
$ cd /usr/local
$ sudo gunzip < /path/to/mysql-VERSION-OS.tar.gz | sudo tar xvf -
$ sudo ln -s full-path-to-mysql-VERSION-OS mysql
$ cd mysql
$ sudo chown -R mysql .
$ sudo chgrp -R mysql .
$ sudo scripts/mysql_install_db --user=mysql  // may be necessary to remove /etc/mysql/my.cnf first
$ sudo chown -R root .
$ sudo chown -R mysql data
$ sudo chmod 2775 data // I like to be able to write to my datadir
// Copy the configuration file
$ sudo cp support_files/my-medium.cnf /etc/my.cnf
// It may be necessary to add the following to my.cnf if back end nodes
// don't have fully qualified domain names
skip-name-resolve
// Set up mysql to start up automatically
$ sudo cp support_files/mysql.server /etc/init.d/.
$ cd /etc/rc3.d
$ sudo ln -s ../init.d/mysql.server S90mysql
$ cd ../rc5.d
$ sudo ln -s ../init.d/mysql.server S90mysql
// Start it up and set a root password.
$ sudo /etc/init.d/mysql.server start
$ mysqladmin -u root password 'PASSWORD'

Databases

MySQL databases will be mirrored over directly as part of the production update process. They should be located in /usr/local/mysql/data. Symlinks, using the symbolic name of the database, point to the current version:

 c_elegans -> c_elegans_WS211

Privileges

Be sure to grant privileges to both master and slave servers to the databases:

$ mysql -u root -p -e 'grant select on c_elegans.* to nobody@localhost'
$ mysql -u root -p -e 'grant select on c_elegans.* to nobody@render-slave-ip'

Install Requisite Perl Modules

All libraries specific for GBrowse are maintained in a local shared directory. This makes it easier and faster to upgrade GBrowse, but more difficult to test multiple versions simultaneously. To test a new version of GBrowse, place extlib inside of the gbrowse-current directory and modify the apache configuration and commands below as appropriate.

Install local::lib to make things easier:

$ sudo perl -MCPAN -e 'CPAN::install(local::lib)'
$ cd /usr/local/wormbase/extlib

To install modules to this path:

$ perl -Mlocal::lib=.
$ eval $(perl -Mlocal::lib=./)
$ sudo rm -rf ~/.cpan
$ perl -MCPAN -e shell
cpan> install Bio::Perl        // may be necessary to install from source
cpan> install Bio::Graphics    // may be necessary to install from source