Difference between revisions of "Administration:Installing WormBase"

From WormBaseWiki
Jump to navigationJump to search
 
(187 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Address =
+
= Overview =
  
wb-dev.oicr.on.ca
+
This document describes how to install WormBase locally on your own machine.  Please note that as of August 2013, WormBase is almost entirely a distributed cloud-based application. Although possible to install and run on a single server (or cloud instance) the directions below may be slightly out-of-sync with how the principal WormBase architecture is built.
  
= Hardware =
+
= Recommendations =
  
The WormBase development server at OICR is a virtual server with the following stats:
+
== Hardware ==
  
* Debian Linux
+
'' As of April 2012, the suggested minimum hardware requirements for a basic installation of WormBase are:''
*500 GB disk space (mounted at /dev/hda1)
 
* 4 GB RAM
 
  
= Server Configuration =
+
* 500 GB disk space (1 TB preferable)
 +
* 8 GB RAM
  
== Basic Directory Layout ==
+
== Operating Systems ==
  
All WormBase components are collected under a single directory: /usr/local/wormbase
+
''You should be able to install and run the site on any Unix-y OS, including Mac OS X.  This document only describes installation on Debian and Debian-ish OSes like Amazon's Linux.  Your actual mileage may vary on other operating systems.''
  
$ ls /usr/local/wormbase
+
= Basic System Configuration =
      acedb/                    // The Acedb database (including bin directory)
 
      util/                      // Utility components such as e-pcr and wublast
 
      extlib/                  // Third party Perl libraries 
 
      website-classic  // The classic WormBase website
 
      website                  // The new-and-improved website!
 
  
Apache2: /usr/local/apache2
+
== Users And Groups ==
MySQL data directory: /usr/local/mysql/data
 
 
 
== Installing Libraries ==
 
 
 
Assuming a vanilla Debian installation, install the following libaries and all of their dependencies via ''sudo apt-get install''. Some of these libraries are discussed below.
 
      gcc
 
      curl
 
      wget
 
      bzip2
 
      mysql-server
 
      mysql-server-5.0
 
      libgd2-xpm-dev
 
      libgd2-xpm
 
      xinetd
 
      libdbd-mysql
 
      libdbd-mysql-perl
 
      apache2
 
      libapache2-mod-perl2
 
      libgtk2.0-0
 
      libgtk2.0-dev
 
      libglib
 
      byacc
 
      libreadline5-dev
 
      flex
 
      libdb4.6
 
      libdb-dev
 
      emacs
 
 
 
== Preparing directories and users ==
 
 
 
WormBase uses several user accounts for directory and server permissions. You will need to create these users and several preliminary directories. Creating a new user and group varies among Unix flavors. On most Linux systems, the following commands will create the new groups. You should have sudo privilege to execute these commands.
 
 
 
=== User and group accounts ===
 
  
 
These users should not have a login password. They are to establish privileges only.
 
These users should not have a login password. They are to establish privileges only.
  
* acedb group
+
'''acedb group'''
  
 
This is the group that will have write privileges to the acedb directory tree. Acedb administrators should be added to this group.
 
This is the group that will have write privileges to the acedb directory tree. Acedb administrators should be added to this group.
  
  $ /usr/sbin/groupadd acedb
+
  $ groupadd acedb
  
* acedb user
+
'''acedb user'''
  
 
This is the user that the acedb server will run as. It should be a member of the acedb group.
 
This is the user that the acedb server will run as. It should be a member of the acedb group.
  
  $ /usr/sbin/useradd -g acedb -d /usr/local/wormbase/acedb acedb
+
  $ useradd -g acedb -d /usr/local/wormbase/acedb acedb
  
 
This ''useradd'' command also adds the new acedb user to the acedb group. Note that the acedb user's home directory was set to /usr/local/acedb, a directory which will be created in the next step.
 
This ''useradd'' command also adds the new acedb user to the acedb group. Note that the acedb user's home directory was set to /usr/local/acedb, a directory which will be created in the next step.
  
* wormbase group
+
'''wormbase group'''
  
 
This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group.
 
This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group.
  
  $ /usr/sbin/groupadd wormbase
+
  $ groupadd wormbase
  
 
This would be a good time to add yourself to the acedb and wormbase groups.
 
This would be a good time to add yourself to the acedb and wormbase groups.
  
  $ /usr/sbin/usermod -a -G acedb,wormbase [your_login_name]
+
  $ usermod -a -G acedb,wormbase [your_login_name]
 
 
[The '-a' argument keeps this command from deleting other, preexisting group memberships.]
 
  
 
You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of:
 
You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of:
Line 91: Line 50:
 
  % groups
 
  % groups
  
=== Directories ===
+
== Create Directories ==
  
The root container for all things WormBase:
+
''Create a root directory for all WormBase-related files. Although this directory can reside anywhere on your filesystem, for consistency with WormBase proper, we recommend installing at /usr/local/wormbase.  You'll need suitable privileges to install in this location.'''
 +
 
 +
'''The root container for all things WormBase'''
  
 
* /usr/local/wormbase, owner=root group=wormbase mode=drwxrwsr-x
 
* /usr/local/wormbase, owner=root group=wormbase mode=drwxrwsr-x
  
  $ mkdir /usr/local/wormbase
+
  $ cd /usr/local
$ chgrp wormbase /usr/local/wormbase
+
  $ sudo mkdir wormbase
$ chmod 2775 /usr/local/wormbase
+
  $ sudo chown root:wormbase wormbase
 
+
  $ sudo chmod 2775 wormbase
* External Perl libraries: /usr/local/wormbase/extlib, owner=tharris group=wormbase mode=drwxrwsr-x
 
 
 
  $ mkdir /usr/local/wormbase/extlib
 
  $ chgrp wormbase /usr/local/wormbase/extlib
 
$ chmod 2775 /usr/local/wormbase/extlib
 
 
 
* The "classic" website: /usr/local/wormbase/website-classic, owner=root group=wormbase mode=drwxrwsr-x
 
 
 
$ mkdir /usr/local/wormbase/website-classic
 
$ chgrp wormbase /usr/local/wormbase/website-classic
 
  $ chmod 2775 /usr/local/wormbase/website-classic
 
 
 
* /usr/local/wormbase/website-classic/logs, owner=root group=wormbase mode=drwxrwsr-x
 
 
 
$ mkdir /usr/local/wormbase/website-classic/logs
 
$ chgrp wormbase /usr/local/wormbase/website-classic/logs
 
$ chmod 2775 /usr/local/wormbase/website-classic/logs
 
 
 
* /usr/local/wormbase/website-classic/cache, owner=nobody group=nobody mode=drwxrwsr-x
 
 
 
$ mkdir /usr/local/wormbase/website-classic/cache
 
$ chown nobody:nobody /usr/local/wormbase/website-classic/cache
 
$ chmod 2775 /usr/local/wormbase/website-classic/cache
 
  
* The "util" directory contains components that apply to both the classic and updated site, like wublast and e-pcr.
+
'''The AceDB directory contains acedb and associated databases:'''
 
 
* /usr/local/wormbase/util/wublast, owner=root group=wormbase mode=drwxrwsr-x
 
 
 
$ mkdir /usr/local/wormbase/util/wublast
 
$ chgrp wormbase /usr/local/wormbase/util/wublast
 
$ chmod 2775 /usr/local/wormbase/util/wublast
 
  
 
* /usr/local/wormbase/acedb, owner=acedb group=acedb,mode=drwxrwsr-x
 
* /usr/local/wormbase/acedb, owner=acedb group=acedb,mode=drwxrwsr-x
  
 
  $ mkdir /usr/local/wormbase/acedb
 
  $ mkdir /usr/local/wormbase/acedb
$ chown acedb:acedb /usr/local/wormbase/acedb
 
 
  $ chmod 2775 /usr/local/wormbase/acedb
 
  $ chmod 2775 /usr/local/wormbase/acedb
 +
$ sudo chown acedb:acedb /usr/local/wormbase/acedb
  
== Configure the FTP/Mirroring directory THIS IS NOT DONE YET ==
+
'''The website/ directory contains one or many versions of the website:'''
  
* ~ftp/pub/wormbase, owner=root group=wormbase mode=drwxrwsr-x
+
* /usr/local/wormbase/website, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
  
  $ mkdir ~ftp/pub/wormbase
+
  $ mkdir /usr/local/wormbase/website
  $ chgrp wormbase ~ftp/pub/wormbase
+
  $ chmod 2775 /usr/local/wormbase/website
  $ chmod 2775 ~ftp/pub/wormbase
+
  $ sudo chgrp wormbase /usr/local/wormbase/website
  
You may ignore this step if you do not plan to mirror the WormBase FTP site. In the examples below, the -p option is used to create the intermediate parents of directories if they don't already exist. If your mkdir doesn't support this option, you will need to create the intermediate directories manually.
+
'''The services/ directory contains optional services such as e-pcr, blast, and nginx:'''
  
= Apache2 and mod_perl =
+
* /usr/local/wormbase/services, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
  
I prefer to build httpd and mod_perl from source; it seems easier than waiting for package repositories to be updated.
+
$ mkdir /usr/local/wormbase/services
 +
$ chmod 2775 /usr/local/wormbase/services
 +
$ sudo chgrp wormbase /usr/local/wormbase/services
  
== Installation ==
+
'''The databases/ directory contains databases driving a number of features:'''
  
''Assuming you have already fetched the source into ~/src:''
+
* /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
  
  # Build httpd 2.2.11
+
  $ mkdir /usr/local/wormbase/databases
  cd ~/src
+
  $ chmod 2775 /usr/local/wormbase/databases
tar xzf httpd-2.2.11.tar.gz
+
  $ sudo chgrp wormbase /usr/local/wormbase/databases
cd httpd-2.2.11
 
./configure --enable-mods-shared=all  // We can enable modules in httpd.conf later
 
make
 
  sudo make install
 
  
# Build mod_perl2-0.4
+
'''The logs/ directory contains WormBase-wide log and pid files'''
cd ~/src
 
cd mod_perl-2.0.4
 
// Get rid of some (possible) cruft
 
sudo rm -rf /usr/local/wormbase/extlib/lib/perl5/i686-linux-thread-multi/Apache2 \
 
                      /usr/local/wormbase/extlib/lib/perl5/i686-linux-thread-multi/ModPerl \
 
                      /usr/local/wormbase/extlib/lib/perl5/i686-linux-thread-multi/mod_perl2.pm
 
make clean
 
perl -I/usr/local/wormbase/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/extlib
 
// When prompted for the apxs path, enter: /usr/local/apache2/bin/apxs
 
make
 
sudo make install
 
  
== Configuration ==
+
* /usr/local/wormbase/logs, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx
  
* Set httpd to listen on port 8080.
+
$ mkdir /usr/local/wormbase/logs
 +
$ chmod 2775 /usr/local/wormbase/logs
 +
$ sudo chgrp wormbase /usr/local/wormbase/logs
  
$ sudo perl -p -i -e's|Listen 80|Listen 8080|' /usr/local/apache2/conf/httpd.conf
+
== Installing Library Prerequisites ==
  
*  Set up a virtual host for WormBase on port 8080
+
Assuming a vanilla Debian installation, install the following libaries and all of their dependencies via ''sudo apt-get install''. Some of these libraries are discussed below.
 +
      sharutils \
 +
      gcc g++ \
 +
      curl wget \
 +
      bzip2 \
 +
      mysql-client \
 +
      mysql-server \
 +
      libgd2-xpm-dev\
 +
      libgd2-xpm \
 +
      xinetd \
 +
      libdbd-mysql \
 +
      libdbd-mysql-perl \
 +
      libapache2-mod-perl2 \
 +
      libgtk2.0-0 \
 +
      libgtk2.0-dev \
 +
      emacs \
 +
      git \
 +
      byacc \
 +
      libreadline-gplv2-dev \
 +
      flex \
 +
      libdb4.6 \
 +
      libdb-dev \
 +
      xml2 libxml2 libxml2-dev \
 +
      libmysql++-dev \
 +
      ncurses-bin \
 +
      libxmu-dev \
 +
      graphviz \
 +
      libssl-dev \
 +
      uuid \
 +
      uuid-dev \
 +
      libxslt1-dev \
 +
      xsltproc \
 +
      libgdbm3 \
 +
      libgdbm3-dev \
 +
      libstdc++ \
 +
      libstdc++-devel
  
$ sudo emacs /usr/local/apache2/conf/extra/httpd-vhosts.conf
+
Build xapian (or install from package if available):
<VirtualHost *:8080>
 
    Include /usr/local/wormbase/website-classic/conf/httpd.conf
 
    PerlOptions +Parent
 
    PerlSwitches  -Mlib=/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi \
 
                -Mlib=/usr/local/wormbase/extlib/gbrowse1/i686-linux-thread-multi \
 
                -Mlib=/usr/local/wormbase/extlib/gbrowse1 \
 
                -Mlib=/usr/local/wormbase/extlib/lib/perl5/i686-linux-thread-multi
 
  
  #  PerlInterpStart 2
+
  $ wget http://oligarchy.co.uk/xapian/1.2.5/xapian-core-1.2.5.tar.gz
#  PerlInterpMax 2
+
$ tar xzf xapian-core*.tgz
    <Directory "/usr/local/wormbase/website-classic/html">
+
  $ cd xapian-core
        DirectoryIndex index.html
+
  $ ./configure
  #      Order deny,allow
+
$ make
  #      Deny from all
+
  $ sudo make install
    </Directory>
 
  </VirtualHost>
 
  
* Enable virtual hosts in httpd.conf
+
Install Search::Xapian
 +
$ perl -MCPAN -e 'install Search::Xapian'
  
$ sudo perl -p -i -e's|\#Include conf/extra/httpd-hosts.conf|Include conf/extra/httpd_hosts.conf|' \
+
== Install Perl 5.10.1 or greater ==
      /usr/local/apache2/conf/httpd.conf
 
  
== Set up httpd to run under inet.d ==
+
You'll need Perl version 5.10.1 or greater.
  
''Save the following file to /etc/init.d/httpd:''
+
'''Upgrading Perl via apt (recommended)'''
  
<pre>
+
Let your operating system managing your version of Perl.
#!/bin/bash
 
#
 
# Startup script for the Apache Web Server
 
#
 
# chkconfig: - 85 15
 
# description: Apache is a World Wide Web server. It is used to serve \
 
#              HTML files and CGI.
 
# processname: httpd
 
# pidfile: /usr/local/apache2/logs/httpd.pid
 
# config: /usr/local/apache2/conf/httpd.conf
 
  
# Source function library.
+
''Configure your apt preferences''
. /etc/rc.d/init.d/functions
 
  
if [ -f /etc/sysconfig/httpd ]; then
+
Add the following to /etc/apt/sources.list
        . /etc/sysconfig/httpd
 
fi
 
  
# This will prevent initlog from swallowing up a pass-phrase prompt if
+
#Stable
# mod_ssl needs a pass-phrase from the user.
+
deb http://ftp.us.debian.org/debian stable main non-free contrib
INITLOG_ARGS=""
+
#Testing
 +
deb http://ftp.us.debian.org/debian testing main non-free contrib
 +
#Unstable
 +
deb http://ftp.us.debian.org/debian unstable main non-free contrib
  
# Path to the apachectl script, server binary, and short-form for messages.
+
Configure your preferences in /etc/apt/preferences:
apachectl=/usr/local/apache2/bin/apachectl
 
httpd=/usr/local/apache2/bin/httpd
 
pid=/usr/local/apache2/logs/httpd.pid
 
prog=httpd
 
RETVAL=0
 
  
 +
Package: *
 +
Pin: release a=stable
 +
Pin-Priority: 700
 +
Package: *
 +
Pin: release a=testing
 +
Pin-Priority: 650
 +
Package: *
 +
Pin: release a=unstable
 +
Pin-Priority: 600
  
# The semantics of these two functions differ from the way apachectl does
+
Finally, install Perl
# things -- attempting to start while running is a failure, and shutdown
 
# when not running is also a failure.  So we just do it the way init scripts
 
# are expected to behave here.
 
start() {
 
        echo -n $"Starting $prog: "
 
        daemon $httpd $OPTIONS
 
        RETVAL=$?
 
        echo
 
        [ $RETVAL = 0 ] && touch /var/lock/subsys/httpd
 
        return $RETVAL
 
}
 
stop() {
 
        echo -n $"Stopping $prog: "
 
        killproc $httpd
 
        RETVAL=$?
 
        echo
 
        [ $RETVAL = 0 ] && rm -f /var/lock/subsys/httpd $pid
 
}
 
reload() {
 
        echo -n $"Reloading $prog: "
 
        killproc $httpd -HUP
 
        RETVAL=$?
 
        echo
 
}
 
  
# See how we were called.
+
sudo apt-get update
case "$1" in
+
sudo apt-get install perl/testing perl-base/testing perl-modules/testing libperl5.10/testing libperl-dev/testing
  start)
 
        start
 
        ;;
 
  stop)
 
        stop
 
        ;;
 
  status)
 
        status $httpd
 
        RETVAL=$?
 
        ;;
 
  restart)
 
        stop
 
        start
 
        ;;
 
  condrestart)
 
        if [ -f $pid ] ; then
 
                stop
 
                start
 
        fi
 
        ;;
 
  reload)
 
        reload
 
        ;;
 
  graceful|help|configtest|fullstatus)
 
        $apachectl $@
 
        RETVAL=$?
 
        ;;
 
  *)
 
        echo $"Usage: $prog {start|stop|restart|condrestart|reload|status"
 
echo $"|fullstatus|graceful|help|configtest}"
 
        exit 1
 
esac
 
  
exit $RETVAL
+
And confirm
</pre>
 
  
''Set which runlevels httpd will run under:''
+
$ perl -v
 +
  This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
 +
  (with 40 registered patches, see perl -V for more detail)
  
chkconfig --add httpd
+
'''Installing Perl from source'''
chkconfig --level 2345 httpd on
 
chkconfig --list
 
  
==Important Fix for mod_perl2/mod_dir incompatibility (the index.html problem) ==
+
We recommend installing a local version of Perl and required libraries. This makes management dramatically easier.  We keep our Perl with the webapp itself.
  
mod_perl2 intercepts Apache requests before any other modules.  This means that for locations/directories configured with PerlHandlers that requests for dir or dir/ will NOT automatically redirect to the value of DirectoryIndex. To fix this problem, the lib/Apache/AddWormBaseBanner.pm needs to be modified:
+
If you have sufficient privileges, you may wish to use your pre-installed system Perl.
  
Insert the following:
+
bash> mkdir ~/website/perl
 +
bash> cd ~/src
 +
bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz
 +
bash> cd ~/build
 +
bash> tar xzf ../src.perl-5.10.1.tar.gz
 +
bash> cd perl-5.10.1
 +
bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1
 +
bash> make
 +
bash> make test
 +
bash> make install
  
# We need two additional modules:
+
Be sure to set up your your system to preferentially use this newly installed Perl:
use Apache2::Const -compile => qw(DIR_MAGIC_TYPE OK DECLINED);
 
use Apache2::SubRequest;
 
  
  sub handler {
+
  bash> export PATH=~/wormbase/perl/5.10.1/bin:${PATH}
    my $r = shift;
 
  # --> Copy and paste starting here...
 
  # A directory request has content-type = httpd/unix directory                                                           
 
  # we check that the uri ends in a slash, since only in that case                                           
 
  # do we want to redirect, and finally to avoid redirect loops                                                                                     
 
  # we only do this on the initial request.                                                                                               
 
  # You must load Apache2::SubRequest in order to run internal_redirect                                                                                                       
 
  if ($r->content_type eq 'httpd/unix-directory'
 
      && $r->uri =~ '/$' && $r->is_initial_req ) {
 
      #        print STDERR "Accepting Directory Request\n";                                                                                                         
 
      # warn "internal request";                                                                                                                                                                               
 
        $r->internal_redirect($r->uri . 'index.html');
 
        return OK;
 
    }
 
  
= Perl modules =
+
= Install the Webapp =
  
=== CPAN / Environment configuration ===
+
'''Fetch the current production code from the git repository on github.'''
 +
$ cd /usr/local/wormbase/website
 +
$ git clone git@github.com:WormBase/website.git
 +
$ mv website YOUR_NAME // name can be anything you want; useful for maintaining multiple versions
  
I maintain a suite of Perl modules common to WormBase at:
+
'''Install local:: lib'''
  
  /usr/local/wormbase/extlib
+
''The preferred method uses the web app's Makefile.PL and Module::Install to build all dependencies. We maintain local copies of all libraries with each version of the webapp.''
  
If you need to over-ride the default version of a module, place it in the extlib directory of either the classic or rearchitecure site:
+
'''Build and install dependencies'''
  
  /usr/local/wormbase/website-classic/extlib
+
$ mkdir /usr/local/wormabase/extlib
    OR
+
$ cd /usr/local/wormbase/website/YOUR_NAME
  /usr/local/wormbase/website/extlib
+
$ perl Makefile.PL
 +
$ make installdeps
  
Set up CPAN to build modules in the local library path (/usr/local/wormbase/extlib):
+
''See the documentation on [[Managing Perl Libraries]] for additional details.''
  
  perl -MCPAN -e shell  // Note that you DO NOT need to be sudo...
+
= Required 3rd-party Applications =
  cpan> o conf init (only necessary if not prompted)
 
  
For the Makefile.PL arguments, enter
+
== Starman: the lightweight http server ==
 
  INSTALL_BASE=/usr/local/wormbase/extlib
 
 
 
And for Build.PL enter
 
 
 
  --install_base /usr/local/wormbase/extlib
 
 
 
Prepare/update your CPAN:
 
  
  cpan> install CPAN
+
PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).
  cpan> reload CPAN
 
  
Before installing modules, you may need to set your PERL5LIB environment variable to point to include the extlib directory.
+
  http://plackperl.org/
  
  emacs ~/.bash_profile
+
Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.mod_perl.
  export PERL5LIB /usr/local/wormbase/extlib:/usr/local/wormbase/extlib/lib:/usr/local/wormbase/extlib/lib/perl5
 
  
=== Basic required modules ===
+
  https://github.com/miyagawa/Starman
  
Install the following Perl modules via CPAN. Note that you DO NOT AND SHOULD NOT be sudo..
+
'''Install Plack and Starman'''
 +
:Catalyst::Controller::Metal
 +
:Catalyst::Engine::PSGI
 +
:Catalyst::Helper::PSGI
 +
:Plack::Test::Adopt::Catalyst
 +
:cpanm Task::Plack
 +
:cpanm Starman
  
perl -I/usr/local/wormbase/extlib/lib -MCPAN -e shell
+
'''Configuration'''
  
YAML
+
The PSGI glue resides at:
LWP
 
ExtUtils::MakeMaker
 
Bundle::CPAN
 
Cache::Cache
 
Cache::FileCache
 
CGI
 
CGI::Session    // CPAN installation fails in local dirs; tries to install man3 in system path.
 
CGI::Cache
 
CGI::Toggle
 
Date::Calc
 
Date::Manip      // CPAN installation fails in local dirs; tries to install man3 in system path.
 
DB_File
 
DBI
 
DBD::mysql  (mysql must be installed first)
 
Digest::MD5
 
GD
 
GD::SVG
 
GD::Graph
 
GD::Graph::pie
 
HTML::TokeParser
 
IO::Scalar
 
IO::String
 
Image::GD::Thumbnail
 
MIME::Lite
 
Net::FTP
 
Proc::Simple
 
readline
 
Search::Indexer
 
SOAP::Lite
 
Statistics::OLS
 
Storable
 
SVG
 
SVG::Graph
 
Test::Pod
 
Text::Shellwords
 
Time::Format
 
WeakRef
 
XML::SAX
 
XML::Parser
 
XML::DOM
 
XML::Writer
 
XML::Twig
 
XML::Simple
 
Flickr::API
 
Flickr::API::Simple (private)
 
Log::Log4perl
 
Bio::Graphics
 
Data::Stag
 
Class::Base
 
  
  YAML LWP ExtUtils::MakeMaker Bundle::CPAN Cache::Cache Cache::FileCache CGI CGI::Session CGI::Cache CGI::Toggle Date::Calc Date::Manip DB_File DBI Digest::MD5 GD  GD::SVG GD::Graph GD::Graph::pie HTML::TokeParser IO::Scalar IO::String Image::GD::Thumbnail MIME::Lite Net::FTP Proc::Simple readline Search::Indexer SOAP::Lite Statistics::OLS Storable SVG SVG::Graph Test::Pod Text::Shellwords Time::Format WeakRef XML::SAX XML::Parser XML::DOM XML::Writer XML::Twig XML::Simple Flickr::API Log::Log4perl Bio::Graphics Data::Stag Class::Base
+
  website/YOUR_NAME/wormbase.psgi
  
*  Note that even with INSTALL_BASE and install_base set for EUMM/MB, installation of some modules into local non-privileged paths will FAIL.  This is almost always due to an attempt to install man pages in system paths not using one of the INSTALL* paths.  The only solution with these modules is to install them by hand...
+
''' Starting Starman '''
  
=== Ace.pm (for the Classic WormBase site) ===
+
starman script/wormbase.psgi
 +
OR
 +
starman -MFindBin script/wormbase_production.psgi
  
Ace.pm provides programmatic access to Acedb. You can install it via CPAN:
+
'''Set up starman to launch automatically'''
  
  cpan> install Ace
+
Copy the stub init script from wormbase/conf/starman/starman.init:
  
During configuration, choose option (3), then set the remaining variables as follows:
+
   cp /usr/local/wormbase/admin/conf/starman/starman.init
 
 
   Site-specific configuration files:  /usr/local/wormbase/website-classic/conf
 
                            CGI path:  /usr/local/wormbase/website-classic/cgi-bin
 
                          HTML path:  /usr/local/wormbase/website-classic/html
 
 
 
=== BioPerl ===
 
 
 
Check out bioperl-live from my developer account:
 
 
 
  $ cd /usr/local/wormbase/build
 
  $ svn co svn+ssh://USERNAME@dev.open-bio.org/home/svn-repositories/bioperl/bioperl-live/trunk bioperl-live
 
  $ mv bioperl-live bioperl-live-tharris
 
  $ cd bioperl-live-tharris
 
  $ perl ./Build.PL install_base /usr/local/wormbase/extlib
 
  $ ./Build test
 
  $ ./Build install
 
 
 
=== Installing additional Perl modules ===
 
 
 
To install Perl modules that are not included on this list (or are new dependencies):
 
 
 
1. Your CPAN is configured to install to /usr/local/wormbase/extlib
 
   
 
You can set this in ~/.cpan/CPAN/MyConfig.pm
 
 
 
'makepl_arg' => q[INSTALL_BASE=/usr/local/wormbase/extlib],
 
 
OR
 
 
 
2. If building by hand, you call Makefile.PL as:
 
 
 
  perl Make.PL INSTALL_BASE=/usr/local/wormbase/extlib
 
 
 
 
 
 
 
= Generic Genome Browser =
 
 
 
GBrowse 2.x
 
 
 
$ cd /usr/local/wormbase/build
 
$ cvs -d:pserver:anonymous@gmod.cvs.sourceforge.net:/cvsroot/gmod login
 
$ cvs -d:pserver:anonymous@gmod.cvs.sourceforge.net:/cvsroot/gmod co Generic-Genome-Browser
 
 
 
$ mv Generic-Genome-Browser Generic-Genome-Browser-head-anonymous
 
$ ln -s Generic-Genome-Browser-head-anonymous development-Generic-Genome-Browser
 
$ perl ./Build.PL --install_base /usr/local/wormbase/gbrowse2/extlib
 
 
 
$ ./Build reconfig
 
NOTE: Run ./Build reconfig to change existing configuration.
 
Reuse previous configuration as defaults? [y ]
 
 
 
**Beginning interactive configuration**
 
Directory for GBrowse's config and support files? [/usr/local/wormbase/website-classic/conf/gb2]
 
 
 
    // DO NOT CHANGE: CGI::Toggle has hard-coded paths
 
Directory for GBrowse's static images & HTML files? [/usr/local/wormbase/website-classic/html/gbrowse2]
 
Directory for GBrowse's temporary data [/usr/local/wormbase/tmp/gbrowse2]
 
Directory for GBrowse's example databases [/usr/local/wormbase/website-classic/html/gbrowse2]
 
Apache CGI scripts directory? [/usr/local/wormbase/website-classic/cgi-perl/seq]    // Will automatically create a gb2/ subdirectory
 
Internet port to run demo web site on (for demo)? [8000]
 
Apache loadable module directory (for demo)? [/usr/lib/apache2/modules]
 
User account under which Apache daemon runs? [www-data]
 
Automatically update Apache config files to run GBrowse? [y] n
 
 
 
This will install all the GBrowse libraries into my shared library path under gbrowse.
 
 
 
  $ ./Build test
 
  $ ./Build install
 
 
 
<pre>
 
[tharris@wb-dev: GBrowse-1.988-2nGezp]> ./Build apache_conf
 
NOTE: Run ./Build reconfig to change existing configuration.
 
 
 
INSTRUCTIONS: Paste the following into your Apache configuration
 
file. You may wish to save it separately and include it using the
 
Apache "Include /path/to/file" directive. Then restart Apache and
 
point your browser to http://your.site/gb2/ to start browsing the
 
sample genomes.
 
 
 
>>>>>> cut here <<<<<
 
Alias        "/gb2/i/" "/var/tmp/gbrowse2/images/"
 
Alias        "/gb2"    "/usr/local/wormbase/website-classic/html/gb2"
 
ScriptAlias  "/gb2"      "/usr/local/wormbase/website-classic/cgi-perl/gb2/gb2"
 
 
 
<Directory "/usr/local/wormbase/website-classic/html/gb2">
 
  Options -Indexes -MultiViews +FollowSymLinks
 
</Directory>
 
 
 
<Directory "/usr/local/wormbase/website-classic/cgi-perl/gb2/gb2">
 
  SetEnv PERL5LIB "/usr/local/wormbase/extlib/lib:/usr/local/wormbase/extlib:/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/extlib/lib/perl5"
 
  SetEnv GBROWSE_CONF  "/usr/local/wormbase/website-classic/conf/gb2"
 
</Directory>
 
 
 
 
 
<IfModule mod_fastcgi.c>
 
  Alias /fgb2 "/usr/local/wormbase/website-classic/cgi-perl/gb2/gb2"
 
  <Location /fgb2>
 
    SetHandler  fastcgi-script
 
  </Location>
 
  FastCgiConfig -initial-env PERL5LIB=/usr/local/wormbase/extlib/lib:/usr/local/wormbase/extlib:/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/extlib/lib/perl5 -initial-env GBROWSE_CONF=/usr/local/wormbase/website-classic/conf/gb2
 
</IfModule>
 
 
 
<IfModule mod_perl.c>
 
  Alias /mgb2 "/usr/local/wormbase/website-classic/cgi-perl/gb2/gb2"
 
  PerlSwitches -I/usr/local/wormbase/extlib/lib -I/usr/local/wormbase/extlib -I/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi -I/usr/local/wormbase/extlib/lib/perl5
 
  <Location /mgb2>
 
    SetHandler perl-script
 
    PerlResponseHandler ModPerl::Registry
 
    PerlOptions +ParseHeaders
 
  </Location>
 
</IfModule>
 
</pre>
 
 
 
 
 
GBrowse 1.x stable
 
 
 
  $ cd /usr/local/wormbase/build
 
  $ cvs -d :pserver:anonymous@gmod.cvs.sourceforge.net:/cvsroot/gmod login
 
  $ cvs -d :pserver:anonymous@gmod.cvs.sourceforge.net:/cvsroot/gmod co -r stable Generic-Genome-Browser
 
  $ ln -s Generic-Genome-Browser-stable-anonymous
 
  $ perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/extlib
 
    Apache root directory (enter '' for none)?
 
    Apache conf directory? [/etc/apache2] /usr/local/wormbase/website-classic/conf
 
    Apache htdocs directory? [/var/www] /usr/local/wormbase/website-classic/html
 
    Apache cgibin directory? [/usr/lib/cgi-bin] /usr/local/wormbase/website-classic/cgi-perl/seq
 
    GBrowse root for static files? [gbrowse] gbrowse
 
 
 
 
 
= 3rd-party Support Applications =
 
  
 
== AceDB ==
 
== AceDB ==
Line 592: Line 292:
 
  $ sudo chown root:root ~acedb/bin/*
 
  $ sudo chown root:root ~acedb/bin/*
  
=== Testing the ACeDB Installation ===
+
'''Testing the ACeDB Installation'''
  
 
At this point, you can test whether the socket server runs correctly. Provided that you have added yourself to the acedb group, you can run the following command:
 
At this point, you can test whether the socket server runs correctly. Provided that you have added yourself to the acedb group, you can run the following command:
Line 604: Line 304:
 
   // ####  Working dir=/usr/local/acedb/elegans
 
   // ####  Working dir=/usr/local/acedb/elegans
 
   // #### clientTimeout=600 serverTimeout=600 maxKbytes=0 autoSaveInterval=600
 
   // #### clientTimeout=600 serverTimeout=600 maxKbytes=0 autoSaveInterval=600
 
 
   // Server listening socket 28 created
 
   // Server listening socket 28 created
  
Line 625: Line 324:
 
  // Bye
 
  // Bye
  
=== Configuring Acedb to start automatically under xinetd ===
+
'''Configuring Acedb to start automatically under xinetd'''
  
 
Install xinetd (not standard in Debian) if you didn't already:
 
Install xinetd (not standard in Debian) if you didn't already:
Line 639: Line 338:
 
  # default: on
 
  # default: on
 
  # description: wormbase acedb database
 
  # description: wormbase acedb database
  service acedb
+
  service acedb-wormbase
 
  {
 
  {
 
         protocol                = tcp
 
         protocol                = tcp
Line 669: Line 368:
 
== MySQL ==
 
== MySQL ==
  
=== Installation ===
+
'''Installation'''
 
 
Install mysql and various libraries via apt-get if you haven't already:
 
  
  $ sudo apt-get install mysql-server-5.0 mysql-server libdbd-mysql libdbd-mysql-perl
+
Install mysql and various libraries via apt-get if you haven't already. You will need the unstable and testing repositories set up when upgrading Perl.
  
If it fails, then disable innodb by default. edit /etc/mysql/my.cnf file (uncomment the line):
+
  $ sudo apt-get install libncurses5/testing
 +
  $ sudo apt-get install mysql-common/testing libmysqlclient16 mysql-server-5.1/testing \
 +
          mysql-client-5.1/testing libdbd-mysql/testing libdbd-mysql-perl/testing
  
  #skip-innodb
+
With this installation, databases are located at '''/var/lib/mysql'''We want to relocate this to /usr/local/mysql/data to be consistent with the WormBase. We also want to able to write to this directory from the command line, so:
  $ sudo apt-get purge  mysql-server-5.0 mysql-server  
 
  $ sudo apt-get install mysql-server-5.0 mysql-server
 
  
With this installation, databases are located at '''/var/lib/mysql'''. We want to able to write to this directory from the command line, so:
+
$ sudo nano /etc/my.cnf
 
+
Set datadir to ''/usr/local/mysql/data''
  $ sudo chmod 2775 /var/lib/mysql
+
$ mkdir -p /usr/local/mysql/data
 +
  $ sudo chown -R mysql:mysql /usr/local/mysql
 +
$ sudo chmod 2775 /usr/local/mysql/data
  
 
Mysqld will automatically be setup to launch at server boot (rc3 and rc5).
 
Mysqld will automatically be setup to launch at server boot (rc3 and rc5).
  
=== Set up mysql permissions ===
+
'''Set up mysql permissions '''
  
 
  $ mysql -u root -pPASSWORD
 
  $ mysql -u root -pPASSWORD
  mysql> grant select on elegans.* to nobody@localhost;
+
  mysql> grant select on c_elegans.* to nobody@localhost;
  
 
Repeat for:  
 
Repeat for:  
 +
* autocomplete
 +
* b_malayi
 
* c_briggsae
 
* c_briggsae
 +
* c_elegans
 +
* c_elegans_gmap
 +
* c_elegans_pmap
 
* c_japonica
 
* c_japonica
 
* c_remanei
 
* c_remanei
 
* c_brenneri
 
* c_brenneri
 +
* h_bacteriophora
 
* p_pacificus
 
* p_pacificus
* b_malayi
+
* ps1010
* c_elegans_gmap
+
 
* c_elegans_pmap
+
== Generic Genome Browser ==
* autocomplete
+
 
* h_bacteriophora
+
See [[GBrowse Administration]] for details on how to install GBrowse.
 +
 
 +
= Optional 3rd-party Applications =
 +
 
 +
== nginx ==
 +
 
 +
We use nginx as a lightweight proxy server and load balancing server.  You can skip this step and simply connect directly to the Starman http server.  Otherwise, see the [[Administration:WormBase_Production_Environment#Installing_nginx|production environment documentation]] on installing and configuring nginx.
 +
 
 +
== BLAST ==
 +
 
 +
The BLAST page requires WU-BLAST. This is a closed-source derivative of NCBI's BLAST. However, WU-BLAST is free to academic users (with licensing) and is thought to have [http://blast.wustl.edu/blast/README.html#Features performance advantages] over NCBI-BLAST; it can be downloaded from http://blast.wustl.edu/. A typical choice of WU-BLAST for Linux is blast2.linux26-i686.tar.gz.
 +
 
 +
You can skip this step if you don't want to provide BLAST services at your site.
 +
 
 +
By default, WormBase expects WU-BLAST to be installed in /usr/local/wormbase/services/wublast. This is the directory structure used by WormBase:
 +
 
 +
$ ls -l /usr/local/wormbase/services/wublast
 +
 
 +
lrwxrwxrwx  1 root  root    18 May  7 12:26  BLOSUM62 -> matrix/aa/BLOSUM62
 +
-rw-r--r--  1 root  root  46789 Feb  5  1998  HISTORY
 +
-rw-r--r--  1 root  root  6648 Mar  4  1997  README
 +
drwxr-xr-x  2 root  root  4096 May  7  12:46 bin/
 +
lrwxrwxrwx  1 root  root    25 Jul 24  08:20 databases -> /usr/local/wormbase/blast/
 +
drwxr-xr-x  2 root  root  4096 Jan 27  2000  filter/
 +
drwxr-xr-x  4 root  root  4096 Oct  4  1998  matrix/
  
 
== e-PCR ==
 
== e-PCR ==
  
* e-PCR (modified version, required for e-PCR search page)
+
A modified version of the epcr source can be found in the checked out website code in src/. See the enclosed README describing changes made to the original e-PCR distribution.  The binary should be placed in /usr/local/wormbase/services.
 
 
This is located in the directory /usr/local/wormbase/e-PCR, which will come into existence after the WormBase site update program wb_update_wormbase.pl has been successfully run (see [[HowTo_Build_A_WormBase_Mirror#Using_wb_update_wormbase.pl_to_update_the_installation|below]] for details). Once the directory has been generated, run:
 
  
     $ cd /usr/local/wormbase/e-PCR
+
     $ cd /usr/local/wormbase/website/production/src
    # Edit 'makefile' to run install rather than ginstall, which doesn't exist on Fedora Linux
 
 
     $ make
 
     $ make
     $ make install  # or just run 'install e-PCR /usr/local/bin'
+
     $ install e-PCR /usr/local/wormbase/services/epcr
 
 
The file /usr/local/wormbase/e-PCR/README-Wormbase describes the changes that were made to the original e-PCR distribution.
 
  
 
== BLAT ==
 
== BLAT ==
Line 721: Line 445:
 
Jim Kent's BLAT (blast-like alignment tool) is a fast nucleotide aligner used by the blast search page. If you do not plan to support blast searches, you may safely skip this step.
 
Jim Kent's BLAT (blast-like alignment tool) is a fast nucleotide aligner used by the blast search page. If you do not plan to support blast searches, you may safely skip this step.
  
<nowiki># mkdir -p /usr/local/blat/bin ; cd /usr/local/blat/bin
+
  $ mkdir -p /usr/local/wormbase/services/blat/bin
% wget </nowiki>http://www.soe.ucsc.edu/~kent/exe/linux/blatSuite.33.zip  (for Intel Linux)
+
  $ cd /usr/local/wormbase/services/blat/bin
% unzip blatSuite.33.zip
+
  $ wget http://www.soe.ucsc.edu/~kent/exe/linux/blatSuite.33.zip  (for Intel Linux)
% rm blatSuite.33.zip version.doc 11.ooc
+
  $ unzip blatSuite.33.zip
 +
  $ rm blatSuite.33.zip version.doc 11.ooc
  
 
Note that this choice gives precompiled binaries for an Intel-based Linux distribution as of March 2006. It would probably be worth checking http://www.soe.ucsc.edu/~kent/exe/linux to see if there is a more up-to-date version than 33. Also, other operating systems will need other binaries. E.g., for Mac OS X, instead run:
 
Note that this choice gives precompiled binaries for an Intel-based Linux distribution as of March 2006. It would probably be worth checking http://www.soe.ucsc.edu/~kent/exe/linux to see if there is a more up-to-date version than 33. Also, other operating systems will need other binaries. E.g., for Mac OS X, instead run:
  
   % wget http://www.soe.ucsc.edu/~kent/exe/osX/blatSuite.33.zip
+
   $ wget http://www.soe.ucsc.edu/~kent/exe/osX/blatSuite.33.zip
  
For other types of operating systems (e.g., Linux on Opteron-based machines), see http://www.soe.ucsc.edu/~kent/exe/ for the available choices.
+
The blat server will be started automatically by the update script. For reference, the blat server is launched using the following command.
  
The blat server will be started automatically by the update script. For reference, the blat server is launched using the following command.
+
$ /usr/local/wormbase/services/blat/bin/gfServer start localhost 2003 \
 +
      /usr/local/wormbase/databases/VERSION/blat/*.nib &amp; &gt; /dev/null 2&gt;&amp;1
 +
 
 +
= Installing Databases NOT DONE =
 +
 
 +
== Primary database (AceDB) ==
 +
 
 +
== GFF Sequence feature database (MySQL) ==
 +
 
 +
== Support Databases ==
 +
 
 +
Support databases are automatically synced to the nodes that require them by wormbase-admin/update/production/cron-rsync_support_dbs.sh. This is run from wb-dev.
 +
 
 +
0 1 * * * /home/tharris/projects/wormbase/wormbase-admin/update/production/cron-rsync_support_dbs.sh
 +
 
 +
= Testing The Site =
  
% /usr/local/blat/bin/gfServer start localhost 2003 \
+
At this point, all components of a WormBase installation have been installed. You can test your installation by restarting the various server components of WormBase.
      /usr/local/wormbase/blat/*.nib &amp; &gt; /dev/null 2&gt;&amp;1
 
  
== BLAST ==
+
=== Restarting AceDB ===
  
The Blast page requires WU-BLAST. This is a closed-source derivative of NCBI's BLAST. However, WU-BLAST is free to academic users (with licensing) and is thought to have [http://blast.wustl.edu/blast/README.html#Features performance advantages] over NCBI-BLAST; it can be downloaded from http://blast.wustl.edu/. A typical choice of WU-BLAST for Linux is blast2.linux26-i686.tar.gz.
+
<nowiki># Via xinetd:
 +
$ /etc/init.d/xinetd reload (or restart)
 +
</nowiki>
  
Conversely, the Blast page can be deactivated if you don't want to provide BLAST searches at your site.
+
<nowiki># ...or using saceclient
 +
% saceclient localhost -port 2005
 +
acedb&gt; password:
 +
acedb&gt; shutdown now
 +
</nowiki>
  
By default, WormBase expects WU-BLAST to be installed in /usr/local/wublast. This is the directory structure used by WormBase:
+
=== Restarting MySQL ===
  
  % ls -l /usr/local/wublast
+
  <nowiki># Via mysqladmin...
  ls -l /usr/local/wublast
+
  % mysqladmin -uroot -pPASSWORD shutdown
total 72
+
   
lrwxrwxrwx  1 root  root    18 May  7 12:26  BLOSUM62 -&gt; matrix/aa/BLOSUM62
+
  # or using init.d
  -rw-r--r--  1 root  root  46789 Feb  5  1998  HISTORY
+
  $ /etc/init.d/mysql restart
  -rw-r--r--  1 root  root  6648 Mar  4  1997  README
+
  </nowiki>
drwxr-xr-x  2 root  root  4096 May  7  12:46 bin/
+
 
  lrwxrwxrwx  1 root  root    25 Jul 24  08:20 databases -&gt; /usr/local/wormbase/blast/
+
=== Starting The Webapp ===
  drwxr-xr-x  2 root  root  4096 Jan 27  2000  filter/
+
 
drwxr-xr-x  4 root  root  4096 Oct  4  1998  matrix/
+
Set up environment:
  
which can be set up in this manner (adapt to your system):
+
$ cd /your/checked/out/source
 +
$ cp wormbase.env.template wormbase.env  // Edit as necessary.
 +
$ source wormbase.env
  
$ cd /usr/local/wublast
+
Start the built in development server:
$ zcat /usr/local/TGZ/blast2.linux26-i686.tar.gz | tar xf -
 
$ chown -R root:root *
 
$ mkdir bin
 
$ mv *fasta tblast* blast* *db xd* memfile pam wu-blastall bin
 
$ ln -s /usr/local/wormbase/blast databases
 
  
The important thing to note is that the databases directory is a symbolic link to /usr/local/wormbase/blast. This is where the update_wormbase.pl script (described in the next section) dumps its BLAST databases.
+
$ script/wormbase_server -p 9999 -r -d  // choose a suitable port.
  
= The WormBase Software =
+
Start up a starman server via plack:
  
Check out the WormBase software from CVS:
+
$ cd /your/checked/out/source
 +
$ source wormbase.env
 +
$ starman -- port 9999 wormbase.psgi // choose a suitable port.
  
  $ cd /usr/local/wormbase
+
At this point, you should now be able to connect to WormBase at:
  $ cvs -d formaggio.cshl.org:/usr/local/cvs_repository co wormbase-website
 
  $ mv wormbase-website website-classic
 
  
== Configure localsdef.pm ==
+
http://localhost:YOURPORT/
  
* $HOST
+
If you are running your site on one of the ports matching the nginx reverse proxy configuration as described above, you should be able to:
  
This is the name of the host where the socket server runs. It is set to "localhost" by default.
+
http://subdomain.wormbase.org/
  
* $PORT
+
To run your site using the init script:
  
This is the port on which the socket server runs, 2005 by default.
+
source wormbase.env
 +
./script/wormbase-init.sh start
  
* $ACEPASS, $USERNAME, $PASSWORD
+
= Administration =
  
These three items define the acedb username and password.
+
'''Checking out the Administration module'''
  
* $MYSQL_HOST, $MYSQL_USER, $MYSQL_PASS
+
Fetch the administration code repository (privileged users only, for now)
  
These three items define the mysql host, username, and password.
+
todd> mkdir -p ~/projects/wormbase
 +
todd> cd ~/projects/wormbase
 +
todd> hg clone ssh://hg@bitbucket.org/tharris/wormbase-admin/  // Assuming you have already set up your ssh keys at bitbucket
  
* $MASTER
+
== Updating the web application ==
  
This is used only for the WormBase master site. Should be set to 0.
+
== Updating databases ==
  
* $MIRROR
+
== Backing up your system ==
  
Whether or not the site is a mirror. Should be set to the name of the mirror.
 
  
* $DEVELOPMENT
 
  
Whether or not the site is a development site. Internally, this controls the nature of caching on the site. Should be set to 0.
+
== Cronjobs ==
  
* $BLAST2WORMBASE, $WORMBASE2BLAST
+
''Set up the following cronjobs:''
  
These two options control where the blast script directs queries, and where those queries are returned. This is provided in the event that a second standalone blast server is provided. If not, these two options should point to:
+
'''Log Rotation (root cron)'''
  
  $WORMBASE2BLAST=http://your.hostname.org/
+
  # Rotate WormBase logs
 +
10 1 * * * /home/todd/projects/wormbase/admin/maintenance/rotate_httpd_logs.pl
  
 +
Or via logrotate:
  
 +
00 00 * * * /usr/sbin/logrotate -s /var/tmp/logrotate.status /usr/local/wormbase/admin/crons/wormbase-logrotate.conf
  
 +
The contents of this file look like this:
  
 +
/usr/local/wormbase/logs/*_log {
 +
        daily
 +
        missingok
 +
        rotate 3650  # keep 10 years worth of logs
 +
        compress
 +
        notifempty
 +
        sharedscripts
 +
        mail wormbase-logs@tharris.org  # when a log file expires, email it to me.
 +
        postrotate
 +
                /usr/local/apache2/bin/apachectl graceful
 +
        endscript
 +
}
  
  
 +
For nodes running acedb, add the following entry which deals with its massive log files:
  
 +
# Purge epic ACEDB logs
 +
35 * * * * /home/todd/projects/wormbase/admin/maintenance/purge_acedb_logs.sh
  
= Configuring Servers To Start Automatically =
+
Watch for runaway httpd processes:
 +
*/5 * * * * /usr/local/wormbase/admin/monitoring/monitor_httpd_runaways.sh
  
 
The final step is to arrange for Acedb to start automatically and for MySQL to restart if necessary.
 
The final step is to arrange for Acedb to start automatically and for MySQL to restart if necessary.
  
=== Installing MySQL and BLAT monitoring scripts ===
+
'''Installing MySQL and BLAT monitoring scripts NOT DONE'''
  
 
Run:
 
Run:
Line 836: Line 599:
 
     0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl
 
     0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl
 
     0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl
 
     0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl
 
=== Acedb log rotation ===
 
 
Acedb generates massive log files. To keep these from growing too large, add the following entry to root's crontab (or that of another privileged user):
 
 
    10 1 * * * /usr/local/wormbase/bin/rotatelogs.pl
 
 
 
  
 
== Installing scripts to verify that the servers are running ==
 
== Installing scripts to verify that the servers are running ==
Line 869: Line 624:
 
This stanza will check that the acedb server logs do not grow larger than 100 MB.
 
This stanza will check that the acedb server logs do not grow larger than 100 MB.
  
 +
== Blocking robots ==
 +
 +
It can be useful to block search engines (such as Google) from crawling over one's mirror. To do this, go to /usr/local/wormbase/html, and make a file called "robots.txt" with the following contents:
 +
 +
    User-agent: *
 +
    Disallow: /
  
 +
= Appendices =
  
 +
== Appendix 1: Mac OS 10.6 (Snow Leopard) Prequisites  ==
  
 +
=== XCode > 3.21 ===
  
= Testing The Site =
+
''You will need this for building C libraries as well as fixing the broken Perl that shipping with 10.6.''
 +
 
 +
=== MySQL >= 5.1 ===
 +
 
 +
''MySQL is not expressly required, but you do need it for building DBD::mysql.''
 +
 
 +
bash> cd src
 +
bash> curl -O http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.39.tar.gz
 +
bash> tar xzf mysql-5.1.41.tar.gz
 +
bash> cd mysql-5.1.41
 +
bash> ./configure --prefix=/usr/local/mysql \
 +
                  --with-extra-charsets=complex \
 +
                  --enable-thread-safe-client \
 +
                  --enable-local-infile \
 +
                  --enable-shared \
 +
                  --with-plugins=innobase
 +
bash> make
 +
bash> sudo make install
 +
bash> cd /usr/local/mysql
 +
bash> sudo ./bin/mysql_install_db --user=mysql
 +
bash> sudo chown -R mysql ./var
 +
 
 +
=== libpng ===
 +
 
 +
bash> curl -O http://softlayer.dl.sourceforge.net/project/libpng/00-libpng-stable/1.2.40/libpng-1.2.40.tar.gz
 +
bash> tar xzf libpng-*
 +
bash> ccd libpng*
 +
bash> ./configure
 +
bash> make
 +
bash> sudo make install
 +
 
 +
=== libgd ===
 +
 
 +
bash> curl -O http://www.libgd.org/releases/gd-2.0.35.tar.gz
 +
bash> tar xzf gd*
 +
bash> cd gd*
 +
bash> ./configure
 +
bash> make
 +
bash> sudo make install
 +
 
 +
== Appendix 2: mod_perl and Apache installation ==
 +
 
 +
WormBase no longer runs off apache and mod_perl, although it is entirely possible to do so.  Here's how to install these components.
  
At this point, all components of a WormBase installation have been installed. You can test your installation by restarting the various server components of WormBase.
+
''I prefer to build httpd and mod_perl from source; it seems easier than waiting for package repositories to be updated.  Besides, I really despise Debian's apache layout. It's horrid.''
  
=== Restarting AceDB ===
+
=== Installation ===
  
<nowiki># Via xinetd:
+
''Assuming you have already fetched the source into ~/src:''
$ /etc/init.d/xinetd reload (or restart)
 
</nowiki>
 
  
  <nowiki># ...or using saceclient
+
  # Build httpd 2.2.11
  % saceclient localhost -port 2005
+
cd ~/src
  acedb&gt; password:
+
tar xzf httpd-2.2.11.tar.gz
  acedb&gt; shutdown now
+
  cd httpd-2.2.11
  </nowiki>
+
  ./configure --enable-mods-shared=all --enable-proxy
 +
  make
 +
  sudo make install
  
=== Restarting MySQL ===
+
# Build mod_perl2-0.4
 +
cd ~/src
 +
cd mod_perl-2.0.4
 +
perl -I/usr/local/wormbase/website-classic/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/website-classic/extlib
 +
// When prompted for the apxs path, enter: /usr/local/apache2/bin/apxs
 +
make
 +
sudo make install
  
<nowiki># Via mysqladmin...
+
=== Configuration ===
% mysqladmin -uroot -pPASSWORD shutdown
 
 
# or using init.d
 
$ /etc/init.d/mysql restart
 
</nowiki>
 
  
=== Restarting Apache ===
+
''We typically set up services as individual virtual servers.  Here's an example.  See "Special Services" below for additional details.''
  
When the configuration files have been checked and adjusted, restart Apache with the following command:
+
Modify the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) as follows:
  
  $ /etc/init.d/apache restart
+
#User daemon
 +
User nobody
  
Check /usr/local/wormbase/logs/classic-error_log for WormBase-specific errors and /var/log/apache2/error_log for general errors.
+
Set up a virtual host on your port.  Adjust PerlSwitches as appropriate for your architecture, and the Listen directive of the primary apaache httpd.conf file as appropriate.
  
=== BLAT ===
+
<VirtualHost *:80>
 +
  Include /usr/local/wormbase/website-classic/conf/httpd.conf
 +
  PerlOptions +Parent
 +
  PerlSwitches -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5 \
 +
                -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5/x86_64-linux-gnu-thread-multi
 +
  #  PerlInterpStart 2                                                                                               
 +
    #  PerlInterpMax 2                                                                                             
 +
  <Directory "/usr/local/wormbase/website-classic/html">
 +
        DirectoryIndex index.html
 +
        #      Order deny,allow       
 +
        #      Deny from all                                                                                           
 +
  </Directory>
 +
  # Redirect requests for the old movable type atom feeds to feedburner
 +
  RewriteEngine ON
 +
  #  RewriteRule ^/rss/wormbase-live-atom\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
 +
  RewriteRule ^/rss/wormbase-live-atom\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
 +
  RewriteRule ^/news\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
 +
  RewriteRule ^/rss/wormbase-live\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
 +
</VirtualHost>
  
% /usr/local/blat/bin/gfServer start localhost 2003 \
+
Enable virtual hosts in httpd.conf by uncommenting the following line:
      /usr/local/wormbase/blat/*.nib &amp; &gt; /dev/null 2&gt;&amp;1
 
  
== Blocking robots ==
+
# Include conf/extra/httpd-hosts.conf
  
It can be useful to block search engines (such as Google) from crawling over one's mirror. To do this, go to /usr/local/wormbase/html, and make a file called "robots.txt" with the following contents:
+
=== Set up httpd to run under inet.d ===
  
    User-agent: *
+
cd /etc/init.d
    Disallow: /
+
sudo ln -s /usr/local/apache2/bin/apachectl httpd
  
=Updating Production Nodes to match this reality=
+
Edit apachectl to make it "LSB" compliant (ie Debian compliant). Add the following after the she-bang:
  
1. Create /usr/local/wormbase/website-classic
+
### BEGIN INIT INFO                                                           
 +
# Provides:          httpd                                                     
 +
# Required-Start:    $all                                                     
 +
# Required-Stop:    $remote_fs $syslog                                       
 +
# Default-Start:    3 5                                                   
 +
# Default-Stop:      0 1 2 4 6                                                     
 +
# Short-Description: Start daemon at boot time                                 
 +
# Description:      Enable service provided by daemon.                        
 +
### END INIT INFO
  
2. Move all website files into website-classic
+
cd /etc/rc3.d
 +
sudo ln -s ../init.d/apache2 S99apache2
 +
cd ../rc5.d
 +
sudo ln -s ../init.d/apache2 S99apache2
  
3. Move database to /usr/local/wormbase
+
Or via update-rc.d
  
4. Create /usr/local/wormbase/logs
+
sudo update-rc.d apache2 start 99 3 5 . stop 0 1 2 4 6 .
  
5. Install Perl Modules as described above
 
  
6. Install GBrowse as described above
+
=== PHP ===
  
7. Update admin module for pushing software to production (and for maintaining staging module)
+
cd ~/src
 +
tar xzf php-5.2.4.tar.gz
 +
./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql
 +
make
 +
sudo make install
  
 
= AUTHOR =
 
= AUTHOR =
  
[[User:Tharris|Todd Harris]] (toddwharris@gmail.com)
+
[[User:Tharris|Todd Harris]] (todd@wormbase.org)
 +
 
 +
 
 +
[[Category: Architecture (Web Dev)]]

Latest revision as of 19:10, 18 June 2014

Overview

This document describes how to install WormBase locally on your own machine. Please note that as of August 2013, WormBase is almost entirely a distributed cloud-based application. Although possible to install and run on a single server (or cloud instance) the directions below may be slightly out-of-sync with how the principal WormBase architecture is built.

Recommendations

Hardware

As of April 2012, the suggested minimum hardware requirements for a basic installation of WormBase are:

  • 500 GB disk space (1 TB preferable)
  • 8 GB RAM

Operating Systems

You should be able to install and run the site on any Unix-y OS, including Mac OS X. This document only describes installation on Debian and Debian-ish OSes like Amazon's Linux. Your actual mileage may vary on other operating systems.

Basic System Configuration

Users And Groups

These users should not have a login password. They are to establish privileges only.

acedb group

This is the group that will have write privileges to the acedb directory tree. Acedb administrators should be added to this group.

$ groupadd acedb

acedb user

This is the user that the acedb server will run as. It should be a member of the acedb group.

$ useradd -g acedb -d /usr/local/wormbase/acedb acedb

This useradd command also adds the new acedb user to the acedb group. Note that the acedb user's home directory was set to /usr/local/acedb, a directory which will be created in the next step.

wormbase group

This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group.

$ groupadd wormbase

This would be a good time to add yourself to the acedb and wormbase groups.

$ usermod -a -G acedb,wormbase [your_login_name]

You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of:

% groups

Create Directories

Create a root directory for all WormBase-related files. Although this directory can reside anywhere on your filesystem, for consistency with WormBase proper, we recommend installing at /usr/local/wormbase. You'll need suitable privileges to install in this location.'

The root container for all things WormBase

  • /usr/local/wormbase, owner=root group=wormbase mode=drwxrwsr-x
$ cd /usr/local
$ sudo mkdir wormbase
$ sudo chown root:wormbase wormbase
$ sudo chmod 2775 wormbase

The AceDB directory contains acedb and associated databases:

  • /usr/local/wormbase/acedb, owner=acedb group=acedb,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/acedb
$ chmod 2775 /usr/local/wormbase/acedb
$ sudo chown acedb:acedb /usr/local/wormbase/acedb

The website/ directory contains one or many versions of the website:

  • /usr/local/wormbase/website, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/website
$ chmod 2775 /usr/local/wormbase/website
$ sudo chgrp wormbase /usr/local/wormbase/website

The services/ directory contains optional services such as e-pcr, blast, and nginx:

  • /usr/local/wormbase/services, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/services
$ chmod 2775 /usr/local/wormbase/services
$ sudo chgrp wormbase /usr/local/wormbase/services

The databases/ directory contains databases driving a number of features:

  • /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/databases
$ chmod 2775 /usr/local/wormbase/databases
$ sudo chgrp wormbase /usr/local/wormbase/databases

The logs/ directory contains WormBase-wide log and pid files

  • /usr/local/wormbase/logs, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx
$ mkdir /usr/local/wormbase/logs
$ chmod 2775 /usr/local/wormbase/logs
$ sudo chgrp wormbase /usr/local/wormbase/logs

Installing Library Prerequisites

Assuming a vanilla Debian installation, install the following libaries and all of their dependencies via sudo apt-get install. Some of these libraries are discussed below.

     sharutils \
     gcc g++ \
     curl wget \
     bzip2 \
     mysql-client \
     mysql-server \
     libgd2-xpm-dev\
     libgd2-xpm \
     xinetd \
     libdbd-mysql \
     libdbd-mysql-perl \
     libapache2-mod-perl2 \
     libgtk2.0-0 \
     libgtk2.0-dev \
     emacs \
     git \
     byacc \
     libreadline-gplv2-dev \
     flex \
     libdb4.6 \
     libdb-dev \
     xml2 libxml2 libxml2-dev \
     libmysql++-dev \
     ncurses-bin \
     libxmu-dev \
     graphviz \
     libssl-dev \
     uuid \
     uuid-dev \
     libxslt1-dev \
     xsltproc \
     libgdbm3 \
     libgdbm3-dev \
     libstdc++ \
     libstdc++-devel 

Build xapian (or install from package if available):

$ wget http://oligarchy.co.uk/xapian/1.2.5/xapian-core-1.2.5.tar.gz
$ tar xzf xapian-core*.tgz
$ cd xapian-core
$ ./configure
$ make
$ sudo make install

Install Search::Xapian

$ perl -MCPAN -e 'install Search::Xapian'

Install Perl 5.10.1 or greater

You'll need Perl version 5.10.1 or greater.

Upgrading Perl via apt (recommended)

Let your operating system managing your version of Perl.

Configure your apt preferences

Add the following to /etc/apt/sources.list

#Stable
deb http://ftp.us.debian.org/debian stable main non-free contrib
#Testing
deb http://ftp.us.debian.org/debian testing main non-free contrib
#Unstable
deb http://ftp.us.debian.org/debian unstable main non-free contrib

Configure your preferences in /etc/apt/preferences:

Package: *
Pin: release a=stable
Pin-Priority: 700
Package: *
Pin: release a=testing
Pin-Priority: 650
Package: *
Pin: release a=unstable
Pin-Priority: 600

Finally, install Perl

sudo apt-get update
sudo apt-get install perl/testing perl-base/testing perl-modules/testing libperl5.10/testing libperl-dev/testing

And confirm

$ perl -v
 This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
 (with 40 registered patches, see perl -V for more detail)

Installing Perl from source

We recommend installing a local version of Perl and required libraries. This makes management dramatically easier. We keep our Perl with the webapp itself.

If you have sufficient privileges, you may wish to use your pre-installed system Perl.

bash> mkdir ~/website/perl
bash> cd ~/src
bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz
bash> cd ~/build
bash> tar xzf ../src.perl-5.10.1.tar.gz
bash> cd perl-5.10.1
bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1
bash> make
bash> make test
bash> make install

Be sure to set up your your system to preferentially use this newly installed Perl:

bash> export PATH=~/wormbase/perl/5.10.1/bin:${PATH}

Install the Webapp

Fetch the current production code from the git repository on github.

$ cd /usr/local/wormbase/website
$ git clone git@github.com:WormBase/website.git
$ mv website YOUR_NAME // name can be anything you want; useful for maintaining multiple versions

Install local:: lib

The preferred method uses the web app's Makefile.PL and Module::Install to build all dependencies. We maintain local copies of all libraries with each version of the webapp.

Build and install dependencies

$ mkdir /usr/local/wormabase/extlib
$ cd /usr/local/wormbase/website/YOUR_NAME
$ perl Makefile.PL
$ make installdeps

See the documentation on Managing Perl Libraries for additional details.

Required 3rd-party Applications

Starman: the lightweight http server

PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).

  http://plackperl.org/

Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.mod_perl.

 https://github.com/miyagawa/Starman

Install Plack and Starman

Catalyst::Controller::Metal
Catalyst::Engine::PSGI
Catalyst::Helper::PSGI
Plack::Test::Adopt::Catalyst
cpanm Task::Plack
cpanm Starman

Configuration

The PSGI glue resides at:

website/YOUR_NAME/wormbase.psgi

Starting Starman

starman script/wormbase.psgi
OR
starman -MFindBin script/wormbase_production.psgi

Set up starman to launch automatically

Copy the stub init script from wormbase/conf/starman/starman.init:

 cp /usr/local/wormbase/admin/conf/starman/starman.init

AceDB

I always build acedb from source.

$ tar xzf ACEDB-source*    // CAUTION: Tarbomb.
// Requires installation of a whole bunch of things first: libgtk2.0-0 libgtk2.0-dev libglib, byacc, etc, etc
// Modify the makefile: create a target for server programs (xace tace saceserver sgifacerver)
// This is all I care about:
   SERVERS = xace tace saceserver sgifaceserver saceclient
   servers: $(SERVERS)
$ export ACEDB_MACHINE=LINUX_4
$ make servers
$ cd ~acedb
$ ln -s bin-VERSION bin 
$ cp tace xace sgifaceserver saceserver saceclient ~acedb/bin/.
$ sudo chown root:root ~acedb/bin/*

Testing the ACeDB Installation

At this point, you can test whether the socket server runs correctly. Provided that you have added yourself to the acedb group, you can run the following command:

 % ~acedb/bin/sgifaceserver ~acedb/wormbase
 // Database directory: /usr/local/wormbase/acedb/wormbase
 // Shared files: /usr/local/acedb
 // #### Server started at 2001-07-23_16:42:31
 // #### host=mondseer.cshl.org  listening port=23100
 // #### Database dir=/usr/local/acedb/elegans
 // ####  Working dir=/usr/local/acedb/elegans
 // #### clientTimeout=600 serverTimeout=600 maxKbytes=0 autoSaveInterval=600
 // Server listening socket 28 created

The line "listening port=23100" indicates that the server is listening to port 23100. Open a new terminal window and use saceclient to confirm that you can communicate with the server:

% ~acedb/bin/saceclient localhost -port 23100
Please enter userid: anonymous
Please enter passwd:
acedb@localhost> find Sequence
// Response: 65 bytes.
// Found 236493 objects in this class
// 236493 Active Objects
acedb@localhost> quit
// Closing connection to server.
// Client sent termination signal by server.
// Response: 13 bytes.
// A bientot
// Please report problems to acedb@sanger.ac.uk
// Bye

Configuring Acedb to start automatically under xinetd

Install xinetd (not standard in Debian) if you didn't already:

  $ sudo apt-get install xinetd

Create a configuration file for acedb:

 $ sudo emacs /etc/xinetd.d/acedb-wormbase 

 # file: /etc/xinetd.d/acedb-wormbase
 # default: on
 # description: wormbase acedb database
 service acedb-wormbase
 {
        protocol                = tcp
        socket_type             = stream
        port                    = 2005
        flags                   = REUSE
        wait                    = yes
        user                    = acedb
        group                   = acedb
        log_on_success          += USERID DURATION
        log_on_failure          += USERID HOST
        server                  = /usr/local/wormbase/acedb/bin/sgifaceserver
        server_args             = /usr/local/wormbase/acedb/wormbase 1200:1200:0
 }
 

Edit /etc/services. Although xinetd is not supposed to use /etc/services, the following line must be added:

acedb-wormbase           2005/tcp

Restart xinetd with the following command:

$ sudo /etc/init.d/xinetd reload (or restart)

You should now be able to talk to the database using saceclient:

$ ~acedb/bin/saceclient localhost -port 2005

MySQL

Installation

Install mysql and various libraries via apt-get if you haven't already. You will need the unstable and testing repositories set up when upgrading Perl.

 $ sudo apt-get install libncurses5/testing
 $ sudo apt-get install mysql-common/testing libmysqlclient16 mysql-server-5.1/testing \
         mysql-client-5.1/testing libdbd-mysql/testing libdbd-mysql-perl/testing

With this installation, databases are located at /var/lib/mysql. We want to relocate this to /usr/local/mysql/data to be consistent with the WormBase. We also want to able to write to this directory from the command line, so:

$ sudo nano /etc/my.cnf
Set datadir to /usr/local/mysql/data
$ mkdir -p /usr/local/mysql/data
$ sudo chown -R mysql:mysql /usr/local/mysql
$ sudo chmod 2775 /usr/local/mysql/data 

Mysqld will automatically be setup to launch at server boot (rc3 and rc5).

Set up mysql permissions

$ mysql -u root -pPASSWORD
mysql> grant select on c_elegans.* to nobody@localhost;

Repeat for:

  • autocomplete
  • b_malayi
  • c_briggsae
  • c_elegans
  • c_elegans_gmap
  • c_elegans_pmap
  • c_japonica
  • c_remanei
  • c_brenneri
  • h_bacteriophora
  • p_pacificus
  • ps1010

Generic Genome Browser

See GBrowse Administration for details on how to install GBrowse.

Optional 3rd-party Applications

nginx

We use nginx as a lightweight proxy server and load balancing server. You can skip this step and simply connect directly to the Starman http server. Otherwise, see the production environment documentation on installing and configuring nginx.

BLAST

The BLAST page requires WU-BLAST. This is a closed-source derivative of NCBI's BLAST. However, WU-BLAST is free to academic users (with licensing) and is thought to have performance advantages over NCBI-BLAST; it can be downloaded from http://blast.wustl.edu/. A typical choice of WU-BLAST for Linux is blast2.linux26-i686.tar.gz.

You can skip this step if you don't want to provide BLAST services at your site.

By default, WormBase expects WU-BLAST to be installed in /usr/local/wormbase/services/wublast. This is the directory structure used by WormBase:

$ ls -l /usr/local/wormbase/services/wublast
lrwxrwxrwx  1 root  root     18 May  7 12:26  BLOSUM62 -> matrix/aa/BLOSUM62
-rw-r--r--  1 root  root  46789 Feb  5  1998  HISTORY
-rw-r--r--  1 root  root   6648 Mar  4  1997  README
drwxr-xr-x  2 root  root   4096 May  7  12:46 bin/
lrwxrwxrwx  1 root  root     25 Jul 24  08:20 databases -> /usr/local/wormbase/blast/
drwxr-xr-x  2 root  root   4096 Jan 27  2000  filter/
drwxr-xr-x  4 root  root   4096 Oct  4  1998  matrix/

e-PCR

A modified version of the epcr source can be found in the checked out website code in src/. See the enclosed README describing changes made to the original e-PCR distribution. The binary should be placed in /usr/local/wormbase/services.

   $ cd /usr/local/wormbase/website/production/src
   $ make
   $ install e-PCR /usr/local/wormbase/services/epcr

BLAT

Jim Kent's BLAT (blast-like alignment tool) is a fast nucleotide aligner used by the blast search page. If you do not plan to support blast searches, you may safely skip this step.

 $ mkdir -p /usr/local/wormbase/services/blat/bin
 $ cd /usr/local/wormbase/services/blat/bin
 $ wget http://www.soe.ucsc.edu/~kent/exe/linux/blatSuite.33.zip  (for Intel Linux)
 $ unzip blatSuite.33.zip
 $ rm blatSuite.33.zip version.doc 11.ooc

Note that this choice gives precompiled binaries for an Intel-based Linux distribution as of March 2006. It would probably be worth checking http://www.soe.ucsc.edu/~kent/exe/linux to see if there is a more up-to-date version than 33. Also, other operating systems will need other binaries. E.g., for Mac OS X, instead run:

 $ wget http://www.soe.ucsc.edu/~kent/exe/osX/blatSuite.33.zip

The blat server will be started automatically by the update script. For reference, the blat server is launched using the following command.

$ /usr/local/wormbase/services/blat/bin/gfServer start localhost 2003 \
     /usr/local/wormbase/databases/VERSION/blat/*.nib & > /dev/null 2>&1

Installing Databases NOT DONE

Primary database (AceDB)

GFF Sequence feature database (MySQL)

Support Databases

Support databases are automatically synced to the nodes that require them by wormbase-admin/update/production/cron-rsync_support_dbs.sh. This is run from wb-dev.

0 1 * * * /home/tharris/projects/wormbase/wormbase-admin/update/production/cron-rsync_support_dbs.sh

Testing The Site

At this point, all components of a WormBase installation have been installed. You can test your installation by restarting the various server components of WormBase.

Restarting AceDB

# Via xinetd:
 $ /etc/init.d/xinetd reload (or restart)
 
# ...or using saceclient
 % saceclient localhost -port 2005
 acedb> password:
 acedb> shutdown now
 

Restarting MySQL

# Via mysqladmin...
 % mysqladmin -uroot -pPASSWORD shutdown
 
 # or using init.d
 $ /etc/init.d/mysql restart
 

Starting The Webapp

Set up environment:

$ cd /your/checked/out/source
$ cp wormbase.env.template wormbase.env  // Edit as necessary.
$ source wormbase.env

Start the built in development server:

$ script/wormbase_server -p 9999 -r -d   // choose a suitable port.

Start up a starman server via plack:

$ cd /your/checked/out/source
$ source wormbase.env
$ starman -- port 9999 wormbase.psgi // choose a suitable port.

At this point, you should now be able to connect to WormBase at:

http://localhost:YOURPORT/

If you are running your site on one of the ports matching the nginx reverse proxy configuration as described above, you should be able to:

http://subdomain.wormbase.org/

To run your site using the init script:

source wormbase.env
./script/wormbase-init.sh start

Administration

Checking out the Administration module

Fetch the administration code repository (privileged users only, for now)

todd> mkdir -p ~/projects/wormbase
todd> cd ~/projects/wormbase
todd> hg clone ssh://hg@bitbucket.org/tharris/wormbase-admin/  // Assuming you have already set up your ssh keys at bitbucket

Updating the web application

Updating databases

Backing up your system

Cronjobs

Set up the following cronjobs:

Log Rotation (root cron)

# Rotate WormBase logs
10 1 * * * /home/todd/projects/wormbase/admin/maintenance/rotate_httpd_logs.pl

Or via logrotate:

00 00 * * * /usr/sbin/logrotate -s /var/tmp/logrotate.status /usr/local/wormbase/admin/crons/wormbase-logrotate.conf

The contents of this file look like this:

/usr/local/wormbase/logs/*_log {
        daily
        missingok
        rotate 3650  # keep 10 years worth of logs
        compress
        notifempty
        sharedscripts
        mail wormbase-logs@tharris.org   # when a log file expires, email it to me.
        postrotate
                /usr/local/apache2/bin/apachectl graceful
        endscript
}


For nodes running acedb, add the following entry which deals with its massive log files:

# Purge epic ACEDB logs
35 * * * * /home/todd/projects/wormbase/admin/maintenance/purge_acedb_logs.sh

Watch for runaway httpd processes:

  • /5 * * * * /usr/local/wormbase/admin/monitoring/monitor_httpd_runaways.sh

The final step is to arrange for Acedb to start automatically and for MySQL to restart if necessary.

Installing MySQL and BLAT monitoring scripts NOT DONE

Run:

   $ cp -i /usr/local/wormbase/util/admin/blat_server.initd /etc/rc.d/init.d/blat_server

Then run:

   $ crontab -u root -e

to add the following entries to root's crontab:

   0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl
   0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl

Installing scripts to verify that the servers are running

Two scripts in the WormBase directory can be used to ensure that the mysql and blat servers are running. To install, them:

% sudo cp /usr/localwormbase/util/admin/blat_server.initd \
          /etc/rc.d/init.d/blat_server

Place the restart scripts under cron control of a privileged user. These commands will check every hour to see that the servers are running.

 % sudo crontab -u root -e
0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl
0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl

At the same time, you might also wish to automate the rotatation of logs to prevent them from growing to an unwieldy size. You'll find an appropriate log rotation configuration stanza in util/rotate_wormbase_logs and a log rotate script in /usr/local/wormbase/bin/rotatelogs.pl. You will need both.

# Rotate httpd logs
 10 1 * * * /usr/local/wormbase/bin/rotatelogs.pl
 # Rotate acedb logs
 10 1 * * * logrotate /usr/local/wormbase/util/rotate_wormbase_logs
 

This stanza will check that the acedb server logs do not grow larger than 100 MB.

Blocking robots

It can be useful to block search engines (such as Google) from crawling over one's mirror. To do this, go to /usr/local/wormbase/html, and make a file called "robots.txt" with the following contents:

   User-agent: *
   Disallow: /

Appendices

Appendix 1: Mac OS 10.6 (Snow Leopard) Prequisites

XCode > 3.21

You will need this for building C libraries as well as fixing the broken Perl that shipping with 10.6.

MySQL >= 5.1

MySQL is not expressly required, but you do need it for building DBD::mysql.

bash> cd src
bash> curl -O http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.39.tar.gz
bash> tar xzf mysql-5.1.41.tar.gz
bash> cd mysql-5.1.41
bash> ./configure --prefix=/usr/local/mysql \
                  --with-extra-charsets=complex \
                  --enable-thread-safe-client \
                  --enable-local-infile \
                  --enable-shared \
                  --with-plugins=innobase
bash> make
bash> sudo make install
bash> cd /usr/local/mysql
bash> sudo ./bin/mysql_install_db --user=mysql
bash> sudo chown -R mysql ./var

libpng

bash> curl -O http://softlayer.dl.sourceforge.net/project/libpng/00-libpng-stable/1.2.40/libpng-1.2.40.tar.gz
bash> tar xzf libpng-*
bash> ccd libpng*
bash> ./configure
bash> make
bash> sudo make install

libgd

bash> curl -O http://www.libgd.org/releases/gd-2.0.35.tar.gz
bash> tar xzf gd*
bash> cd gd*
bash> ./configure
bash> make
bash> sudo make install

Appendix 2: mod_perl and Apache installation

WormBase no longer runs off apache and mod_perl, although it is entirely possible to do so. Here's how to install these components.

I prefer to build httpd and mod_perl from source; it seems easier than waiting for package repositories to be updated. Besides, I really despise Debian's apache layout. It's horrid.

Installation

Assuming you have already fetched the source into ~/src:

# Build httpd 2.2.11
cd ~/src
tar xzf httpd-2.2.11.tar.gz
cd httpd-2.2.11
./configure --enable-mods-shared=all --enable-proxy
make
sudo make install
# Build mod_perl2-0.4
cd ~/src
cd mod_perl-2.0.4
perl -I/usr/local/wormbase/website-classic/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/website-classic/extlib
// When prompted for the apxs path, enter: /usr/local/apache2/bin/apxs
make
sudo make install

Configuration

We typically set up services as individual virtual servers. Here's an example. See "Special Services" below for additional details.

Modify the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) as follows:

#User daemon
User nobody

Set up a virtual host on your port. Adjust PerlSwitches as appropriate for your architecture, and the Listen directive of the primary apaache httpd.conf file as appropriate.

<VirtualHost *:80>
  Include /usr/local/wormbase/website-classic/conf/httpd.conf
  PerlOptions +Parent
  PerlSwitches -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5 \
               -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5/x86_64-linux-gnu-thread-multi 
  #   PerlInterpStart 2                                                                                                
   #   PerlInterpMax 2                                                                                              
  <Directory "/usr/local/wormbase/website-classic/html">
       DirectoryIndex index.html
       #       Order deny,allow         
       #       Deny from all                                                                                            
  </Directory>
  # Redirect requests for the old movable type atom feeds to feedburner
  RewriteEngine ON
  #   RewriteRule ^/rss/wormbase-live-atom\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
  RewriteRule ^/rss/wormbase-live-atom\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
  RewriteRule ^/news\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
  RewriteRule ^/rss/wormbase-live\.xml$ http://feeds.feedburner.com/wormbase [R=301,L]
</VirtualHost>

Enable virtual hosts in httpd.conf by uncommenting the following line:

# Include conf/extra/httpd-hosts.conf

Set up httpd to run under inet.d

cd /etc/init.d
sudo ln -s /usr/local/apache2/bin/apachectl httpd

Edit apachectl to make it "LSB" compliant (ie Debian compliant). Add the following after the she-bang:

### BEGIN INIT INFO                                                             
# Provides:          httpd                                                      
# Required-Start:    $all                                                       
# Required-Stop:     $remote_fs $syslog                                         
# Default-Start:     3 5                                                    
# Default-Stop:      0 1 2 4 6                                                      
# Short-Description: Start daemon at boot time                                  
# Description:       Enable service provided by daemon.                         
### END INIT INFO
cd /etc/rc3.d
sudo ln -s ../init.d/apache2 S99apache2
cd ../rc5.d
sudo ln -s ../init.d/apache2 S99apache2

Or via update-rc.d

sudo update-rc.d apache2 start 99 3 5 . stop 0 1 2 4 6 .


PHP

cd ~/src
tar xzf php-5.2.4.tar.gz 
./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql
make
sudo make install

AUTHOR

Todd Harris (todd@wormbase.org)