Difference between revisions of "Building Frozen Releases"

From WormBaseWiki
Jump to navigationJump to search
Line 174: Line 174:
  
 
= TODO =
 
= TODO =
 +
 +
* Install NCBI and WU-BLAST
 +
* Install BLAT and monitoring scripts; modify paths as req'd
  
 
* WS190 index.html is a new version and needs to be pulled from the VMX
 
* WS190 index.html is a new version and needs to be pulled from the VMX
Line 179: Line 182:
 
* WS170 index.html is a new version and needs to be pulled from the VMX
 
* WS170 index.html is a new version and needs to be pulled from the VMX
 
* WS160 index.html is a new version and needs to be pulled from the VMX
 
* WS160 index.html is a new version and needs to be pulled from the VMX
* Install NCBI and WU-BLAST
 
* Install BLAT and monitoring scripts; modify paths as req'd
 
 
* WS130: modules, Ace, GBrowse, enable system config
 
* WS120: modules, Ace, GBrowse, enable system config
 
 
* WS100: modules, Ace, GBrowse, enable system config, local config, missing files!
 
* WS100: modules, Ace, GBrowse, enable system config, local config, missing files!

Revision as of 17:01, 30 July 2009

Hardware

WormBase frozen releases are hosted on the virtualized server:

wb-dev2.oicr.on.ca

  • 500 GB disk space
  • 4 GB RAM
  • Debian "Lenny"

Server Configuration

All WormBase-specific files are maintained in a shared directory.

ls /usr/local/wormbase
    acedb/
    logs/    // Cross-site logs
    WS100    // WS100 website
    WS110    // WS110 website
    WS120    // WS120 website
    ...

Each frozen release is a separate virtual host running under a single Apache instance. To prevent collision of libraries, version specific Perl modules are maintained in:

  /usr/local/wormbase/VERSION/extlib

@INC is prepended with this path for each virtual host during apache launch.

Users, groups, directories

Set up the basic users, groups, and directories as previously described.

Installing system libraries

Prepare the system by installing a slew of libraries.

AceDB

Build AceDB as previously described.

Note that EACH virtual host will need its own xinetd configuration file and unique port on which acedb will bind.

This port MUST correspond in the localdefs.pm file (discussed below).

WS100 ... 2005
WS110 ... 2006
WS120 ... 2007
WS130 ... 2008
WS140 ... 2009
WS150 ... 2010
WS160 ... 2011
WS170 ... 2012
WS180 ... 2013
WS190 ... 2014
WS200 ... 2015

Furthermore, the names of each service (acedb_WSXXX) and its port needs to correspond to the name of the service in /etc/services.

MySQL

Install and configure MySQL as previously described.

Each frozen release will have one or more database of the form species_VERSION or g_species_VERSION. Give access to each of these for the www-data user:

grant select on elegans_WS110.* to www-data@localhost;

Apache2 and mod_perl2

Install and configurate Apache2 and mod_perl2 as described.

Each frozen release will be associated with its own Perl interpreter to prevent namespace collisions. Here's how.


Set up name based virtual hosts for each frozen release as follows:

 $ cd /etc/sites-available/ ; sudo emace default

Add a virtual host entry for the server.

 <VirtualHost *:80>
   Include /usr/local/wormbase/VERSION/conf/httpd.conf
   ServerName ws[VERSION].wormbase.org
   UseCanonicalName on
   PerlOptions +Parent
   PerlSwitches -Mlib=/usr/local/wormbase/VERSION/extlib \
                -Mlib=/usr/local/wormbase/WS200/lib \
                -Mlib=/usr/local/wormbase/WS200/cgi-perl/lib \
                -Mlib=/usr/local/wormbase/VERSION/extlib/lib \
                -Mlib=/usr/local/wormbase/VERSION/extlib/lib/perl5
   PerlInterpStart 2
   PerlInterpMax 2
 </VirtualHost>

Be certain to remove or comment out the default configuration.

Fetch frozen releases and unpack

$ ssh brie4 
$ cd ~ftp/pub/wormbase
$ screen
$ scp -r data_freezes wb-dev.oicr.on.ca:.
  • Unpack acedb_VERSION.ace.tgz or elegans_VERSION.ace.tgz into /usr/local/wormbase/acedb/
  • Unpack GFF databases into /var/lib/mysql/
  • Prepend the version onto each mysql database: species_VERSION or g_species_VERSION

WHERE DO THE BLAST/BLAT DATABASES GO?

Perl Libraries

Each frozen release has its OWN set of Perl libraries located at /usr/local/wormbase/VERSION/extlib. I realize this is a little redundant but this is the best and easiest way to keep critical components like BioPerl and GBrowse distinct.

To do this, follow the basic instructions on Managing Perl Libraries.

There are a few modifications; see "install_perl_libraries.sh" for details. This modified version doesn't try to install mod_perl2 (not necessary for each frozen release, just globally) and installs into a path specified on the command line:

 cd /usr/local/wormbase
 install_perl_libraries.sh /usr/local/wormbase/VERSION/extlib

Be sure to "rm -rf ~/.cpan" before beginning.

BioPerl and GBrowse

For releases < WS200, install BioPerl 1.6.0 and GBrowse 1.69. This isn't entirely historically accurate, but it avoids security holes in GBrowse < 1.69.

$ cd /usr/local/wormbase/build/bioperl-1.6.0
$ perl ./Build.PL --install_base /usr/local/wormbase/VERSION/extlib
$ ./Build install
$ cd /usr/local/wormbase/build/Generic-Genome-Browser-1.69
$ perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/VERSION/extlib
$ make
$ make install

Fix configuration files

xinetd.d/acedb_[WSVERSION]

Create an xinetd.d/acedb_WSVERSION file for the server. The server port and path should correspond to that listed above.

/etc/services

Add an entry in /etc/services for the corresponding service and port.

perl.startup

  • Insert the VERSION into all appropriate "use lib" statements.
  • Add "use Apache2::compat" for mod_perl1 backwards compatability.

httpd.conf

  • Change all occurrences of /usr/local/wormbase to /usr/local/wormbase/VERSION
  • Change the ServerName as appropriate
  • Prepend the version to both the error and access logs
  • Remove any unnecessary configuration (DAV, mailarch, etc)
  • In the "Location /db/" stanza, replace the PerlHandler line with:
  PerlResponseHandler   ModPerl::Registry
  • Edit the PerlRequire so that perl.startup is brought in AFTER configuration:
 PerlPostConfigRequire /usr/local/wormbase/VERSION/conf/perl.startu

elegans.pm

  • Change the path to our root: /usr/local/wormbase/VERSION
  • Change the symbolic name of the acedb database to wormbase_VERSION
  • Append the version to all GFF databases

gbrowse.conf

  • Append the version to all GFF database conf files (conf/gbrowse.conf/*.conf).

Apache/AddWormBaseBanner.pm

The mod_perl handler resposible for adding headers and footers needs to be fixed so that it can correctly redirect / request to /index.html. See here for details.

TODO

  • Install NCBI and WU-BLAST
  • Install BLAT and monitoring scripts; modify paths as req'd
  • WS190 index.html is a new version and needs to be pulled from the VMX
  • WS180 index.html is a new version and needs to be pulled from the VMX
  • WS170 index.html is a new version and needs to be pulled from the VMX
  • WS160 index.html is a new version and needs to be pulled from the VMX
  • WS100: modules, Ace, GBrowse, enable system config, local config, missing files!