Difference between revisions of "Building Frozen Releases"
(→TODO) |
|||
(23 intermediate revisions by 3 users not shown) | |||
Line 9: | Line 9: | ||
* Debian "Lenny" | * Debian "Lenny" | ||
− | =Server Configuration = | + | =General Server Configuration = |
All WormBase-specific files are maintained in a shared directory. | All WormBase-specific files are maintained in a shared directory. | ||
Line 21: | Line 21: | ||
... | ... | ||
− | Each frozen release is a separate | + | Each frozen release is a separate name-based VirtualHost. To prevent collision of libraries, version specific Perl modules are maintained in: |
/usr/local/wormbase/VERSION/extlib | /usr/local/wormbase/VERSION/extlib | ||
− | @INC is prepended with this path | + | @INC is prepended with this path using mod_perl2 "PerlSwitches" option described below. |
== Users, groups, directories == | == Users, groups, directories == | ||
Line 40: | Line 40: | ||
Note that EACH virtual host will need its own xinetd configuration file and unique port on which acedb will bind. | Note that EACH virtual host will need its own xinetd configuration file and unique port on which acedb will bind. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
Furthermore, the names of each service (acedb_WSXXX) and its port needs to correspond to the name of the service in /etc/services. | Furthermore, the names of each service (acedb_WSXXX) and its port needs to correspond to the name of the service in /etc/services. | ||
Line 61: | Line 47: | ||
Install and configure MySQL as [[OICR-Configuring_the_development_machine#MySQL|previously described]]. | Install and configure MySQL as [[OICR-Configuring_the_development_machine#MySQL|previously described]]. | ||
− | + | == Apache2 and mod_perl2 == | |
+ | |||
+ | Install and configurate Apache2 and mod_perl2 as [[OICR-Configuring_the_development_machine#Apache2_and_mod_perl|described]]. | ||
+ | |||
+ | = Establishing A New Frozen Release = | ||
+ | |||
+ | == Copy software and libraries == | ||
+ | |||
+ | It is easiest to set up a new frozen release if you do this when the desired release still resides on the development machine. | ||
+ | |||
+ | $ cd /usr/local/wormbase | ||
+ | $ rsync -Cav website-classic user@wb-dev2:/usr/local/wormbase | ||
+ | $ ssh wb-dev2 | ||
+ | $ cd /usr/local/wormbase | ||
+ | $ mv website-classic [VERSION] | ||
+ | |||
+ | == AceDB database == | ||
+ | |||
+ | Copy the appropriate AceDB database to: | ||
+ | |||
+ | $ /usr/local/wormbase/acedb/wormbase_VERSION | ||
+ | |||
+ | == MySQL databases == | ||
+ | |||
+ | Copy mysql databases to the mysql data dir (/var/lib/mysql). | ||
+ | |||
+ | Append the database version to each mysql database if it isn't already versioned: | ||
+ | |||
+ | species_[VERSION] or g_species_[VERSION] | ||
+ | |||
+ | Give access to each of these for the www-data user: | ||
grant select on elegans_WS110.* to www-data@localhost; | grant select on elegans_WS110.* to www-data@localhost; | ||
− | == | + | == Perl Libraries == |
+ | |||
+ | If you rsync the website as decribed above, you shouldn't need to install Perl libraries. If you do, use follow the basic instructions on [[Managing Perl Libraries]]. | ||
+ | |||
+ | Invoke it as: | ||
+ | |||
+ | $ ./install_perl_libraries.sh /usr/local/wormbase/[VERSION]/extlib | ||
+ | |||
+ | Be sure to "rm -rf ~/.cpan" before beginning, as well as clearing your PERL5LIB environment variable. | ||
+ | |||
+ | == BioPerl and GBrowse == | ||
+ | |||
+ | For releases < WS200, install BioPerl 1.6.0 and GBrowse 1.69. This isn't entirely historically accurate, but it avoids security holes in GBrowse < 1.69. | ||
+ | |||
+ | $ cd /usr/local/wormbase/build/bioperl-1.6.0 | ||
+ | $ perl ./Build.PL --install_base /usr/local/wormbase/VERSION/extlib | ||
+ | $ ./Build install | ||
+ | |||
+ | $ cd /usr/local/wormbase/build/Generic-Genome-Browser-1.69 | ||
+ | $ perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/VERSION/extlib | ||
+ | $ make | ||
+ | $ make install | ||
+ | |||
+ | For releases > WS200, install GBrowse 1.70: | ||
− | + | $ cd /usr/local/wormbase/build/Generic-Genome-Browser-1.70 | |
+ | $ perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/VERSION/extlib | ||
+ | $ make | ||
+ | $ make install | ||
− | + | == Fix configuration files == | |
+ | === VirtualHost Configuration === | ||
− | + | Add an appropriate VirtualHost stanza for the frozen release to the apache2 configuration. | |
− | $ | + | $ sudo emacs /etc/apache2/sites-enabled/000-default |
− | |||
− | |||
<VirtualHost *:80> | <VirtualHost *:80> | ||
Line 83: | Line 124: | ||
UseCanonicalName on | UseCanonicalName on | ||
PerlOptions +Parent | PerlOptions +Parent | ||
+ | |||
+ | # Versions prior to WS204 | ||
PerlSwitches -Mlib=/usr/local/wormbase/VERSION/extlib \ | PerlSwitches -Mlib=/usr/local/wormbase/VERSION/extlib \ | ||
-Mlib=/usr/local/wormbase/WS200/lib \ | -Mlib=/usr/local/wormbase/WS200/lib \ | ||
Line 88: | Line 131: | ||
-Mlib=/usr/local/wormbase/VERSION/extlib/lib \ | -Mlib=/usr/local/wormbase/VERSION/extlib/lib \ | ||
-Mlib=/usr/local/wormbase/VERSION/extlib/lib/perl5 | -Mlib=/usr/local/wormbase/VERSION/extlib/lib/perl5 | ||
+ | |||
+ | # Versions >= WS204 which used local::lib for library management | ||
+ | PerlSwitches -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5 \ | ||
+ | -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5/x86_64-linux-gnu-thread-multi | ||
+ | |||
PerlInterpStart 2 | PerlInterpStart 2 | ||
PerlInterpMax 2 | PerlInterpMax 2 | ||
Line 94: | Line 142: | ||
Be certain to remove or comment out the default configuration. | Be certain to remove or comment out the default configuration. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | === xinetd.d/acedb_[WSVERSION] === | |
− | + | Create an xinetd.d/acedb_WSVERSION file for the server. The server port and path should correspond to one of the following: | |
− | + | WS100 ... 2005 | |
− | + | WS110 ... 2006 | |
− | + | WS120 ... 2007 | |
− | + | WS130 ... 2008 | |
− | + | WS140 ... 2009 | |
− | + | WS150 ... 2010 | |
− | + | WS160 ... 2011 | |
− | + | WS170 ... 2012 | |
− | + | WS180 ... 2013 | |
− | + | WS190 ... 2014 | |
+ | WS200 ... 2015 | ||
+ | WS204 ... 2016 | ||
+ | WS205 ... 2017 | ||
− | + | ''Please keep this list up-to-date when establishing a new frozen release.'' | |
− | + | === /etc/services === | |
− | + | Add an entry in /etc/services for the corresponding service and port. | |
=== perl.startup === | === perl.startup === | ||
+ | * /usr/local/wormbase/[VERSION]/conf/perl.startup | ||
* Insert the VERSION into all appropriate "use lib" statements. | * Insert the VERSION into all appropriate "use lib" statements. | ||
* Add "use Apache2::compat" for mod_perl1 backwards compatability. | * Add "use Apache2::compat" for mod_perl1 backwards compatability. | ||
Line 133: | Line 175: | ||
=== httpd.conf === | === httpd.conf === | ||
+ | * /usr/local/wormbase/[VERSION]/conf/httpd.conf | ||
* Change all occurrences of /usr/local/wormbase to /usr/local/wormbase/VERSION | * Change all occurrences of /usr/local/wormbase to /usr/local/wormbase/VERSION | ||
* Change the ServerName as appropriate | * Change the ServerName as appropriate | ||
Line 144: | Line 187: | ||
=== elegans.pm === | === elegans.pm === | ||
+ | * /usr/local/wormbase/[VERSION]/conf/elegans.pm | ||
* Change the path to our root: /usr/local/wormbase/VERSION | * Change the path to our root: /usr/local/wormbase/VERSION | ||
* Change the symbolic name of the acedb database to wormbase_VERSION | * Change the symbolic name of the acedb database to wormbase_VERSION | ||
Line 150: | Line 194: | ||
=== gbrowse.conf === | === gbrowse.conf === | ||
+ | * /usr/local/wormbase/[VERSION]/conf/gbrowse.conf/*.conf | ||
* Append the version to all GFF database conf files (conf/gbrowse.conf/*.conf). | * Append the version to all GFF database conf files (conf/gbrowse.conf/*.conf). | ||
=== Apache/AddWormBaseBanner.pm === | === Apache/AddWormBaseBanner.pm === | ||
− | The mod_perl handler resposible for adding headers and footers needs to be fixed so that it can correctly redirect / request to /index.html. See [http://www.wormbase.org/wiki/index.php/ | + | * /usr/local/wormbase/[VERSION]/lib/Apache/AddWormBaseBanner.pm |
+ | |||
+ | The mod_perl handler resposible for adding headers and footers needs to be fixed so that it can correctly redirect / request to /index.html. See [http://www.wormbase.org/wiki/index.php/WormBase_Development_Environment#Important_Fix_for_mod_perl2.2Fmod_dir_incompatibility_.28the_index.html_problem.29 here] for details. | ||
+ | |||
+ | == Static Files == | ||
+ | |||
+ | Copy appropriate static files to /usr/local/wormbase/data_freezes. See the existing contents for examples. | ||
+ | |||
+ | == Support Databases == | ||
+ | |||
+ | Copy the [VERSION] support databases file to | ||
+ | |||
+ | /usr/local/wormbase/databases/. | ||
+ | |||
+ | = Migrating Frozen Releases = | ||
+ | |||
+ | == Fetch frozen releases and unpack == | ||
+ | |||
+ | $ ssh brie4 | ||
+ | $ cd ~ftp/pub/wormbase | ||
+ | $ screen | ||
+ | $ scp -r data_freezes wb-dev.oicr.on.ca:. | ||
+ | |||
+ | * Unpack acedb_VERSION.ace.tgz or elegans_VERSION.ace.tgz into /usr/local/wormbase/acedb/ | ||
+ | * Unpack GFF databases into /var/lib/mysql/ | ||
+ | * Prepend the version onto each mysql database: species_VERSION or g_species_VERSION | ||
+ | |||
+ | WHERE DO THE BLAST/BLAT DATABASES GO? | ||
+ | |||
+ | Follow the configuration directives above as described for establishing a new frozen release. | ||
+ | |||
+ | = TODO = | ||
+ | |||
+ | * Install NCBI and WU-BLAST | ||
+ | * Install BLAT and monitoring scripts; modify paths as req'd | ||
+ | |||
+ | * WS190 index.html is a new version and needs to be pulled from the VMX | ||
+ | * WS180 index.html is a new version and needs to be pulled from the VMX | ||
+ | * WS170 index.html is a new version and needs to be pulled from the VMX | ||
+ | * WS160 index.html is a new version and needs to be pulled from the VMX | ||
+ | * WS100: modules, Ace, GBrowse, enable system config, local config, missing files! | ||
+ | |||
+ | |||
+ | [[Category: Maintenance (Web Dev)]] | ||
+ | [[Category:Developer documentation]] |
Latest revision as of 17:35, 19 June 2014
Contents
Hardware
WormBase frozen releases are hosted on the virtualized server:
wb-dev2.oicr.on.ca
- 500 GB disk space
- 4 GB RAM
- Debian "Lenny"
General Server Configuration
All WormBase-specific files are maintained in a shared directory.
ls /usr/local/wormbase acedb/ logs/ // Cross-site logs WS100 // WS100 website WS110 // WS110 website WS120 // WS120 website ...
Each frozen release is a separate name-based VirtualHost. To prevent collision of libraries, version specific Perl modules are maintained in:
/usr/local/wormbase/VERSION/extlib
@INC is prepended with this path using mod_perl2 "PerlSwitches" option described below.
Users, groups, directories
Set up the basic users, groups, and directories as previously described.
Installing system libraries
Prepare the system by installing a slew of libraries.
AceDB
Build AceDB as previously described.
Note that EACH virtual host will need its own xinetd configuration file and unique port on which acedb will bind.
Furthermore, the names of each service (acedb_WSXXX) and its port needs to correspond to the name of the service in /etc/services.
MySQL
Install and configure MySQL as previously described.
Apache2 and mod_perl2
Install and configurate Apache2 and mod_perl2 as described.
Establishing A New Frozen Release
Copy software and libraries
It is easiest to set up a new frozen release if you do this when the desired release still resides on the development machine.
$ cd /usr/local/wormbase $ rsync -Cav website-classic user@wb-dev2:/usr/local/wormbase $ ssh wb-dev2 $ cd /usr/local/wormbase $ mv website-classic [VERSION]
AceDB database
Copy the appropriate AceDB database to:
$ /usr/local/wormbase/acedb/wormbase_VERSION
MySQL databases
Copy mysql databases to the mysql data dir (/var/lib/mysql).
Append the database version to each mysql database if it isn't already versioned:
species_[VERSION] or g_species_[VERSION]
Give access to each of these for the www-data user:
grant select on elegans_WS110.* to www-data@localhost;
Perl Libraries
If you rsync the website as decribed above, you shouldn't need to install Perl libraries. If you do, use follow the basic instructions on Managing Perl Libraries.
Invoke it as:
$ ./install_perl_libraries.sh /usr/local/wormbase/[VERSION]/extlib
Be sure to "rm -rf ~/.cpan" before beginning, as well as clearing your PERL5LIB environment variable.
BioPerl and GBrowse
For releases < WS200, install BioPerl 1.6.0 and GBrowse 1.69. This isn't entirely historically accurate, but it avoids security holes in GBrowse < 1.69.
$ cd /usr/local/wormbase/build/bioperl-1.6.0 $ perl ./Build.PL --install_base /usr/local/wormbase/VERSION/extlib $ ./Build install
$ cd /usr/local/wormbase/build/Generic-Genome-Browser-1.69 $ perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/VERSION/extlib $ make $ make install
For releases > WS200, install GBrowse 1.70:
$ cd /usr/local/wormbase/build/Generic-Genome-Browser-1.70 $ perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/VERSION/extlib $ make $ make install
Fix configuration files
VirtualHost Configuration
Add an appropriate VirtualHost stanza for the frozen release to the apache2 configuration.
$ sudo emacs /etc/apache2/sites-enabled/000-default
<VirtualHost *:80> Include /usr/local/wormbase/VERSION/conf/httpd.conf ServerName ws[VERSION].wormbase.org UseCanonicalName on PerlOptions +Parent # Versions prior to WS204 PerlSwitches -Mlib=/usr/local/wormbase/VERSION/extlib \ -Mlib=/usr/local/wormbase/WS200/lib \ -Mlib=/usr/local/wormbase/WS200/cgi-perl/lib \ -Mlib=/usr/local/wormbase/VERSION/extlib/lib \ -Mlib=/usr/local/wormbase/VERSION/extlib/lib/perl5
# Versions >= WS204 which used local::lib for library management PerlSwitches -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5 \ -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5/x86_64-linux-gnu-thread-multi
PerlInterpStart 2 PerlInterpMax 2 </VirtualHost>
Be certain to remove or comment out the default configuration.
xinetd.d/acedb_[WSVERSION]
Create an xinetd.d/acedb_WSVERSION file for the server. The server port and path should correspond to one of the following:
WS100 ... 2005 WS110 ... 2006 WS120 ... 2007 WS130 ... 2008 WS140 ... 2009 WS150 ... 2010 WS160 ... 2011 WS170 ... 2012 WS180 ... 2013 WS190 ... 2014 WS200 ... 2015 WS204 ... 2016 WS205 ... 2017
Please keep this list up-to-date when establishing a new frozen release.
/etc/services
Add an entry in /etc/services for the corresponding service and port.
perl.startup
- /usr/local/wormbase/[VERSION]/conf/perl.startup
- Insert the VERSION into all appropriate "use lib" statements.
- Add "use Apache2::compat" for mod_perl1 backwards compatability.
httpd.conf
- /usr/local/wormbase/[VERSION]/conf/httpd.conf
- Change all occurrences of /usr/local/wormbase to /usr/local/wormbase/VERSION
- Change the ServerName as appropriate
- Prepend the version to both the error and access logs
- Remove any unnecessary configuration (DAV, mailarch, etc)
- In the "Location /db/" stanza, replace the PerlHandler line with:
PerlResponseHandler ModPerl::Registry
- Edit the PerlRequire so that perl.startup is brought in AFTER configuration:
PerlPostConfigRequire /usr/local/wormbase/VERSION/conf/perl.startu
elegans.pm
- /usr/local/wormbase/[VERSION]/conf/elegans.pm
- Change the path to our root: /usr/local/wormbase/VERSION
- Change the symbolic name of the acedb database to wormbase_VERSION
- Append the version to all GFF databases
gbrowse.conf
- /usr/local/wormbase/[VERSION]/conf/gbrowse.conf/*.conf
- Append the version to all GFF database conf files (conf/gbrowse.conf/*.conf).
Apache/AddWormBaseBanner.pm
- /usr/local/wormbase/[VERSION]/lib/Apache/AddWormBaseBanner.pm
The mod_perl handler resposible for adding headers and footers needs to be fixed so that it can correctly redirect / request to /index.html. See here for details.
Static Files
Copy appropriate static files to /usr/local/wormbase/data_freezes. See the existing contents for examples.
Support Databases
Copy the [VERSION] support databases file to
/usr/local/wormbase/databases/.
Migrating Frozen Releases
Fetch frozen releases and unpack
$ ssh brie4 $ cd ~ftp/pub/wormbase $ screen $ scp -r data_freezes wb-dev.oicr.on.ca:.
- Unpack acedb_VERSION.ace.tgz or elegans_VERSION.ace.tgz into /usr/local/wormbase/acedb/
- Unpack GFF databases into /var/lib/mysql/
- Prepend the version onto each mysql database: species_VERSION or g_species_VERSION
WHERE DO THE BLAST/BLAT DATABASES GO?
Follow the configuration directives above as described for establishing a new frozen release.
TODO
- Install NCBI and WU-BLAST
- Install BLAT and monitoring scripts; modify paths as req'd
- WS190 index.html is a new version and needs to be pulled from the VMX
- WS180 index.html is a new version and needs to be pulled from the VMX
- WS170 index.html is a new version and needs to be pulled from the VMX
- WS160 index.html is a new version and needs to be pulled from the VMX
- WS100: modules, Ace, GBrowse, enable system config, local config, missing files!