Difference between revisions of "GBrowse Administration"
(33 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
= Overview = | = Overview = | ||
− | GBrowse is one of the central -- and most heavily used -- components of the WormBase site. To meet this demand, we use a number of dedicated servers using GBrowse's built in renderfarm capability. This document describes how to | + | GBrowse is one of the central -- and most heavily used -- components of the WormBase site. To meet this demand, we use a number of dedicated servers using GBrowse's built in renderfarm capability. This document describes how to install GBrowse and configure these servers. |
= Hardware Topology = | = Hardware Topology = | ||
Line 17: | Line 17: | ||
Note that although this topology presents a single master with multiple slave servers, in reality each node is redundant with each other. This makes failover easier in case the master server dies. See the Administration & Troubleshooting section for details. | Note that although this topology presents a single master with multiple slave servers, in reality each node is redundant with each other. This makes failover easier in case the master server dies. See the Administration & Troubleshooting section for details. | ||
+ | |||
+ | |||
+ | === Quick Overview === | ||
+ | |||
+ | 1. GBrowse is included with our application and configured as follows. | ||
+ | |||
+ | 2. $app_root/$app/wormbase.env sets the GBROWSE_CONF and GBROWSE_HTDOCS environment variables. | ||
+ | |||
+ | 3. Configuration is located at $app_root/$app/conf/gbrowse | ||
+ | |||
+ | 4. GBrowse is installed at: $app_root/$app/root/gbrowse | ||
+ | |||
+ | 5. wormbase.psgi file mounts | ||
+ | |||
+ | * /tools/genome => gbrowse CGIs | ||
+ | * /gbrowse-static => $app_root/$app/root/gbrowse | ||
+ | |||
+ | 6. Even though every node can host gbrowse, nginx proxy passes requests to dedicated nodes. | ||
= Basic Server Configuration = | = Basic Server Configuration = | ||
Line 29: | Line 47: | ||
== Create directories == | == Create directories == | ||
− | ''Create a directory for GBrowse | + | ''Create a directory for GBrowse's temporary files.'' |
− | > | + | > mkdir -p /usr/local/wormbase/tmp |
− | > sudo | + | > sudo chown -R www-daemon /usr/local/wormbase/tmp |
− | |||
− | |||
− | |||
− | |||
− | |||
== Install prerequisites (Debian) == | == Install prerequisites (Debian) == | ||
Line 79: | Line 92: | ||
bash> sudo make install | bash> sudo make install | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== Install Requisite Perl Modules == | == Install Requisite Perl Modules == | ||
Line 297: | Line 131: | ||
> tar xzf Bio* | > tar xzf Bio* | ||
> cd Bio* | > cd Bio* | ||
− | > export SAMTOOLS=/usr/local/wormbase | + | > export SAMTOOLS=/usr/local/wormbase/src/samtools-0.1.16 |
> perl Build.PL --install_base=/usr/local/wormbase/extlib | > perl Build.PL --install_base=/usr/local/wormbase/extlib | ||
> ./Build | > ./Build | ||
Line 307: | Line 141: | ||
To compile this module, you must first download, unpack and compile Jim Kent's source tree to create the main library file jkweb.a: | To compile this module, you must first download, unpack and compile Jim Kent's source tree to create the main library file jkweb.a: | ||
− | > wget http://hgdownload.cse.ucsc.edu/admin/jksrc.zip | + | > wget http://hgdownload.cse.ucsc.edu/admin/jksrc.zip |
> unzip jksrc.zip | > unzip jksrc.zip | ||
− | > cd | + | > cd kent/src/lib |
> export MACHTYPE=x86_64 | > export MACHTYPE=x86_64 | ||
> mkdir $MACHTYPE | > mkdir $MACHTYPE | ||
Line 318: | Line 152: | ||
> wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/Bio-BigFile-1.06.tar.gz | > wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/Bio-BigFile-1.06.tar.gz | ||
> tar xzf Bio-* ; cd Bio-Big* | > tar xzf Bio-* ; cd Bio-Big* | ||
− | > export KENT_SRC=/usr/local/wormbase | + | > export KENT_SRC=/usr/local/wormbase/src/kent/src/ |
> perl Build.PL --install_base=/usr/local/wormbase/extlib | > perl Build.PL --install_base=/usr/local/wormbase/extlib | ||
> ./Build | > ./Build | ||
Line 324: | Line 158: | ||
> (sudo) ./Build install | > (sudo) ./Build install | ||
− | == Install | + | == Install GBrowse == |
+ | |||
+ | GBrowse CGI components are distributed as part and parcel of the WormBase site. Thus each version of the site is directly associated with a distinct version of GBrowse, too. This means that installing new versions will necessarily overwrite old and need to be recommitted to the repository. | ||
+ | |||
+ | Substitute the symbolic name for the site you are installing into. | ||
− | $ | + | $ cd /usr/ocal/wormbase/shared/src |
− | + | $ wget http://sourceforge.net/projects/gmod/files/Generic%20Genome%20Browser/GBrowse-2.39/GBrowse-2.39.tar.gz/download | |
− | $ perl ./Build.PL --install_base | + | // GBROWSE_ROOT = /usr/local/wormbase/website/YOURSITE |
+ | $ perl ./Build.PL --install_base ${GBROWSE_ROOT} | ||
$ perl ./Build --reconfig | $ perl ./Build --reconfig | ||
− | Directory for GBrowse's config and support files? / | + | Directory for GBrowse's config and support files? ${GBROWSE_ROOT}/conf/gbrowse |
− | Directory for GBrowse's static images & HTML files? / | + | Directory for GBrowse's static images & HTML files? ${GBROWSE_ROOT}/root/gbrowse |
− | Directory for GBrowse's temporary data /usr/local/wormbase/tmp | + | Directory for GBrowse's temporary data /usr/local/wormbase/tmp/gbrowse |
− | Directory for GBrowse's sessions, uploaded tracks and other persistent data: /usr/local/wormbase/tmp | + | Directory for GBrowse's sessions, uploaded tracks and other persistent data: /usr/local/wormbase/tmp/gbrowse/persistent |
− | Directory for GBrowse's example databases /usr/local/wormbase/ | + | Directory for GBrowse's example databases /usr/local/wormbase/tmp/gbrowse/databases |
− | Directory for GBrowse's CGI script executables? / | + | Directory for GBrowse's CGI script executables? ${GBROWSE_ROOT}/root/cgi/gbrowse |
Internet port to run demo web site on (for demo)? [8000] | Internet port to run demo web site on (for demo)? [8000] | ||
Apache loadable module directory (for demo)? [/usr/local/apache2/modules] | Apache loadable module directory (for demo)? [/usr/local/apache2/modules] | ||
− | User account under which Apache daemon runs? [www-data] | + | User account under which Apache daemon runs? [www-data] www-data |
Automatically update Apache config files to run GBrowse? [n] | Automatically update Apache config files to run GBrowse? [n] | ||
Automatically update system config files to run gbrowse-slave? [n] y | Automatically update system config files to run gbrowse-slave? [n] y | ||
Line 351: | Line 190: | ||
./Build install | ./Build install | ||
− | You may need to fix permissions | + | You may need to create directories and fix permissions. |
chmod 777 /usr/local/wormbase/tmp | chmod 777 /usr/local/wormbase/tmp | ||
Line 393: | Line 232: | ||
> mysql -u root -p -e 'grant all privileges on `userdata\_%`.* to 'nobody'@localhost identified by '3l3g@nz'; | > mysql -u root -p -e 'grant all privileges on `userdata\_%`.* to 'nobody'@localhost identified by '3l3g@nz'; | ||
− | == | + | == Install Apache2 == |
+ | |||
+ | ''Technically, none of the slave renderers require Apache. We install it anyways just in case the primary master server dies.'' | ||
+ | |||
+ | |||
+ | === Apache Configuration === | ||
+ | |||
+ | Edit the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) with the appropriate port, virtual host, and fcgi settings: | ||
+ | |||
+ | * Set httpd to listen on your desired port. In the examples below, we assume port 8080. | ||
+ | |||
+ | # Edit "Listen 80" to read | ||
+ | Listen 8080 | ||
+ | |||
+ | Enable virtual hosts by uncommenting out the following line: | ||
+ | |||
+ | #Include conf/extra/httpd_hosts.conf | ||
+ | |||
+ | Set up a virtual host on your port by editing /usr/local/apache2/conf/extras/httpd-vhosts.conf: | ||
+ | |||
+ | # GBrowse_syn | ||
+ | # Running under it's own virtual host since it has some weird configuration quirks. | ||
+ | <VirtualHost *:8080> | ||
+ | PerlSwitches -Mlib=/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi \ | ||
+ | -Mlib=/usr/local/wormbase/extlib/lib/perl5 | ||
+ | |||
+ | ScriptAlias "/db/gb2" "/usr/local/wormbase/services/gbrowse2/current/cgi/gb2" | ||
+ | Alias "/gb2-support/" "/usr/local/wormbase/services/gbrowse2/support-files/" | ||
+ | Alias "/gbrowse2/i" "/usr/local/wormbase/tmp/gbrowse/images" | ||
+ | Alias "/gbrowse2" "/usr/local/wormbase/services/gbrowse2/current/html" | ||
+ | |||
+ | LogFormat "%h %l %u %t \"%r\" %s %b" common | ||
+ | LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format | ||
+ | #LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b %P" combined_format | ||
+ | LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous | ||
+ | |||
+ | ErrorLog /usr/local/wormbase/logs/gbrowse_syn/error.log | ||
+ | CustomLog /usr/local/wormbase/logs/gbrowse_syn access.log combined_format | ||
+ | |||
+ | SetEnv DOCUMENT_ROOT "/usr/local/wormbase/services/gbrowse2/current/html" | ||
+ | <Directory "/usr/local/wormbase/services/gbrowse2/current/cgi/gb2"> | ||
+ | SetEnv PERL5LIB "/usr/local/wormbase/extlib/lib/perl5:/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/services/gb\ | ||
+ | rowse2/support-files/lib:/usr/local/wormbase/services/gbrowse2/support-files/lib/Bio:/usr/local/wormbase/services/gbrowse2/support-files/lib/Bio/Graphics:/usr/\ | ||
+ | local/wormbase/services/gbrowse2/support-files/lib/Bio/Graphics/Glyph" | ||
+ | SetEnv GBROWSE_CONF "/usr/local/wormbase/services/gbrowse2/support-files/conf" | ||
+ | SetEnv DOCUMENT_ROOT "/usr/local/wormbase/services/gbrowse2/current/html" | ||
+ | </Directory> | ||
+ | </VirtualHost> | ||
+ | |||
+ | |||
+ | </pre> | ||
+ | |||
+ | === Set up httpd to run under inet.d === | ||
+ | |||
+ | ''Only required for the master server.'' | ||
+ | |||
+ | Remove /etc/init.d/apache2. It just confuses things. | ||
+ | |||
+ | ''Set which runlevels httpd will run under by chkconfig or command line:'' | ||
+ | |||
+ | chkconfig --add httpd | ||
+ | chkconfig --level 2345 httpd on | ||
+ | chkconfig --list | ||
+ | |||
+ | sudo cp /usr/local/apache2/bin/apachectl /etc/init.d/httpd | ||
+ | cd /etc/rc3.d | ||
+ | sudo ln -s ../init.d/httpd S90httpd | ||
+ | cd /etc/rc5.d | ||
+ | sudo ln -s ../init.d/httpd S90httpd | ||
+ | |||
+ | == Install mod_fcgid == | ||
+ | |||
+ | > wget http://apache.sunsite.ualberta.ca/httpd/mod_fcgid/mod_fcgid-2.3.6.tar.gz | ||
+ | > tar zxvf mod_fcgid-2.3.6.tar.gz | ||
+ | > cd mod_fcgid-2.3.6 | ||
+ | > perl -pi.bak -e 's/INITENV_VAL_LEN 128/INITENV_VAL_LEN 512/' modules/fcgid/fcgid_conf.h | ||
+ | > APXS=/usr/bin/apxs2 ./configure.apxs | ||
+ | > make | ||
+ | > sudo make install | ||
+ | |||
+ | Configure apache as described above. | ||
+ | |||
+ | For additional notes, see: | ||
+ | http://gmod.org/wiki/Recompiling_mod_fcgid_to_avoid_truncated_Perl_library_paths | ||
+ | |||
+ | == MySQL == | ||
+ | |||
+ | ''Each node in the renderfarm has its own MySQL databases.'' | ||
+ | |||
+ | === Installation (>= 5.1) === | ||
+ | |||
+ | ''We need a newer version of mysql than is available from the package manager. We'lll install the binary for x86_64.'' | ||
+ | |||
+ | $ sudo groupadd mysql | ||
+ | $ sudo useradd -g mysql mysql | ||
+ | $ sudo usermod -a -G mysql tharris | ||
+ | $ cd /usr/local | ||
+ | $ sudo gunzip < /path/to/mysql-VERSION-OS.tar.gz | sudo tar xvf - | ||
+ | $ sudo ln -s full-path-to-mysql-VERSION-OS mysql | ||
+ | $ cd mysql | ||
+ | $ sudo chown -R mysql . | ||
+ | $ sudo chgrp -R mysql . | ||
+ | $ sudo scripts/mysql_install_db --user=mysql // may be necessary to remove /etc/mysql/my.cnf first | ||
+ | $ sudo chown -R root . | ||
+ | $ sudo chown -R mysql data | ||
+ | $ sudo chmod 2775 data // I like to be able to write to my datadir | ||
+ | |||
+ | // Copy the configuration file | ||
+ | $ sudo cp support_files/my-medium.cnf /etc/my.cnf | ||
+ | |||
+ | // It may be necessary to add the following to my.cnf if back end nodes | ||
+ | // don't have fully qualified domain names | ||
+ | skip-name-resolve | ||
+ | |||
+ | // Set up mysql to start up automatically | ||
+ | $ sudo cp support_files/mysql.server /etc/init.d/. | ||
+ | $ cd /etc/rc3.d | ||
+ | $ sudo ln -s ../init.d/mysql.server S90mysql | ||
+ | $ cd ../rc5.d | ||
+ | $ sudo ln -s ../init.d/mysql.server S90mysql | ||
− | + | // Start it up and set a root password. | |
+ | $ sudo /etc/init.d/mysql.server start | ||
+ | $ mysqladmin -u root password 'PASSWORD' | ||
− | + | == Fetch configuration and support files == | |
− | + | Configuration files are maintained within the "website" git repository. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | $ cd /usr/local/wormbase/website | |
+ | $ git clone git@github.com/WormBase/website.git | ||
+ | $ mv website production // for example | ||
− | + | Continue with the apache configuration as appropriate. | |
== Configure NFS for load balancing == | == Configure NFS for load balancing == | ||
Line 534: | Line 489: | ||
GBrowse generates a slew of temporary files. Under production, these temporary files can quickly exhaust the ulimit of a machine. Set up the following cronjob to purge them periodically. | GBrowse generates a slew of temporary files. Under production, these temporary files can quickly exhaust the ulimit of a machine. Set up the following cronjob to purge them periodically. | ||
+ | 3 4,16 * * * find /usr/local/wormbase/tmp/gbrowse/cache -type f -mtime +2 -delete | ||
+ | 26 6,18 * * * find /usr/local/wormbase/tmp/gbrowse/images -type f -mtime +1 -delete | ||
+ | 31 10,22 * * * find /usr/local/wormbase/tmp/gbrowse/sessions -type f -mtime +60 -delete | ||
+ | 40 2,14 * * * find /var/www/synteny/images -type f -mtime +1 -delete | ||
+ | 5 3 * * 6 find /usr/local/wormbase/tmp/gbrowse/cache -type d -empty -delete | ||
+ | 35 3 * * 6 find /usr/local/wormbase/tmp/gbrowse/images -type d -empty -delete | ||
+ | 5 5 * * 6 find find /usr/local/wormbase/tmp/gbrowse/sessions -type d -empty -delete | ||
+ | 35 5 * * 6 find /var/www/synteny/images -type d -empty -delete | ||
+ | |||
+ | <!-- | ||
1 1 * * * /usr/local/wormbase/gbrowse-support-files/purge_temporary_files.cron.sh | 1 1 * * * /usr/local/wormbase/gbrowse-support-files/purge_temporary_files.cron.sh | ||
Line 540: | Line 505: | ||
cd /usr/local/wormbase/gbrowse-current/tmp | cd /usr/local/wormbase/gbrowse-current/tmp | ||
find . -type f -atime +20 -print -exec rm {} \; | find . -type f -atime +20 -print -exec rm {} \; | ||
+ | --> | ||
=== Log Rotation === | === Log Rotation === | ||
Line 601: | Line 567: | ||
[[Category:User Guide]] | [[Category:User Guide]] | ||
[[Category:Developer documentation]] | [[Category:Developer documentation]] | ||
+ | [[Category: Architecture (Web Dev)]] |
Latest revision as of 15:25, 4 August 2016
Contents
- 1 Overview
- 2 Hardware Topology
- 3 Basic Server Configuration
- 3.1 Create groups
- 3.2 Create directories
- 3.3 Install prerequisites (Debian)
- 3.4 Install Perl 5.10.1 or greater
- 3.5 Install Requisite Perl Modules
- 3.6 Install SamTools and Bio::SamTools
- 3.7 Install Bio::BigFile.pm
- 3.8 Install GBrowse
- 3.9 Install Apache2
- 3.10 Install mod_fcgid
- 3.11 MySQL
- 3.12 Fetch configuration and support files
- 3.13 Configure NFS for load balancing
- 3.14 Configure the master server
- 3.15 GBrowse slave servers
- 3.16 Databases
- 3.17 Database Privileges
- 4 Administration
- 5 References
- 6 Author
Overview
GBrowse is one of the central -- and most heavily used -- components of the WormBase site. To meet this demand, we use a number of dedicated servers using GBrowse's built in renderfarm capability. This document describes how to install GBrowse and configure these servers.
Hardware Topology
The GBrowse topology looks like this:
Master ------------- ----------------------------- | gb1 | \ \ | | ------------- ------------- | | | gb2 | | gb3 | Slaves |* * * * * *| |* * * * * *| |* * * * * *| ------------- ------------- ------------- * * * * * = relational databases running on each node.
Note that although this topology presents a single master with multiple slave servers, in reality each node is redundant with each other. This makes failover easier in case the master server dies. See the Administration & Troubleshooting section for details.
Quick Overview
1. GBrowse is included with our application and configured as follows.
2. $app_root/$app/wormbase.env sets the GBROWSE_CONF and GBROWSE_HTDOCS environment variables.
3. Configuration is located at $app_root/$app/conf/gbrowse
4. GBrowse is installed at: $app_root/$app/root/gbrowse
5. wormbase.psgi file mounts
- /tools/genome => gbrowse CGIs
- /gbrowse-static => $app_root/$app/root/gbrowse
6. Even though every node can host gbrowse, nginx proxy passes requests to dedicated nodes.
Basic Server Configuration
Create groups
sudo addgroup wormbase sudo addgroup gbrowse # Add yourself to the groups sudo usermod -a -G gbrowse, wormbase tharris
Create directories
Create a directory for GBrowse's temporary files.
> mkdir -p /usr/local/wormbase/tmp > sudo chown -R www-daemon /usr/local/wormbase/tmp
Install prerequisites (Debian)
// Update apt $ sudo apt-get update
// Make tools $ sudo apt-get install gcc make
// expat $ sudo apt-get install expat libexpat1 libexpat1-dev
// GD support $ sudo apt-get install libgd2-noxpm libgd2-noxpm-dev
// Graphviz, now apparently required by BioPerl $ sudo apt-get install graphviz
Install Perl 5.10.1 or greater
You'll need Perl version 5.10.1 or greater.
Debian:
$ sudo apt-get install perl
NOTE: This version of Perl on Debian Lenny doesn't ship with perl.h. You won't be able to build mod_perl without it! You should probably build from source.
Building from source:
bash> cd ~/src bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz bash> cd ~/build bash> tar xzf ../src/perl-5.10.1.tar.gz bash> cd perl-5.10.1 bash> ./Configure -des // Or, to install in a local path: bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1 bash> make bash> make test bash> sudo make install
Install Requisite Perl Modules
All libraries specific for GBrowse are maintained in a local shared directory. This makes it easier and faster to upgrade GBrowse, but more difficult to test multiple versions simultaneously. To test a new version of GBrowse, place extlib inside of the gbrowse-current directory and modify the apache configuration and commands below as appropriate.
Install local::lib to make things easier:
$ sudo perl -MCPAN -e 'CPAN::install(local::lib)' $ mkdir /usr/local/wormbase/extlib $ cd /usr/local/wormbase/extlib
To install modules to this path:
$ perl -Mlocal::lib=. $ eval $(perl -Mlocal::lib=./) $ sudo rm -rf ~/.cpan $ perl -MCPAN -e shell
cpan> install YAML DBI DBD::mysql CGI::Session JSON Storable FreezeThaw FCGI SVG GD::SVG Text::Shellwords \ Algorithm::Munkres Array::Compare Clone Convert::Binary::C Error Graph GraphViz HTML::Entities HTML::HeadParser HTTP::Request::Common LWP::UserAgent \ List::MoreUtils Math::Random PostScript::TextBlock SOAP::Lite SVG::Graph Set::Scalar Sort::Naturally Spreadsheet::ParseExcel URI::Escape \ XML::DOM::XPath XML::Parser XML::Parser::PerlSAX XML::SAX XML::SAX::Writer XML::Simple XML::Twig XML::Writer \ Capture::Tiny Template Term::ReadKey
cpan> install Bio::Perl // may be necessary to install from source cpan> install Bio::Graphics // may be necessary to install from source
Install SamTools and Bio::SamTools
> cd /usr/local/wormbase/services/gbrowse2/src > wget http://sourceforge.net/projects/samtools/files/samtools/0.1.16/samtools-0.1.16.tar.bz2/download > bunzip2 samtools* > cd samtools* > make CXXFLAGS=-fPIC CFLAGS=-fPIC CPPFLAGS=-fPIC > sudo cp misc/*.pl /usr/local/bin/
> wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/Bio-SamTools-1.28.tar.gz > tar xzf Bio* > cd Bio* > export SAMTOOLS=/usr/local/wormbase/src/samtools-0.1.16 > perl Build.PL --install_base=/usr/local/wormbase/extlib > ./Build > ./Build test > (sudo) ./Build install
Install Bio::BigFile.pm
To compile this module, you must first download, unpack and compile Jim Kent's source tree to create the main library file jkweb.a:
> wget http://hgdownload.cse.ucsc.edu/admin/jksrc.zip > unzip jksrc.zip > cd kent/src/lib > export MACHTYPE=x86_64 > mkdir $MACHTYPE > make CXXFLAGS=-fPIC CFLAGS=-fPIC CPPFLAGS=-fPIC
Download and build Bio::BigFile, eg v1.06:
> wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/Bio-BigFile-1.06.tar.gz > tar xzf Bio-* ; cd Bio-Big* > export KENT_SRC=/usr/local/wormbase/src/kent/src/ > perl Build.PL --install_base=/usr/local/wormbase/extlib > ./Build > ./Build test > (sudo) ./Build install
Install GBrowse
GBrowse CGI components are distributed as part and parcel of the WormBase site. Thus each version of the site is directly associated with a distinct version of GBrowse, too. This means that installing new versions will necessarily overwrite old and need to be recommitted to the repository.
Substitute the symbolic name for the site you are installing into.
$ cd /usr/ocal/wormbase/shared/src $ wget http://sourceforge.net/projects/gmod/files/Generic%20Genome%20Browser/GBrowse-2.39/GBrowse-2.39.tar.gz/download // GBROWSE_ROOT = /usr/local/wormbase/website/YOURSITE $ perl ./Build.PL --install_base ${GBROWSE_ROOT} $ perl ./Build --reconfig Directory for GBrowse's config and support files? ${GBROWSE_ROOT}/conf/gbrowse Directory for GBrowse's static images & HTML files? ${GBROWSE_ROOT}/root/gbrowse Directory for GBrowse's temporary data /usr/local/wormbase/tmp/gbrowse Directory for GBrowse's sessions, uploaded tracks and other persistent data: /usr/local/wormbase/tmp/gbrowse/persistent Directory for GBrowse's example databases /usr/local/wormbase/tmp/gbrowse/databases Directory for GBrowse's CGI script executables? ${GBROWSE_ROOT}/root/cgi/gbrowse Internet port to run demo web site on (for demo)? [8000] Apache loadable module directory (for demo)? [/usr/local/apache2/modules] User account under which Apache daemon runs? [www-data] www-data Automatically update Apache config files to run GBrowse? [n] Automatically update system config files to run gbrowse-slave? [n] y
To fetch suggested apache configuration:
./Build apache_conf
Install dependencies
./Build installdeps
To install:
./Build install
You may need to create directories and fix permissions.
chmod 777 /usr/local/wormbase/tmp
Install gbrowse-slave components on all servers
sudo ./Build install_slave
Edit /etc/default/gbrowse_slave:
# Make sure that Perl5Lib includes that path to our external aggregator modules: export PERL5LIB=/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:\ /usr/local/wormbase/extlib/lib/perl5:\ /usr/local/wormbase/services/gbrowse2/current/lib/perl5:\ /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib:\ /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib/Bio:\ /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib/Bio/Graphics:\ /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib/Bio/Graphics/Glyph
DAEMON=/usr/local/bin/gbrowse_slave # Be certain this matches the httpd daemon user on the master! USER=www-data #PRELOAD=/etc/gbrowse2/slave_preload.conf RUNDIR=/usr/local/wormbase/logs LOGDIR=/usr/local/wormbase/logs PORT="8101 8102 8103" VERBOSITY=3 NICE=0
Then set up the slave server to start automatically:
$ cd /etc/rc3.d $ sudo ln -s ../init.d/gbrowse-slave S99gbrowse-slave $ cd /etc/rc5.d $ sudo ln -s ../init.d/gbrowse-slave S99gbrowse-slave
Configure sessions and locks databases
Sessions and locks will be stored in the sessions database.
> mysql -u root -p -e 'create database sessions' > mysql -u root -p -e 'grant select on sessions.* to nobody@localhost' identified by '3l3g@nz' // or whatever > mysql -u root -p -e 'grant all privileges on `userdata\_%`.* to 'nobody'@localhost identified by '3l3g@nz';
Install Apache2
Technically, none of the slave renderers require Apache. We install it anyways just in case the primary master server dies.
Apache Configuration
Edit the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) with the appropriate port, virtual host, and fcgi settings:
- Set httpd to listen on your desired port. In the examples below, we assume port 8080.
# Edit "Listen 80" to read Listen 8080
Enable virtual hosts by uncommenting out the following line:
#Include conf/extra/httpd_hosts.conf
Set up a virtual host on your port by editing /usr/local/apache2/conf/extras/httpd-vhosts.conf:
- GBrowse_syn
- Running under it's own virtual host since it has some weird configuration quirks.
<VirtualHost *:8080>
PerlSwitches -Mlib=/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi \ -Mlib=/usr/local/wormbase/extlib/lib/perl5
ScriptAlias "/db/gb2" "/usr/local/wormbase/services/gbrowse2/current/cgi/gb2" Alias "/gb2-support/" "/usr/local/wormbase/services/gbrowse2/support-files/" Alias "/gbrowse2/i" "/usr/local/wormbase/tmp/gbrowse/images" Alias "/gbrowse2" "/usr/local/wormbase/services/gbrowse2/current/html"
LogFormat "%h %l %u %t \"%r\" %s %b" common LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format #LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b %P" combined_format LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous
ErrorLog /usr/local/wormbase/logs/gbrowse_syn/error.log CustomLog /usr/local/wormbase/logs/gbrowse_syn access.log combined_format
SetEnv DOCUMENT_ROOT "/usr/local/wormbase/services/gbrowse2/current/html" <Directory "/usr/local/wormbase/services/gbrowse2/current/cgi/gb2"> SetEnv PERL5LIB "/usr/local/wormbase/extlib/lib/perl5:/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:/usr/local/wormbase/services/gb\
rowse2/support-files/lib:/usr/local/wormbase/services/gbrowse2/support-files/lib/Bio:/usr/local/wormbase/services/gbrowse2/support-files/lib/Bio/Graphics:/usr/\ local/wormbase/services/gbrowse2/support-files/lib/Bio/Graphics/Glyph"
SetEnv GBROWSE_CONF "/usr/local/wormbase/services/gbrowse2/support-files/conf" SetEnv DOCUMENT_ROOT "/usr/local/wormbase/services/gbrowse2/current/html" </Directory>
</VirtualHost>
Set up httpd to run under inet.d
Only required for the master server.
Remove /etc/init.d/apache2. It just confuses things.
Set which runlevels httpd will run under by chkconfig or command line:
chkconfig --add httpd chkconfig --level 2345 httpd on chkconfig --list
sudo cp /usr/local/apache2/bin/apachectl /etc/init.d/httpd cd /etc/rc3.d sudo ln -s ../init.d/httpd S90httpd cd /etc/rc5.d sudo ln -s ../init.d/httpd S90httpd
Install mod_fcgid
> wget http://apache.sunsite.ualberta.ca/httpd/mod_fcgid/mod_fcgid-2.3.6.tar.gz > tar zxvf mod_fcgid-2.3.6.tar.gz > cd mod_fcgid-2.3.6 > perl -pi.bak -e 's/INITENV_VAL_LEN 128/INITENV_VAL_LEN 512/' modules/fcgid/fcgid_conf.h > APXS=/usr/bin/apxs2 ./configure.apxs > make > sudo make install
Configure apache as described above.
For additional notes, see:
http://gmod.org/wiki/Recompiling_mod_fcgid_to_avoid_truncated_Perl_library_paths
MySQL
Each node in the renderfarm has its own MySQL databases.
Installation (>= 5.1)
We need a newer version of mysql than is available from the package manager. We'lll install the binary for x86_64.
$ sudo groupadd mysql $ sudo useradd -g mysql mysql $ sudo usermod -a -G mysql tharris $ cd /usr/local $ sudo gunzip < /path/to/mysql-VERSION-OS.tar.gz | sudo tar xvf - $ sudo ln -s full-path-to-mysql-VERSION-OS mysql $ cd mysql $ sudo chown -R mysql . $ sudo chgrp -R mysql . $ sudo scripts/mysql_install_db --user=mysql // may be necessary to remove /etc/mysql/my.cnf first $ sudo chown -R root . $ sudo chown -R mysql data $ sudo chmod 2775 data // I like to be able to write to my datadir
// Copy the configuration file $ sudo cp support_files/my-medium.cnf /etc/my.cnf
// It may be necessary to add the following to my.cnf if back end nodes // don't have fully qualified domain names skip-name-resolve
// Set up mysql to start up automatically $ sudo cp support_files/mysql.server /etc/init.d/. $ cd /etc/rc3.d $ sudo ln -s ../init.d/mysql.server S90mysql $ cd ../rc5.d $ sudo ln -s ../init.d/mysql.server S90mysql
// Start it up and set a root password. $ sudo /etc/init.d/mysql.server start $ mysqladmin -u root password 'PASSWORD'
Fetch configuration and support files
Configuration files are maintained within the "website" git repository.
$ cd /usr/local/wormbase/website $ git clone git@github.com/WormBase/website.git $ mv website production // for example
Continue with the apache configuration as appropriate.
Configure NFS for load balancing
In lieue of the built-in GBrowse master/slave renderfarm configuration, we load balance across a pool of servers using a shared filesystem. See the NFS documentation in the main WormBase Production environment for details on how to install NFS.
NFS Server
The master server (wb-gb1.oicr.on.ca) provides an NFS mount of core GBrowse files and temporary session data.
/usr/local/wormbase/services/gbrowse2
This directory will be mounted at the same path for each client.
Contents:
Specify what to share, to whom, and with which privileges
Share mounts are configured through /etc/exports on wb-gb1.oicr.on.ca.
# WormBase NFS for temporary and static content /usr/local/wormbase/services/gbrowse2 206.108.125.174(rw,no_root_squash)
After making changes to /etc/exports, load them by:
sudo exportfs -a
Lock down access to NFS from most hosts
We can use /etc/hosts.deny to quickly lock down NFS:
portmap:ALL lockd:ALL mountd:ALL rquotad:ALL statd:ALL
Allow access to select hosts
We can use /etc/hosts.allow to specifically allow access to NFS:
# WormBase NFS services portmap: ***.***.***.*** lockd: ***.***.***.*** rquotad: ***.***.***.*** mountd: ***.***.***.*** statd: ***.***.***.***
NFS clients
On secondary GBrowse nodes, mount the gbrowse share. The gbrowse2 directory must exist.
sudo mount 206.108.125.173:/usr/local/wormbase/services/gbrowse2 /usr/local/wormbase/services/gbrowse2
Unmounting the NFS share
sudo umount /usr/local/wormbase/services/gbrowse2
Setting up NFS share to mount at boot
sudo emacs /etc/fstab
Add
wb-gb1.oicr.on.ca:/usr/local/wormbase/services/gbrowse2 /usr/local/wormbase/services/gbrowse2 nfs rw 0 0
Configure the master server
The configuration file of the master server should be set up with appropriate render slaves in the TRACK DEFAULTS section of each genome-level configuration file.
# wb-www1: XXX.XXX.XXX.173 (master) # wb-www2: XXX.XXX.XXX.174 (slave) # wb-www3: XXX.XXX.XXX.175 (slave)
remote renderer = http://XXX.XXX.XXX.173:8101 http://XXX.XXX.XXX.173:8102 http://XXX.XXX.XXX.173:8103 http://XXX.XXX.XXX.174:8101 http://XXX.XXX.XXX.174:8102 http://XXX.XXX.XXX.174:8103 http://XXX.XXX.XXX.175:8101 http://XXX.XXX.XXX.175:8102 http://XXX.XXX.XXX.175:8103
Define dsn arguments (here we specify that the MySQL databases are running on the localhost).
db_args = -dsn DBI:mysql:c_elegans;user=nobody;host=localhost
# Here's a second example with all databases running on the master server. db_args = -dsn DBI:mysql:c_elegans;user=nobody;host=XXX.XXX.XXX.173
GBrowse slave servers
To add an additional slave server, simply follow the steps above, adding the name of the server to the TRACK DEFAULTS section of the genome-level configuration file.
No special configuration is required for GBrowse slave servers. Simply ensure that the slave has access to the MySQL databases using the username and password supplied in the primary configuration file.
Make sure the slave servers are running by:
sudo /etc/init.d/gbrowse_slave start
Databases
MySQL databases will be mirrored over directly as part of the production update process. They should be located in /usr/local/mysql/data. Symlinks, using the symbolic name of the database, point to the current version:
c_elegans -> c_elegans_WS211
Database Privileges
Be sure to grant privileges to both master and slave servers to the databases:
$ mysql -u root -p -e 'grant select on c_elegans.* to nobody@localhost' $ mysql -u root -p -e 'grant select on c_elegans.* to nobody@render-slave-ip'
Administration
Updating GBrowse
To update GBrowse:
- Check out the source to /usr/local/wormbase/build
- Follow the directions above to install GBrowse, resetting the current symlink
- Check out the "wormbase-gbrowse2" configuration repository
Maintenance
Purging Temporary Files
GBrowse generates a slew of temporary files. Under production, these temporary files can quickly exhaust the ulimit of a machine. Set up the following cronjob to purge them periodically.
3 4,16 * * * find /usr/local/wormbase/tmp/gbrowse/cache -type f -mtime +2 -delete 26 6,18 * * * find /usr/local/wormbase/tmp/gbrowse/images -type f -mtime +1 -delete 31 10,22 * * * find /usr/local/wormbase/tmp/gbrowse/sessions -type f -mtime +60 -delete 40 2,14 * * * find /var/www/synteny/images -type f -mtime +1 -delete 5 3 * * 6 find /usr/local/wormbase/tmp/gbrowse/cache -type d -empty -delete 35 3 * * 6 find /usr/local/wormbase/tmp/gbrowse/images -type d -empty -delete 5 5 * * 6 find find /usr/local/wormbase/tmp/gbrowse/sessions -type d -empty -delete 35 5 * * 6 find /var/www/synteny/images -type d -empty -delete
Log Rotation
gbrowse-support-files/
contains two configuration files for the unix logrotate
command. For the master server copy logrotate-gbrowse-httpd.conf
and logrotate-gbrowse-slave.conf
to /etc/logrotate.d/gbrowse-httpd and gbrowse-slave
. The slave nodes obviously only require the gbrowse-slave file.
The contents of these files are included here for reference:
# logrotate-gbrowse-httpd.conf # GBrowse httpd logrotate conf file /usr/local/wormbase/logs/*.log { daily missingok rotate 7 compress delaycompress notifempty create 640 root adm sharedscripts postrotate /usr/local/apache2/bin/apachectl graceful endscript } # logrotate-gbrowse-slave.conf # GBrowse slave files /usr/local/wormbase/logs/*_slave { daily missingok rotate 7 compress delaycompress notifempty sharedscripts postrotate /etc/init.d/gbrowse-slave stop /etc/init.d/gbrowse-slave start endscript }
Establishing a new slave node
Here's how to build a new slave node:
1. Set up directories, users, groups 2. Install apache, fastcgi, mysql 3. Copy /usr/local/wormbase from an existing node
References
- The official GBrowse Install HOWTO
- The GBrowse 2.0 renderfarm HowTO at GMOD.org
Author
Todd Harris, PhD (info@toddharris.net)