GBrowse Administration

From WormBaseWiki
Jump to navigationJump to search

Overview

GBrowse is one of the central -- and most heavily used -- components of the WormBase site. To meet this demand, we use a number of dedicated servers using GBrowse's built in renderfarm capability. This document describes how to build and configure these servers.

Hardware Topology

The GBrowse topology looks like this: 

 Master
 ------------- -----------------------------
 |    gb1    |              \               \
 |           |        -------------    -------------   
 |           |        |   gb2     |    |    gb3    |   Slaves
 |* * * * * *|        |* * * * * *|    |* * * * * *|
 -------------        -------------    -------------  
 * * * * * = relational databases running on each node.

Note that although this topology presents a single master with multiple slave servers, in reality each node is redundant with each other. This makes failover easier in case the master server dies. See the Administration & Troubleshooting section for details.

Basic Server Configuration

Create groups

sudo addgroup wormbase
sudo addgroup gbrowse
# Add yourself to the groups
sudo usermod -a -G gbrowse, wormbase tharris

Create directories

Create a directory for GBrowse. For now, we're maintaining GBrowse on dedicated hardware completely distinct from the rest of the web application. This makes updates significantly easier although it increases maintenance (ie boilerplate headers and footers are maintained in two places). We'll also keep gbrowse inside the wormbase/services directory should we wish to install other components on these servers at a later date. 

> sudo mkdir -p /usr/local/wormbase
> sudo chgrp -R wormbase /usr/local/wormbase
> sudo chmod 2775 /usr/local/wormbase 
> cd wormbase
> mkdir 
> mkdir -p logs services/gbrowse2
> chmod 666 logs

Install prerequisites (Debian)

// Update apt
$ sudo apt-get update

// Make tools
$ sudo apt-get install gcc make

// expat
$ sudo apt-get install expat libexpat1 libexpat1-dev

// GD support
$ sudo apt-get install libgd2-noxpm libgd2-noxpm-dev

// Graphviz, now apparently required by BioPerl
$ sudo apt-get install graphviz

Install Perl 5.10.1 or greater

You'll need Perl version 5.10.1 or greater.

Debian: 

$ sudo apt-get install perl

NOTE: This version of Perl on Debian Lenny doesn't ship with perl.h. You won't be able to build mod_perl without it! You should probably build from source.

Building from source: 

bash> cd ~/src
bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz
bash> cd ~/build
bash> tar xzf ../src/perl-5.10.1.tar.gz
bash> cd perl-5.10.1
bash> ./Configure -des
  // Or, to install in a local path:
bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1
bash> make
bash> make test
bash> sudo make install

Install Apache2

Technically, none of the slave renderers require Apache. We install it anyways just in case the primary master server dies.

If you're running Debian, get rid of the installed apache. The default layout is just plain annoying. 

$ sudo apt-get remove apache2

Assuming you have already fetched the source into ~/src: 

# Build httpd 2.2.11
cd ~/src
tar xzf httpd-2.2.11.tar.gz
cd httpd-2.2.11
./configure --enable-mods-shared=all --enable-proxy
make
sudo make install

Apache Configuration

Edit the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) with the appropriate port, virtual host, and fcgi settings:

  • Set httpd to listen on your desired port. In the examples below, we assume port 8080.
# Edit "Listen 80" to read
Listen 8080

Enable virtual hosts by uncommenting out the following line: 

#Include conf/extra/httpd_hosts.conf

Set up a virtual host on your port by editing /usr/local/apache2/conf/extras/httpd-vhosts.conf: 


<VirtualHost *:80>
    ServerName www.wormbase.org
    ErrorLog  "/usr/local/wormbase/logs/gbrowse2-error_log"
    CustomLog "/usr/local/wormbase/logs/gbrowse2-access_log" common

    # Support files: the wormbase css file for gbrowse, banner image, etc
    Alias	"/gb2-support/"	    "/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/"

#    Alias	  "/gbrowse2/i/" "/usr/local/wormbase/gbrowse2/current/tmp/images/"
    Alias        "/gbrowse2/i" "/usr/local/wormbase/tmp/images"
    Alias        "/gbrowse2"   "/usr/local/wormbase/services/gbrowse2/current/html"
#    ScriptAlias  "/db/gb2"	"/usr/local/wormbase/services/gbrowse2/current/cgi/gb2"
#    ScriptAlias  "/gb2"	"/usr/local/wormbase/services/gbrowse2/current/cgi/gb2"

#    <Directory "/usr/local/wormbase/gbrowse2/current/tmp">
    <Directory "/usr/local/wormbase/tmp">
       Allow from all
    </Directory>

    <Directory "/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support">
      Options -Indexes -MultiViews +FollowSymLinks
      Order allow,deny
      Allow from all
    </Directory>

    <Directory "/usr/local/wormbase/services/gbrowse2/current/html">
      Options -Indexes -MultiViews +FollowSymLinks
      Order allow,deny
      Allow from all
    </Directory>

#    <Directory "/usr/local/wormbase/services/gbrowse2/current/cgi/gb2">
#      SetEnv PERL5LIB "/usr/local/wormbase/extlib/lib/perl5: \
#			/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi: \
#			/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib: \
#			/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib/Bio: \
#			/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib/Bio/Graphics: \
#			/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib/Bio/Graphics/Glyph"
#
#	SetEnv GBROWSE_CONF   "/usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/conf"
#	AllowOverride None
#	Options None
#	Order allow,deny
#	Allow from all
#   </Directory>

    <IfModule mod_fcgid.c>
	Alias /db/gb2 "/usr/local/wormbase/services/gbrowse2/current/cgi/gb2"
	<Location /db/gb2>
            SetHandler	 fcgid-script
            Options	 ExecCGI
            Order allow,deny
            Allow from all
        </Location>
	IdleTimeout 300
	FcgidInitialEnv GBROWSE_CONF /usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/conf
	FcgidInitialEnv PERL5LIB "/usr/local/wormbase/extlib/lib/perl5: \
		       /usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi: \
		       /usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib: \
		       /usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib/Bio: \
		       /usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib/Bio/Graphics: \
		       /usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/lib/Bio/Graphics/Glyph"
    </IfModule>
</VirtualHost>

Set up httpd to run under inet.d

Only required for the master server.

Remove /etc/init.d/apache2. It just confuses things.

Set which runlevels httpd will run under by chkconfig or command line: 

chkconfig --add httpd
chkconfig --level 2345 httpd on
chkconfig --list

sudo cp /usr/local/apache2/bin/apachectl /etc/init.d/httpd
cd /etc/rc3.d
sudo ln -s ../init.d/httpd S90httpd
cd /etc/rc5.d
sudo ln -s ../init.d/httpd S90httpd

Install mod_fcgid

> wget http://apache.sunsite.ualberta.ca/httpd/mod_fcgid/mod_fcgid-2.3.6.tar.gz
> tar zxvf mod_fcgid-2.3.6.tar.gz
> cd mod_fcgid-2.3.6
> perl -pi.bak -e 's/INITENV_VAL_LEN 128/INITENV_VAL_LEN 512/' modules/fcgid/fcgid_conf.h
> APXS=/usr/bin/apxs2 ./configure.apxs
> make
> sudo make install

Configure apache as described above.

For additional notes, see: 

http://gmod.org/wiki/Recompiling_mod_fcgid_to_avoid_truncated_Perl_library_paths

MySQL

Each node in the renderfarm has its own MySQL databases.

Installation (>= 5.1)

We need a newer version of mysql than is available from the package manager. We'lll install the binary for x86_64. 

$ sudo groupadd mysql
$ sudo useradd -g mysql mysql
$ sudo usermod -a -G mysql tharris
$ cd /usr/local
$ sudo gunzip < /path/to/mysql-VERSION-OS.tar.gz | sudo tar xvf -
$ sudo ln -s full-path-to-mysql-VERSION-OS mysql
$ cd mysql
$ sudo chown -R mysql .
$ sudo chgrp -R mysql .
$ sudo scripts/mysql_install_db --user=mysql  // may be necessary to remove /etc/mysql/my.cnf first
$ sudo chown -R root .
$ sudo chown -R mysql data
$ sudo chmod 2775 data // I like to be able to write to my datadir

// Copy the configuration file
$ sudo cp support_files/my-medium.cnf /etc/my.cnf

// It may be necessary to add the following to my.cnf if back end nodes
// don't have fully qualified domain names
skip-name-resolve

// Set up mysql to start up automatically
$ sudo cp support_files/mysql.server /etc/init.d/.
$ cd /etc/rc3.d
$ sudo ln -s ../init.d/mysql.server S90mysql
$ cd ../rc5.d
$ sudo ln -s ../init.d/mysql.server S90mysql

// Start it up and set a root password.
$ sudo /etc/init.d/mysql.server start
$ mysqladmin -u root password 'PASSWORD'

Install Requisite Perl Modules

All libraries specific for GBrowse are maintained in a local shared directory. This makes it easier and faster to upgrade GBrowse, but more difficult to test multiple versions simultaneously. To test a new version of GBrowse, place extlib inside of the gbrowse-current directory and modify the apache configuration and commands below as appropriate.

Install local::lib to make things easier: 

$ sudo perl -MCPAN -e 'CPAN::install(local::lib)'
$ mkdir /usr/local/wormbase/extlib
$ cd /usr/local/wormbase/extlib

To install modules to this path: 

$ perl -Mlocal::lib=.
$ eval $(perl -Mlocal::lib=./)
$ sudo rm -rf ~/.cpan
$ perl -MCPAN -e shell

cpan> install YAML DBI DBD::mysql CGI::Session JSON Storable FreezeThaw FCGI SVG GD::SVG Text::Shellwords \
           Algorithm::Munkres Array::Compare  Clone Convert::Binary::C Error Graph GraphViz HTML::Entities  HTML::HeadParser HTTP::Request::Common LWP::UserAgent \
           List::MoreUtils Math::Random PostScript::TextBlock SOAP::Lite SVG::Graph Set::Scalar Sort::Naturally Spreadsheet::ParseExcel URI::Escape \
           XML::DOM::XPath XML::Parser XML::Parser::PerlSAX XML::SAX XML::SAX::Writer XML::Simple XML::Twig XML::Writer \
           Capture::Tiny Template Term::ReadKey

cpan> install Bio::Perl        // may be necessary to install from source
cpan> install Bio::Graphics    // may be necessary to install from source

Install SamTools and Bio::SamTools

> cd /usr/local/wormbase/services/gbrowse2/src
> wget http://sourceforge.net/projects/samtools/files/samtools/0.1.16/samtools-0.1.16.tar.bz2/download
> bunzip2 samtools*
> cd samtools*
> make CXXFLAGS=-fPIC CFLAGS=-fPIC CPPFLAGS=-fPIC
> sudo cp misc/*.pl /usr/local/bin/

> wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/Bio-SamTools-1.28.tar.gz
> tar xzf Bio*
> cd Bio*
> export SAMTOOLS=/usr/local/wormbase/services/gbrowse2/src/samtools-0.1.16/
> perl Build.PL --install_base=/usr/local/wormbase/extlib
> ./Build
> ./Build test
> (sudo) ./Build install

Install Bio::BigFile.pm

To compile this module, you must first download, unpack and compile Jim Kent's source tree to create the main library file jkweb.a: 

 > wget http://hgdownload.cse.ucsc.edu/admin/jksrc.zip. 
 > unzip jksrc.zip
 > cd /usr/local/wormbase/build/kent/src/lib
 > export MACHTYPE=x86_64
 > mkdir $MACHTYPE
 > make CXXFLAGS=-fPIC CFLAGS=-fPIC CPPFLAGS=-fPIC

Download and build Bio::BigFile, eg v1.06: 

 > wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/Bio-BigFile-1.06.tar.gz
 > tar xzf Bio-* ; cd Bio-Big*
 > export KENT_SRC=/usr/local/wormbase/services/gbrowse2/src/kent/src/
 > perl Build.PL --install_base=/usr/local/wormbase/extlib
 > ./Build
 > ./Build test
 > (sudo) ./Build install

Install GBrowse2

$ mkdir /usr/local/wormbase/gbrowse2/gbrowse-version // ie gbrowse-2.03
$ ln -s gbrowse-version current
$ perl ./Build.PL --install_base /usr/local/wormbase/gbrowse2/current
$ perl ./Build --reconfig
Directory for GBrowse's config and support files? /usr/local/wormbase/services/gbrowse2/website-gbrowse2-support/conf
Directory for GBrowse's static images & HTML files? /usr/local/wormbase/services/gbrowse2/current/html
Directory for GBrowse's temporary data /usr/local/wormbase/tmp
Directory for GBrowse's sessions, uploaded tracks and other persistent data: /usr/local/wormbase/tmp-persistent
Directory for GBrowse's example databases /usr/local/wormbase/services/gbrowse2/current/databases
Directory for GBrowse's CGI script executables? /usr/local/wormbase/services/gbrowse2/current/cgi/gb2  
Internet port to run demo web site on (for demo)? [8000] 
Apache loadable module directory (for demo)? [/usr/local/apache2/modules] 
User account under which Apache daemon runs? [www-data] daemon
Automatically update Apache config files to run GBrowse? [n] 
Automatically update system config files to run gbrowse-slave? [n] y

To fetch suggested apache configuration: 

./Build apache_conf

Install dependencies 

./Build installdeps

To install: 

./Build install

You may need to fix permissions: 

chmod 777 /usr/local/wormbase/tmp

Install gbrowse-slave components on all servers

sudo ./Build install_slave

Edit /etc/default/gbrowse_slave: 

# Make sure that Perl5Lib includes that path to our external aggregator modules:
 export PERL5LIB=/usr/local/wormbase/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:\
    /usr/local/wormbase/extlib/lib/perl5:\
    /usr/local/wormbase/services/gbrowse2/current/lib/perl5:\
    /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib:\
    /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib/Bio:\
    /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib/Bio/Graphics:\
    /usr/local/wormbase/services/gbrowse2/current/website-gbrowse2-support/lib/Bio/Graphics/Glyph

DAEMON=/usr/local/bin/gbrowse_slave
# Be certain this matches the httpd daemon user on the master!
USER=www-data
#PRELOAD=/etc/gbrowse2/slave_preload.conf
RUNDIR=/usr/local/wormbase/logs
LOGDIR=/usr/local/wormbase/logs
PORT="8101 8102 8103"
VERBOSITY=3
NICE=0

Then set up the slave server to start automatically: 

$ cd /etc/rc3.d
$ sudo ln -s ../init.d/gbrowse-slave S99gbrowse-slave
$ cd /etc/rc5.d
$ sudo ln -s ../init.d/gbrowse-slave S99gbrowse-slave

Fetch configuration and support files

Configuration and all related gbrowse support files are maintained in git.

This includes conf, html, libs, css and some images. Apache2 configuration contains directives that map URIs to files in both the conf and support-files directories. In addition, the WormBase instance of GBrowse requires a few custom glyphs that seem to have disappeared in GB2. These are all contained in the support-files/lib directory. 

lib/Bio/Graphics/Glyph/full_transcript.pm
lib/Bio/Graphics/Glyph/wormbase_transcript.pm
lib/Bio/DB/GFF/Aggregator/
   blastz_alignment_briggsae.pm
   blastz_alignment_elegans.pm
   waba_alignment_briggsae.pm
   waba_alignment_elegans.pm
   waba_alignment.pm

Fetch the repository and rename it to "website-gbrowse2-support".

Note: Some of the files in the support-files directory may have already been installed by the GBrowse installer. You'll need to either replace them or update them in the repository

Configure NFS for load balancing

In lieue of the built-in GBrowse master/slave renderfarm configuration, we load balance across a pool of servers using a shared filesystem. See the NFS documentation in the main WormBase Production environment for details on how to install NFS.

NFS Server

The master server (wb-gb1.oicr.on.ca) provides an NFS mount of core GBrowse files and temporary session data. 

 /usr/local/wormbase/services/gbrowse2

This directory will be mounted at the same path for each client.

Contents:

Specify what to share, to whom, and with which privileges

Share mounts are configured through /etc/exports on wb-gb1.oicr.on.ca. 

# WormBase NFS for temporary and static content
/usr/local/wormbase/services/gbrowse2 206.108.125.174(rw,no_root_squash)

After making changes to /etc/exports, load them by: 

sudo exportfs -a

Lock down access to NFS from most hosts

We can use /etc/hosts.deny to quickly lock down NFS: 

portmap:ALL
lockd:ALL
mountd:ALL
rquotad:ALL
statd:ALL

Allow access to select hosts

We can use /etc/hosts.allow to specifically allow access to NFS: 

# WormBase NFS services 
portmap: ***.***.***.***
lockd: ***.***.***.***
rquotad: ***.***.***.***
mountd: ***.***.***.***
statd: ***.***.***.***

NFS clients

Manually mount the NFS share 

sudo mount 206.108.125.173:/usr/local/wormbase/tmp /usr/local/wormbase/shared

Unmounting the NFS share 

sudo umount /usr/local/wormbase/shared

Setting up NFS share to mount at boot 

sudo emacs /etc/fstab

Add 

wb-gb1.oicr.on.ca:/usr/local/wormbase/shared /usr/local/wormbase/shared nfs rw   0 0


Configure the master server

The configuration file of the master server should be set up with appropriate render slaves in the TRACK DEFAULTS section of each genome-level configuration file. 

# wb-www1: XXX.XXX.XXX.173 (master)                                                              
# wb-www2: XXX.XXX.XXX.174 (slave) 
# wb-www3: XXX.XXX.XXX.175 (slave) 

remote renderer = http://XXX.XXX.XXX.173:8101 http://XXX.XXX.XXX.173:8102 http://XXX.XXX.XXX.173:8103
                  http://XXX.XXX.XXX.174:8101 http://XXX.XXX.XXX.174:8102 http://XXX.XXX.XXX.174:8103
                  http://XXX.XXX.XXX.175:8101 http://XXX.XXX.XXX.175:8102 http://XXX.XXX.XXX.175:8103

Define dsn arguments (here we specify that the MySQL databases are running on the localhost). 

db_args     = -dsn DBI:mysql:c_elegans;user=nobody;host=localhost

# Here's a second example with all databases running on the master server.
db_args     = -dsn DBI:mysql:c_elegans;user=nobody;host=XXX.XXX.XXX.173

GBrowse slave servers

To add an additional slave server, simply follow the steps above, adding the name of the server to the TRACK DEFAULTS section of the genome-level configuration file.

No special configuration is required for GBrowse slave servers. Simply ensure that the slave has access to the MySQL databases using the username and password supplied in the primary configuration file.

Make sure the slave servers are running by: 

sudo /etc/init.d/gbrowse_slave start

Databases

MySQL databases will be mirrored over directly as part of the production update process. They should be located in /usr/local/mysql/data. Symlinks, using the symbolic name of the database, point to the current version: 

 c_elegans -> c_elegans_WS211

Database Privileges

Be sure to grant privileges to both master and slave servers to the databases: 

$ mysql -u root -p -e 'grant select on c_elegans.* to nobody@localhost'
$ mysql -u root -p -e 'grant select on c_elegans.* to nobody@render-slave-ip'

Administration

Updating GBrowse

To update GBrowse:

  1. Check out the source to /usr/local/wormbase/build
  2. Follow the directions above to install GBrowse, resetting the current symlink
  3. Check out the "wormbase-gbrowse2" configuration repository

Maintenance

Purging Temporary Files

GBrowse generates a slew of temporary files. Under production, these temporary files can quickly exhaust the ulimit of a machine. Set up the following cronjob to purge them periodically. 

1 1 * * * /usr/local/wormbase/gbrowse-support-files/purge_temporary_files.cron.sh

The contents of this script are: 

cd /usr/local/wormbase/gbrowse-current/tmp
find . -type f -atime +20 -print -exec rm {} \;

Log Rotation

gbrowse-support-files/ contains two configuration files for the unix logrotate command. For the master server copy logrotate-gbrowse-httpd.conf and logrotate-gbrowse-slave.conf to /etc/logrotate.d/gbrowse-httpd and gbrowse-slave. The slave nodes obviously only require the gbrowse-slave file.

The contents of these files are included here for reference: 

# logrotate-gbrowse-httpd.conf
# GBrowse httpd logrotate conf file
/usr/local/wormbase/logs/*.log {
	daily
	missingok
	rotate 7
	compress
	delaycompress
	notifempty
	create 640 root adm
	sharedscripts
	postrotate
                /usr/local/apache2/bin/apachectl graceful
	endscript
}

# logrotate-gbrowse-slave.conf
# GBrowse slave files 
/usr/local/wormbase/logs/*_slave {
	daily
	missingok
	rotate 7
	compress
	delaycompress
	notifempty
	sharedscripts
	postrotate
		/etc/init.d/gbrowse-slave stop
      	      	/etc/init.d/gbrowse-slave start
        endscript
}

Establishing a new slave node

Here's how to build a new slave node: 

1. Set up directories, users, groups
2. Install apache, fastcgi, mysql
3. Copy /usr/local/wormbase from an existing node

References

Author

Todd Harris, PhD (info@toddharris.net)

Retrieved from "https://wiki.wormbase.org/index.php?title=GBrowse_Administration&oldid=11195"