Difference between revisions of "Administration:Installing WormBase"
Line 143: | Line 143: | ||
$ sudo chown acedb:acedb /usr/local/wormbase/acedb | $ sudo chown acedb:acedb /usr/local/wormbase/acedb | ||
− | The website/ directory contains one or many versions of the website: | + | ;The website/ directory contains one or many versions of the website: |
* /usr/local/wormbase/website, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | * /usr/local/wormbase/website, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | ||
Line 151: | Line 151: | ||
$ sudo chgrp wormbase /usr/local/wormbase/website | $ sudo chgrp wormbase /usr/local/wormbase/website | ||
− | The services/ directory contains optional services such as e-pcr, blast, and nginx: | + | ;The services/ directory contains optional services such as e-pcr, blast, and nginx: |
* /usr/local/wormbase/services, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | * /usr/local/wormbase/services, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | ||
Line 159: | Line 159: | ||
$ sudo chgrp wormbase /usr/local/wormbase/services | $ sudo chgrp wormbase /usr/local/wormbase/services | ||
− | The databases/ directory contains support text-file database driving a number of features: | + | ;The databases/ directory contains support text-file database driving a number of features: |
* /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | * /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | ||
Line 167: | Line 167: | ||
$ sudo chgrp wormbase /usr/local/wormbase/databases | $ sudo chgrp wormbase /usr/local/wormbase/databases | ||
− | The logs/ directory contains WormBase-wide log and pid files | + | ;The logs/ directory contains WormBase-wide log and pid files |
* /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx | * /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx |
Revision as of 16:38, 24 December 2010
Contents
- 1 Overview
- 2 Recommended Hardware
- 3 Recommended Operating Systems
- 4 Prerequisites
- 5 Users And Groups
- 6 Create Directories
- 7 Set up your system
- 8 Checking out the Administration module
- 9 Generic Genome Browser
- 10 3rd-party Support Applications
- 11 Cronjobs
- 12 Testing The Site
- 13 FORMER VERSION OF THIS DOCUMENT
- 14 AUTHOR
Overview
This document describes how to install WormBase locally on your own machine.
Old documentation from Installing the Web Application, folding into this document
Recommended Hardware
As of Dec 2010, the suggested minimum hardware requirements for a basic installation of WormBase are:
- 500 GB disk space or greater
- 4 GB RAM or greater
Recommended Operating Systems
- Debian Linux "Lenny", 5.07
- Ubuntu 10.10
- Mac OS 10.6
This document only describes installation on Debian. Your actual mileage may vary on other operating systems.
Prerequisites
Ubuntu 9.04
libgd
install libgd2-xpm-dev(must be the developer version)
mysql
install mysql-client sudo apt-get install libmysqlclient16-dev
GraphViz
sudo apt-get install graphviz
BerkeleyDB
need to specify the lib&conf path in config.in
Mac OS 10.6 (Snow Leopard)
XCode > 3.21
You will need this for building C libraries as well as fixing the broken Perl that shipping with 10.6.
MySQL >= 5.1
MySQL is not expressly required, but you do need it for building DBD::mysql.
bash> cd src bash> curl -O http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.39.tar.gz bash> tar xzf mysql-5.1.41.tar.gz bash> cd mysql-5.1.41 bash> ./configure --prefix=/usr/local/mysql \ --with-extra-charsets=complex \ --enable-thread-safe-client \ --enable-local-infile \ --enable-shared \ --with-plugins=innobase bash> make bash> sudo make install bash> cd /usr/local/mysql bash> sudo ./bin/mysql_install_db --user=mysql bash> sudo chown -R mysql ./var
libpng
bash> curl -O http://softlayer.dl.sourceforge.net/project/libpng/00-libpng-stable/1.2.40/libpng-1.2.40.tar.gz bash> tar xzf libpng-* bash> ccd libpng* bash> ./configure bash> make bash> sudo make install
libgd
bash> curl -O http://www.libgd.org/releases/gd-2.0.35.tar.gz bash> tar xzf gd* bash> cd gd* bash> ./configure bash> make bash> sudo make install
Users And Groups
WormBase uses several user accounts for directory and server permissions. You will need to create these users and several preliminary directories. Creating a new user and group varies among Unix flavors. On most Linux systems, the following commands will create the new groups.
These users should not have a login password. They are to establish privileges only.
- acedb group
This is the group that will have write privileges to the acedb directory tree. Acedb administrators should be added to this group.
$ /usr/sbin/groupadd acedb
- acedb user
This is the user that the acedb server will run as. It should be a member of the acedb group.
$ /usr/sbin/useradd -g acedb -d /usr/local/wormbase/acedb acedb
This useradd command also adds the new acedb user to the acedb group. Note that the acedb user's home directory was set to /usr/local/acedb, a directory which will be created in the next step.
- wormbase group
This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group.
$ /usr/sbin/groupadd wormbase
This would be a good time to add yourself to the acedb and wormbase groups.
$ /usr/sbin/usermod -a -G acedb,wormbase [your_login_name]
You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of:
% groups
Create Directories
Create a root directory for all WormBase-related files. Although this directory can reside anywhere on your filesystem, for consistency with WormBase proper, we recommend installing at /usr/local/wormbase. You'll need suitable privileges to install in this location.'
- The root container for all things WormBase
- /usr/local/wormbase, owner=root group=wormbase mode=drwxrwsr-x
$ cd /usr/local $ sudo mkdir wormbase $ sudo chown root:wormbase wormbase $ sudo chmod 2775 wormbase
- The AceDB directory contains acedb and associated databases
- /usr/local/wormbase/acedb, owner=acedb group=acedb,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/acedb $ chmod 2775 usr/local/wormbase/acedb $ sudo chown acedb:acedb /usr/local/wormbase/acedb
- The website/ directory contains one or many versions of the website
- /usr/local/wormbase/website, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/website $ chmod 2775 usr/local/wormbase/website $ sudo chgrp wormbase /usr/local/wormbase/website
- The services/ directory contains optional services such as e-pcr, blast, and nginx
- /usr/local/wormbase/services, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/services $ chmod 2775 usr/local/wormbase/services $ sudo chgrp wormbase /usr/local/wormbase/services
- The databases/ directory contains support text-file database driving a number of features
- /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/databases $ chmod 2775 usr/local/wormbase/databases $ sudo chgrp wormbase /usr/local/wormbase/databases
- The logs/ directory contains WormBase-wide log and pid files
- /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx
$ mkdir /usr/local/wormbase/logs $ chmod 2775 usr/local/wormbase/logs $ sudo chgrp wormbase /usr/local/wormbase/logs
MySQL data directory: /usr/local/mysql/data
Installing Libraries
Assuming a vanilla Debian installation, install the following libaries and all of their dependencies via sudo apt-get install. Some of these libraries are discussed below.
gcc curl wget bzip2 mysql-server mysql-server-5.0 libgd2-xpm-dev libgd2-xpm xinetd libdbd-mysql libdbd-mysql-perl libapache2-mod-perl2 libgtk2.0-0 libgtk2.0-dev libglib byacc libreadline5-dev flex libdb4.6 libdb-dev emacs hg
Set up your system
Install Perl 5.10.1 or greater
You'll need Perl version 5.10.1 or greater. We recommend installing a local version of Perl and required libraries. This makes management dramatically easier. We keep our Perl with the webapp itself.
If you have sufficient privileges, you may wish to use your pre-installed system Perl.
bash> mkdir ~/website/perl bash> cd ~/src bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz bash> cd ~/build bash> tar xzf ../src.perl-5.10.1.tar.gz bash> cd perl-5.10.1 bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1 bash> make bash> make test bash> make install
Be sure to set up your your system to preferentially use this newly installed Perl:
bash> export PATH=~/wormbase/perl/5.10.1/bin:${PATH}
Check out the source code
See the WormBase source code repository documentation for additional details on using Mercurial. You may need to install mercurial first.
Fetch the current production code from the mercurial repository.
bash> cd $ROOT bash> hg clone ssh://hg@bitbucket.org/tharris/wormbase/ website-2.0 bash> ln -s website-2.0 website
Build the web app (and Perl dependencies)
Automated Method
The preferred method uses the web app's Makefile.PL and Module::Install to build all dependencies.
If you've installed your own Perl (or have sufficient privileges for the system Perl) you can install modules within the default site_perl paths.
However, as above, we recommend installing Perl modules in a local path (and not in the system path). This makes management and upgrading substantially easier.
local::lib makes installling libraries in a local path a snap.
# Install local::lib. You may need to be root depending on your Perl config. bash> perl -MCPAN -e 'CPAN::install(local::lib)' bash> cd ~/projects; mkdir extlib # a local directory for libraries bash> perl -Mlocal::lib=./ bash> eval $(perl -Mlocal::lib=./)
# Build the web app and all Perl dependencies into ~/wormbase/extlib # You may need to "perl -MCPAN -e 'CPAN::install(Module::Install)' first bash> cd $ROOT bash> perl Makefile.PL bash> make
make will install all necessary dependencies, choosing appropriate defaults. Still, some modules require extra care on select architectures. See below for details.
Manual Method
You can also install all modules manually. Follow the documentation at Managing Perl Libraries for details on how to build and install required Perl modules.
Checking out the Administration module
Fetch the administration code repository (privileged users only, for now)
todd> mkdir -p ~/projects/wormbase todd> cd ~/projects/wormbase todd> hg clone ssh://hg@bitbucket.org/tharris/wormbase-admin/ // Assuming you have already set up your ssh keys at bitbucket
Generic Genome Browser
See Managing GBrowse for details on how to build and install GBrowse.
3rd-party Support Applications
nginx
We use nginx as an (optional) lightweight proxy server. You can skip this step if desired.
By default, we set nginx to listen on port 80 and forward requests to starman listening on port 5000.
Configuration
starman: the lightweight http server
AceDB
I always build acedb from source.
$ tar xzf ACEDB-source* // CAUTION: Tarbomb. // Requires installation of a whole bunch of things first: libgtk2.0-0 libgtk2.0-dev libglib, byacc, etc, etc // Modify the makefile: create a target for server programs (xace tace saceserver sgifacerver) // This is all I care about: SERVERS = xace tace saceserver sgifaceserver saceclient servers: $(SERVERS) $ export ACEDB_MACHINE=LINUX_4 $ make servers $ cd ~acedb $ ln -s bin-VERSION bin $ cp tace xace sgifaceserver saceserver saceclient ~acedb/bin/. $ sudo chown root:root ~acedb/bin/*
Testing the ACeDB Installation
At this point, you can test whether the socket server runs correctly. Provided that you have added yourself to the acedb group, you can run the following command:
% ~acedb/bin/sgifaceserver ~acedb/wormbase // Database directory: /usr/local/wormbase/acedb/wormbase // Shared files: /usr/local/acedb // #### Server started at 2001-07-23_16:42:31 // #### host=mondseer.cshl.org listening port=23100 // #### Database dir=/usr/local/acedb/elegans // #### Working dir=/usr/local/acedb/elegans // #### clientTimeout=600 serverTimeout=600 maxKbytes=0 autoSaveInterval=600
// Server listening socket 28 created
The line "listening port=23100" indicates that the server is listening to port 23100. Open a new terminal window and use saceclient to confirm that you can communicate with the server:
% ~acedb/bin/saceclient localhost -port 23100 Please enter userid: anonymous Please enter passwd: acedb@localhost> find Sequence // Response: 65 bytes.
// Found 236493 objects in this class // 236493 Active Objects acedb@localhost> quit // Closing connection to server. // Client sent termination signal by server. // Response: 13 bytes. // A bientot // Please report problems to acedb@sanger.ac.uk // Bye
Configuring Acedb to start automatically under xinetd
Install xinetd (not standard in Debian) if you didn't already:
$ sudo apt-get install xinetd
Create a configuration file for acedb:
$ sudo emacs /etc/xinetd.d/acedb-wormbase # file: /etc/xinetd.d/acedb-wormbase # default: on # description: wormbase acedb database service acedb { protocol = tcp socket_type = stream port = 2005 flags = REUSE wait = yes user = acedb group = acedb log_on_success += USERID DURATION log_on_failure += USERID HOST server = /usr/local/wormbase/acedb/bin/sgifaceserver server_args = /usr/local/wormbase/acedb/wormbase 1200:1200:0 }
Edit /etc/services. Although xinetd is not supposed to use /etc/services, the following line must be added:
acedb-wormbase 2005/tcp
Restart xinetd with the following command:
$ sudo /etc/init.d/xinetd reload (or restart)
You should now be able to talk to the database using saceclient:
$ ~acedb/bin/saceclient localhost -port 2005
MySQL
Installation
Install mysql and various libraries via apt-get if you haven't already:
$ sudo apt-get install mysql-server-5.0 mysql-server libdbd-mysql libdbd-mysql-perl
If it fails, then disable innodb by default. edit /etc/mysql/my.cnf file (uncomment the line):
#skip-innodb $ sudo apt-get purge mysql-server-5.0 mysql-server $ sudo apt-get install mysql-server-5.0 mysql-server
With this installation, databases are located at /var/lib/mysql. We want to relocate this to /usr/local/mysql/data to be consistent with existing nodes. We also want to able to write to this directory from the command line, so:
$ sudo nano /etc/my.cnf Set datadir to /usr/local/mysql/data $ mkdir -p /usr/local/mysql/data $ sudo chown -R mysql:mysql /usr/local/mysql $ sudo chmod 2775 /usr/local/mysql/data
Mysqld will automatically be setup to launch at server boot (rc3 and rc5).
Set up mysql permissions
$ mysql -u root -pPASSWORD mysql> grant select on elegans.* to nobody@localhost;
Repeat for:
- c_briggsae
- c_japonica
- c_remanei
- c_brenneri
- p_pacificus
- b_malayi
- c_elegans_gmap
- c_elegans_pmap
- autocomplete
- h_bacteriophora
e-PCR
- e-PCR (modified version, required for e-PCR search page)
This is located in the directory /usr/local/wormbase/e-PCR, which will come into existence after the WormBase site update program wb_update_wormbase.pl has been successfully run (see below for details). Once the directory has been generated, run:
$ cd /usr/local/wormbase/e-PCR # Edit 'makefile' to run install rather than ginstall, which doesn't exist on Fedora Linux $ make $ make install # or just run 'install e-PCR /usr/local/bin'
The file /usr/local/wormbase/e-PCR/README-Wormbase describes the changes that were made to the original e-PCR distribution.
BLAT
Jim Kent's BLAT (blast-like alignment tool) is a fast nucleotide aligner used by the blast search page. If you do not plan to support blast searches, you may safely skip this step.
# mkdir -p /usr/local/blat/bin ; cd /usr/local/blat/bin % wget http://www.soe.ucsc.edu/~kent/exe/linux/blatSuite.33.zip (for Intel Linux) % unzip blatSuite.33.zip % rm blatSuite.33.zip version.doc 11.ooc
Note that this choice gives precompiled binaries for an Intel-based Linux distribution as of March 2006. It would probably be worth checking http://www.soe.ucsc.edu/~kent/exe/linux to see if there is a more up-to-date version than 33. Also, other operating systems will need other binaries. E.g., for Mac OS X, instead run:
% wget http://www.soe.ucsc.edu/~kent/exe/osX/blatSuite.33.zip
For other types of operating systems (e.g., Linux on Opteron-based machines), see http://www.soe.ucsc.edu/~kent/exe/ for the available choices.
The blat server will be started automatically by the update script. For reference, the blat server is launched using the following command.
% /usr/local/blat/bin/gfServer start localhost 2003 \ /usr/local/wormbase/blat/*.nib & > /dev/null 2>&1
BLAST
The Blast page requires WU-BLAST. This is a closed-source derivative of NCBI's BLAST. However, WU-BLAST is free to academic users (with licensing) and is thought to have performance advantages over NCBI-BLAST; it can be downloaded from http://blast.wustl.edu/. A typical choice of WU-BLAST for Linux is blast2.linux26-i686.tar.gz.
Conversely, the Blast page can be deactivated if you don't want to provide BLAST searches at your site.
By default, WormBase expects WU-BLAST to be installed in /usr/local/wublast. This is the directory structure used by WormBase:
% ls -l /usr/local/wublast ls -l /usr/local/wublast total 72 lrwxrwxrwx 1 root root 18 May 7 12:26 BLOSUM62 -> matrix/aa/BLOSUM62 -rw-r--r-- 1 root root 46789 Feb 5 1998 HISTORY -rw-r--r-- 1 root root 6648 Mar 4 1997 README drwxr-xr-x 2 root root 4096 May 7 12:46 bin/ lrwxrwxrwx 1 root root 25 Jul 24 08:20 databases -> /usr/local/wormbase/blast/ drwxr-xr-x 2 root root 4096 Jan 27 2000 filter/ drwxr-xr-x 4 root root 4096 Oct 4 1998 matrix/
which can be set up in this manner (adapt to your system):
$ cd /usr/local/wublast $ zcat /usr/local/TGZ/blast2.linux26-i686.tar.gz | tar xf - $ chown -R root:root * $ mkdir bin $ mv *fasta tblast* blast* *db xd* memfile pam wu-blastall bin $ ln -s /usr/local/wormbase/blast databases
The important thing to note is that the databases directory is a symbolic link to /usr/local/wormbase/blast. This is where the update_wormbase.pl script (described in the next section) dumps its BLAST databases.
Cronjobs
Set up the following cronjobs:
Log Rotation (root cron)
# Rotate WormBase logs 10 1 * * * /home/todd/projects/wormbase/admin/maintenance/rotate_httpd_logs.pl
For nodes running acedb, add the following entry which deals with its massive log files:
# Purge epic ACEDB logs 35 * * * * /home/todd/projects/wormbase/admin/maintenance/purge_acedb_logs.sh
The final step is to arrange for Acedb to start automatically and for MySQL to restart if necessary.
Installing MySQL and BLAT monitoring scripts
Run:
$ cp -i /usr/local/wormbase/util/admin/blat_server.initd /etc/rc.d/init.d/blat_server
Then run:
$ crontab -u root -e
to add the following entries to root's crontab:
0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl 0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl
Installing scripts to verify that the servers are running
Two scripts in the WormBase directory can be used to ensure that the mysql and blat servers are running. To install, them:
% sudo cp /usr/localwormbase/util/admin/blat_server.initd \ /etc/rc.d/init.d/blat_server
Place the restart scripts under cron control of a privileged user. These commands will check every hour to see that the servers are running.
% sudo crontab -u root -e
0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl 0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl
At the same time, you might also wish to automate the rotatation of logs to prevent them from growing to an unwieldy size. You'll find an appropriate log rotation configuration stanza in util/rotate_wormbase_logs and a log rotate script in /usr/local/wormbase/bin/rotatelogs.pl. You will need both.
# Rotate httpd logs 10 1 * * * /usr/local/wormbase/bin/rotatelogs.pl # Rotate acedb logs 10 1 * * * logrotate /usr/local/wormbase/util/rotate_wormbase_logs
This stanza will check that the acedb server logs do not grow larger than 100 MB.
Testing The Site
At this point, all components of a WormBase installation have been installed. You can test your installation by restarting the various server components of WormBase.
Restarting AceDB
# Via xinetd: $ /etc/init.d/xinetd reload (or restart)
# ...or using saceclient % saceclient localhost -port 2005 acedb> password: acedb> shutdown now
Restarting MySQL
# Via mysqladmin... % mysqladmin -uroot -pPASSWORD shutdown # or using init.d $ /etc/init.d/mysql restart
Restarting Apache
When the configuration files have been checked and adjusted, restart Apache with the following command:
$ /etc/init.d/apache restart
Check /usr/local/wormbase/logs/classic-error_log for WormBase-specific errors and /var/log/apache2/error_log for general errors.
BLAT
% /usr/local/blat/bin/gfServer start localhost 2003 \ /usr/local/wormbase/blat/*.nib & > /dev/null 2>&1
Blocking robots
It can be useful to block search engines (such as Google) from crawling over one's mirror. To do this, go to /usr/local/wormbase/html, and make a file called "robots.txt" with the following contents:
User-agent: * Disallow: /
FORMER VERSION OF THIS DOCUMENT
A much earlier, and now obsolete, version of these site-installation instructions can be seen here.
AUTHOR
Todd Harris (toddwharris@gmail.com)