Difference between revisions of "Administration:Installing WormBase"
(10 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
= Overview = | = Overview = | ||
− | This document describes how to install WormBase locally on your own machine. | + | This document describes how to install WormBase locally on your own machine. Please note that as of August 2013, WormBase is almost entirely a distributed cloud-based application. Although possible to install and run on a single server (or cloud instance) the directions below may be slightly out-of-sync with how the principal WormBase architecture is built. |
= Recommendations = | = Recommendations = | ||
Line 14: | Line 14: | ||
== Operating Systems == | == Operating Systems == | ||
− | ''You should be able to install and run the site on any Unix-y OS, including Mac OS X. This document only describes installation on Debian. Your actual mileage may vary on other operating systems.'' | + | ''You should be able to install and run the site on any Unix-y OS, including Mac OS X. This document only describes installation on Debian and Debian-ish OSes like Amazon's Linux. Your actual mileage may vary on other operating systems.'' |
= Basic System Configuration = | = Basic System Configuration = | ||
Line 40: | Line 40: | ||
This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group. | This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group. | ||
− | $ | + | $ groupadd wormbase |
This would be a good time to add yourself to the acedb and wormbase groups. | This would be a good time to add yourself to the acedb and wormbase groups. | ||
− | $ | + | $ usermod -a -G acedb,wormbase [your_login_name] |
You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of: | You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of: | ||
Line 87: | Line 87: | ||
$ sudo chgrp wormbase /usr/local/wormbase/services | $ sudo chgrp wormbase /usr/local/wormbase/services | ||
− | '''The databases/ directory contains | + | '''The databases/ directory contains databases driving a number of features:''' |
* /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | * /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x | ||
Line 97: | Line 97: | ||
'''The logs/ directory contains WormBase-wide log and pid files''' | '''The logs/ directory contains WormBase-wide log and pid files''' | ||
− | * /usr/local/wormbase/ | + | * /usr/local/wormbase/logs, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx |
$ mkdir /usr/local/wormbase/logs | $ mkdir /usr/local/wormbase/logs | ||
Line 121: | Line 121: | ||
libgtk2.0-dev \ | libgtk2.0-dev \ | ||
emacs \ | emacs \ | ||
− | git | + | git \ |
byacc \ | byacc \ | ||
− | + | libreadline-gplv2-dev \ | |
flex \ | flex \ | ||
libdb4.6 \ | libdb4.6 \ | ||
Line 133: | Line 133: | ||
graphviz \ | graphviz \ | ||
libssl-dev \ | libssl-dev \ | ||
− | |||
− | |||
uuid \ | uuid \ | ||
uuid-dev \ | uuid-dev \ | ||
Line 140: | Line 138: | ||
xsltproc \ | xsltproc \ | ||
libgdbm3 \ | libgdbm3 \ | ||
− | libgdbm3-dev | + | libgdbm3-dev \ |
− | libstdc++ | + | libstdc++ \ |
− | libstdc++-devel | + | libstdc++-devel |
Build xapian (or install from package if available): | Build xapian (or install from package if available): | ||
− | wget http://oligarchy.co.uk/xapian/1.2.5/xapian-core-1.2.5.tar.gz | + | $ wget http://oligarchy.co.uk/xapian/1.2.5/xapian-core-1.2.5.tar.gz |
− | tar xzf xapian-core*.tgz | + | $ tar xzf xapian-core*.tgz |
− | cd xapian-core | + | $ cd xapian-core |
− | ./configure | + | $ ./configure |
− | make | + | $ make |
− | sudo make install | + | $ sudo make install |
− | |||
− | |||
− | |||
− | + | Install Search::Xapian | |
+ | $ perl -MCPAN -e 'install Search::Xapian' | ||
== Install Perl 5.10.1 or greater == | == Install Perl 5.10.1 or greater == | ||
Line 223: | Line 219: | ||
= Install the Webapp = | = Install the Webapp = | ||
− | + | '''Fetch the current production code from the git repository on github.''' | |
− | |||
− | '''Fetch the current production code from the | ||
$ cd /usr/local/wormbase/website | $ cd /usr/local/wormbase/website | ||
$ git clone git@github.com:WormBase/website.git | $ git clone git@github.com:WormBase/website.git | ||
− | $ mv website | + | $ mv website YOUR_NAME // name can be anything you want; useful for maintaining multiple versions |
'''Install local:: lib''' | '''Install local:: lib''' | ||
Line 242: | Line 236: | ||
''See the documentation on [[Managing Perl Libraries]] for additional details.'' | ''See the documentation on [[Managing Perl Libraries]] for additional details.'' | ||
− | |||
= Required 3rd-party Applications = | = Required 3rd-party Applications = | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== Starman: the lightweight http server == | == Starman: the lightweight http server == | ||
Line 330: | Line 261: | ||
The PSGI glue resides at: | The PSGI glue resides at: | ||
− | + | website/YOUR_NAME/wormbase.psgi | |
''' Starting Starman ''' | ''' Starting Starman ''' | ||
− | starman script/ | + | starman script/wormbase.psgi |
OR | OR | ||
starman -MFindBin script/wormbase_production.psgi | starman -MFindBin script/wormbase_production.psgi | ||
Line 844: | Line 775: | ||
= AUTHOR = | = AUTHOR = | ||
− | [[User:Tharris|Todd Harris]] ( | + | [[User:Tharris|Todd Harris]] (todd@wormbase.org) |
+ | |||
+ | |||
+ | [[Category: Architecture (Web Dev)]] |
Latest revision as of 19:10, 18 June 2014
Contents
Overview
This document describes how to install WormBase locally on your own machine. Please note that as of August 2013, WormBase is almost entirely a distributed cloud-based application. Although possible to install and run on a single server (or cloud instance) the directions below may be slightly out-of-sync with how the principal WormBase architecture is built.
Recommendations
Hardware
As of April 2012, the suggested minimum hardware requirements for a basic installation of WormBase are:
- 500 GB disk space (1 TB preferable)
- 8 GB RAM
Operating Systems
You should be able to install and run the site on any Unix-y OS, including Mac OS X. This document only describes installation on Debian and Debian-ish OSes like Amazon's Linux. Your actual mileage may vary on other operating systems.
Basic System Configuration
Users And Groups
These users should not have a login password. They are to establish privileges only.
acedb group
This is the group that will have write privileges to the acedb directory tree. Acedb administrators should be added to this group.
$ groupadd acedb
acedb user
This is the user that the acedb server will run as. It should be a member of the acedb group.
$ useradd -g acedb -d /usr/local/wormbase/acedb acedb
This useradd command also adds the new acedb user to the acedb group. Note that the acedb user's home directory was set to /usr/local/acedb, a directory which will be created in the next step.
wormbase group
This is a group that will have write privileges to the wormbase directory tree. WormBase administrators and authors should be added to this group.
$ groupadd wormbase
This would be a good time to add yourself to the acedb and wormbase groups.
$ usermod -a -G acedb,wormbase [your_login_name]
You may need to re-login for these changes to take effect. Use the groups command to check which groups you are a member of:
% groups
Create Directories
Create a root directory for all WormBase-related files. Although this directory can reside anywhere on your filesystem, for consistency with WormBase proper, we recommend installing at /usr/local/wormbase. You'll need suitable privileges to install in this location.'
The root container for all things WormBase
- /usr/local/wormbase, owner=root group=wormbase mode=drwxrwsr-x
$ cd /usr/local $ sudo mkdir wormbase $ sudo chown root:wormbase wormbase $ sudo chmod 2775 wormbase
The AceDB directory contains acedb and associated databases:
- /usr/local/wormbase/acedb, owner=acedb group=acedb,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/acedb $ chmod 2775 /usr/local/wormbase/acedb $ sudo chown acedb:acedb /usr/local/wormbase/acedb
The website/ directory contains one or many versions of the website:
- /usr/local/wormbase/website, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/website $ chmod 2775 /usr/local/wormbase/website $ sudo chgrp wormbase /usr/local/wormbase/website
The services/ directory contains optional services such as e-pcr, blast, and nginx:
- /usr/local/wormbase/services, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/services $ chmod 2775 /usr/local/wormbase/services $ sudo chgrp wormbase /usr/local/wormbase/services
The databases/ directory contains databases driving a number of features:
- /usr/local/wormbase/databases, owner=[ANYONE] group=wormbase,mode=drwxrwsr-x
$ mkdir /usr/local/wormbase/databases $ chmod 2775 /usr/local/wormbase/databases $ sudo chgrp wormbase /usr/local/wormbase/databases
The logs/ directory contains WormBase-wide log and pid files
- /usr/local/wormbase/logs, owner=[ANYONE] group=wormbase,mode=drwxrwwrwx
$ mkdir /usr/local/wormbase/logs $ chmod 2775 /usr/local/wormbase/logs $ sudo chgrp wormbase /usr/local/wormbase/logs
Installing Library Prerequisites
Assuming a vanilla Debian installation, install the following libaries and all of their dependencies via sudo apt-get install. Some of these libraries are discussed below.
sharutils \ gcc g++ \ curl wget \ bzip2 \ mysql-client \ mysql-server \ libgd2-xpm-dev\ libgd2-xpm \ xinetd \ libdbd-mysql \ libdbd-mysql-perl \ libapache2-mod-perl2 \ libgtk2.0-0 \ libgtk2.0-dev \ emacs \ git \ byacc \ libreadline-gplv2-dev \ flex \ libdb4.6 \ libdb-dev \ xml2 libxml2 libxml2-dev \ libmysql++-dev \ ncurses-bin \ libxmu-dev \ graphviz \ libssl-dev \ uuid \ uuid-dev \ libxslt1-dev \ xsltproc \ libgdbm3 \ libgdbm3-dev \ libstdc++ \ libstdc++-devel
Build xapian (or install from package if available):
$ wget http://oligarchy.co.uk/xapian/1.2.5/xapian-core-1.2.5.tar.gz $ tar xzf xapian-core*.tgz $ cd xapian-core $ ./configure $ make $ sudo make install
Install Search::Xapian
$ perl -MCPAN -e 'install Search::Xapian'
Install Perl 5.10.1 or greater
You'll need Perl version 5.10.1 or greater.
Upgrading Perl via apt (recommended)
Let your operating system managing your version of Perl.
Configure your apt preferences
Add the following to /etc/apt/sources.list
#Stable deb http://ftp.us.debian.org/debian stable main non-free contrib #Testing deb http://ftp.us.debian.org/debian testing main non-free contrib #Unstable deb http://ftp.us.debian.org/debian unstable main non-free contrib
Configure your preferences in /etc/apt/preferences:
Package: * Pin: release a=stable Pin-Priority: 700 Package: * Pin: release a=testing Pin-Priority: 650 Package: * Pin: release a=unstable Pin-Priority: 600
Finally, install Perl
sudo apt-get update sudo apt-get install perl/testing perl-base/testing perl-modules/testing libperl5.10/testing libperl-dev/testing
And confirm
$ perl -v This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi (with 40 registered patches, see perl -V for more detail)
Installing Perl from source
We recommend installing a local version of Perl and required libraries. This makes management dramatically easier. We keep our Perl with the webapp itself.
If you have sufficient privileges, you may wish to use your pre-installed system Perl.
bash> mkdir ~/website/perl bash> cd ~/src bash> curl -O http://www.cpan.org/src/perl-5.10.1.tar.gz bash> cd ~/build bash> tar xzf ../src.perl-5.10.1.tar.gz bash> cd perl-5.10.1 bash> ./Configure -des -Dprefix=$HOME/website/perl/5.10.1 bash> make bash> make test bash> make install
Be sure to set up your your system to preferentially use this newly installed Perl:
bash> export PATH=~/wormbase/perl/5.10.1/bin:${PATH}
Install the Webapp
Fetch the current production code from the git repository on github.
$ cd /usr/local/wormbase/website $ git clone git@github.com:WormBase/website.git $ mv website YOUR_NAME // name can be anything you want; useful for maintaining multiple versions
Install local:: lib
The preferred method uses the web app's Makefile.PL and Module::Install to build all dependencies. We maintain local copies of all libraries with each version of the webapp.
Build and install dependencies
$ mkdir /usr/local/wormabase/extlib $ cd /usr/local/wormbase/website/YOUR_NAME $ perl Makefile.PL $ make installdeps
See the documentation on Managing Perl Libraries for additional details.
Required 3rd-party Applications
Starman: the lightweight http server
PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).
http://plackperl.org/
Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.mod_perl.
https://github.com/miyagawa/Starman
Install Plack and Starman
- Catalyst::Controller::Metal
- Catalyst::Engine::PSGI
- Catalyst::Helper::PSGI
- Plack::Test::Adopt::Catalyst
- cpanm Task::Plack
- cpanm Starman
Configuration
The PSGI glue resides at:
website/YOUR_NAME/wormbase.psgi
Starting Starman
starman script/wormbase.psgi OR starman -MFindBin script/wormbase_production.psgi
Set up starman to launch automatically
Copy the stub init script from wormbase/conf/starman/starman.init:
cp /usr/local/wormbase/admin/conf/starman/starman.init
AceDB
I always build acedb from source.
$ tar xzf ACEDB-source* // CAUTION: Tarbomb. // Requires installation of a whole bunch of things first: libgtk2.0-0 libgtk2.0-dev libglib, byacc, etc, etc // Modify the makefile: create a target for server programs (xace tace saceserver sgifacerver) // This is all I care about: SERVERS = xace tace saceserver sgifaceserver saceclient servers: $(SERVERS) $ export ACEDB_MACHINE=LINUX_4 $ make servers $ cd ~acedb $ ln -s bin-VERSION bin $ cp tace xace sgifaceserver saceserver saceclient ~acedb/bin/. $ sudo chown root:root ~acedb/bin/*
Testing the ACeDB Installation
At this point, you can test whether the socket server runs correctly. Provided that you have added yourself to the acedb group, you can run the following command:
% ~acedb/bin/sgifaceserver ~acedb/wormbase // Database directory: /usr/local/wormbase/acedb/wormbase // Shared files: /usr/local/acedb // #### Server started at 2001-07-23_16:42:31 // #### host=mondseer.cshl.org listening port=23100 // #### Database dir=/usr/local/acedb/elegans // #### Working dir=/usr/local/acedb/elegans // #### clientTimeout=600 serverTimeout=600 maxKbytes=0 autoSaveInterval=600 // Server listening socket 28 created
The line "listening port=23100" indicates that the server is listening to port 23100. Open a new terminal window and use saceclient to confirm that you can communicate with the server:
% ~acedb/bin/saceclient localhost -port 23100 Please enter userid: anonymous Please enter passwd: acedb@localhost> find Sequence // Response: 65 bytes.
// Found 236493 objects in this class // 236493 Active Objects acedb@localhost> quit // Closing connection to server. // Client sent termination signal by server. // Response: 13 bytes. // A bientot // Please report problems to acedb@sanger.ac.uk // Bye
Configuring Acedb to start automatically under xinetd
Install xinetd (not standard in Debian) if you didn't already:
$ sudo apt-get install xinetd
Create a configuration file for acedb:
$ sudo emacs /etc/xinetd.d/acedb-wormbase # file: /etc/xinetd.d/acedb-wormbase # default: on # description: wormbase acedb database service acedb-wormbase { protocol = tcp socket_type = stream port = 2005 flags = REUSE wait = yes user = acedb group = acedb log_on_success += USERID DURATION log_on_failure += USERID HOST server = /usr/local/wormbase/acedb/bin/sgifaceserver server_args = /usr/local/wormbase/acedb/wormbase 1200:1200:0 }
Edit /etc/services. Although xinetd is not supposed to use /etc/services, the following line must be added:
acedb-wormbase 2005/tcp
Restart xinetd with the following command:
$ sudo /etc/init.d/xinetd reload (or restart)
You should now be able to talk to the database using saceclient:
$ ~acedb/bin/saceclient localhost -port 2005
MySQL
Installation
Install mysql and various libraries via apt-get if you haven't already. You will need the unstable and testing repositories set up when upgrading Perl.
$ sudo apt-get install libncurses5/testing $ sudo apt-get install mysql-common/testing libmysqlclient16 mysql-server-5.1/testing \ mysql-client-5.1/testing libdbd-mysql/testing libdbd-mysql-perl/testing
With this installation, databases are located at /var/lib/mysql. We want to relocate this to /usr/local/mysql/data to be consistent with the WormBase. We also want to able to write to this directory from the command line, so:
$ sudo nano /etc/my.cnf Set datadir to /usr/local/mysql/data $ mkdir -p /usr/local/mysql/data $ sudo chown -R mysql:mysql /usr/local/mysql $ sudo chmod 2775 /usr/local/mysql/data
Mysqld will automatically be setup to launch at server boot (rc3 and rc5).
Set up mysql permissions
$ mysql -u root -pPASSWORD mysql> grant select on c_elegans.* to nobody@localhost;
Repeat for:
- autocomplete
- b_malayi
- c_briggsae
- c_elegans
- c_elegans_gmap
- c_elegans_pmap
- c_japonica
- c_remanei
- c_brenneri
- h_bacteriophora
- p_pacificus
- ps1010
Generic Genome Browser
See GBrowse Administration for details on how to install GBrowse.
Optional 3rd-party Applications
nginx
We use nginx as a lightweight proxy server and load balancing server. You can skip this step and simply connect directly to the Starman http server. Otherwise, see the production environment documentation on installing and configuring nginx.
BLAST
The BLAST page requires WU-BLAST. This is a closed-source derivative of NCBI's BLAST. However, WU-BLAST is free to academic users (with licensing) and is thought to have performance advantages over NCBI-BLAST; it can be downloaded from http://blast.wustl.edu/. A typical choice of WU-BLAST for Linux is blast2.linux26-i686.tar.gz.
You can skip this step if you don't want to provide BLAST services at your site.
By default, WormBase expects WU-BLAST to be installed in /usr/local/wormbase/services/wublast. This is the directory structure used by WormBase:
$ ls -l /usr/local/wormbase/services/wublast
lrwxrwxrwx 1 root root 18 May 7 12:26 BLOSUM62 -> matrix/aa/BLOSUM62 -rw-r--r-- 1 root root 46789 Feb 5 1998 HISTORY -rw-r--r-- 1 root root 6648 Mar 4 1997 README drwxr-xr-x 2 root root 4096 May 7 12:46 bin/ lrwxrwxrwx 1 root root 25 Jul 24 08:20 databases -> /usr/local/wormbase/blast/ drwxr-xr-x 2 root root 4096 Jan 27 2000 filter/ drwxr-xr-x 4 root root 4096 Oct 4 1998 matrix/
e-PCR
A modified version of the epcr source can be found in the checked out website code in src/. See the enclosed README describing changes made to the original e-PCR distribution. The binary should be placed in /usr/local/wormbase/services.
$ cd /usr/local/wormbase/website/production/src $ make $ install e-PCR /usr/local/wormbase/services/epcr
BLAT
Jim Kent's BLAT (blast-like alignment tool) is a fast nucleotide aligner used by the blast search page. If you do not plan to support blast searches, you may safely skip this step.
$ mkdir -p /usr/local/wormbase/services/blat/bin $ cd /usr/local/wormbase/services/blat/bin $ wget http://www.soe.ucsc.edu/~kent/exe/linux/blatSuite.33.zip (for Intel Linux) $ unzip blatSuite.33.zip $ rm blatSuite.33.zip version.doc 11.ooc
Note that this choice gives precompiled binaries for an Intel-based Linux distribution as of March 2006. It would probably be worth checking http://www.soe.ucsc.edu/~kent/exe/linux to see if there is a more up-to-date version than 33. Also, other operating systems will need other binaries. E.g., for Mac OS X, instead run:
$ wget http://www.soe.ucsc.edu/~kent/exe/osX/blatSuite.33.zip
The blat server will be started automatically by the update script. For reference, the blat server is launched using the following command.
$ /usr/local/wormbase/services/blat/bin/gfServer start localhost 2003 \ /usr/local/wormbase/databases/VERSION/blat/*.nib & > /dev/null 2>&1
Installing Databases NOT DONE
Primary database (AceDB)
GFF Sequence feature database (MySQL)
Support Databases
Support databases are automatically synced to the nodes that require them by wormbase-admin/update/production/cron-rsync_support_dbs.sh. This is run from wb-dev.
0 1 * * * /home/tharris/projects/wormbase/wormbase-admin/update/production/cron-rsync_support_dbs.sh
Testing The Site
At this point, all components of a WormBase installation have been installed. You can test your installation by restarting the various server components of WormBase.
Restarting AceDB
# Via xinetd: $ /etc/init.d/xinetd reload (or restart)
# ...or using saceclient % saceclient localhost -port 2005 acedb> password: acedb> shutdown now
Restarting MySQL
# Via mysqladmin... % mysqladmin -uroot -pPASSWORD shutdown # or using init.d $ /etc/init.d/mysql restart
Starting The Webapp
Set up environment:
$ cd /your/checked/out/source $ cp wormbase.env.template wormbase.env // Edit as necessary. $ source wormbase.env
Start the built in development server:
$ script/wormbase_server -p 9999 -r -d // choose a suitable port.
Start up a starman server via plack:
$ cd /your/checked/out/source $ source wormbase.env $ starman -- port 9999 wormbase.psgi // choose a suitable port.
At this point, you should now be able to connect to WormBase at:
http://localhost:YOURPORT/
If you are running your site on one of the ports matching the nginx reverse proxy configuration as described above, you should be able to:
http://subdomain.wormbase.org/
To run your site using the init script:
source wormbase.env ./script/wormbase-init.sh start
Administration
Checking out the Administration module
Fetch the administration code repository (privileged users only, for now)
todd> mkdir -p ~/projects/wormbase todd> cd ~/projects/wormbase todd> hg clone ssh://hg@bitbucket.org/tharris/wormbase-admin/ // Assuming you have already set up your ssh keys at bitbucket
Updating the web application
Updating databases
Backing up your system
Cronjobs
Set up the following cronjobs:
Log Rotation (root cron)
# Rotate WormBase logs 10 1 * * * /home/todd/projects/wormbase/admin/maintenance/rotate_httpd_logs.pl
Or via logrotate:
00 00 * * * /usr/sbin/logrotate -s /var/tmp/logrotate.status /usr/local/wormbase/admin/crons/wormbase-logrotate.conf
The contents of this file look like this:
/usr/local/wormbase/logs/*_log { daily missingok rotate 3650 # keep 10 years worth of logs compress notifempty sharedscripts mail wormbase-logs@tharris.org # when a log file expires, email it to me. postrotate /usr/local/apache2/bin/apachectl graceful endscript }
For nodes running acedb, add the following entry which deals with its massive log files:
# Purge epic ACEDB logs 35 * * * * /home/todd/projects/wormbase/admin/maintenance/purge_acedb_logs.sh
Watch for runaway httpd processes:
- /5 * * * * /usr/local/wormbase/admin/monitoring/monitor_httpd_runaways.sh
The final step is to arrange for Acedb to start automatically and for MySQL to restart if necessary.
Installing MySQL and BLAT monitoring scripts NOT DONE
Run:
$ cp -i /usr/local/wormbase/util/admin/blat_server.initd /etc/rc.d/init.d/blat_server
Then run:
$ crontab -u root -e
to add the following entries to root's crontab:
0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl 0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl
Installing scripts to verify that the servers are running
Two scripts in the WormBase directory can be used to ensure that the mysql and blat servers are running. To install, them:
% sudo cp /usr/localwormbase/util/admin/blat_server.initd \ /etc/rc.d/init.d/blat_server
Place the restart scripts under cron control of a privileged user. These commands will check every hour to see that the servers are running.
% sudo crontab -u root -e
0 * * * * /usr/local/wormbase/util/admin/restart_mysqld.pl 0 * * * * /usr/local/wormbase/util/admin/restart_blat.pl
At the same time, you might also wish to automate the rotatation of logs to prevent them from growing to an unwieldy size. You'll find an appropriate log rotation configuration stanza in util/rotate_wormbase_logs and a log rotate script in /usr/local/wormbase/bin/rotatelogs.pl. You will need both.
# Rotate httpd logs 10 1 * * * /usr/local/wormbase/bin/rotatelogs.pl # Rotate acedb logs 10 1 * * * logrotate /usr/local/wormbase/util/rotate_wormbase_logs
This stanza will check that the acedb server logs do not grow larger than 100 MB.
Blocking robots
It can be useful to block search engines (such as Google) from crawling over one's mirror. To do this, go to /usr/local/wormbase/html, and make a file called "robots.txt" with the following contents:
User-agent: * Disallow: /
Appendices
Appendix 1: Mac OS 10.6 (Snow Leopard) Prequisites
XCode > 3.21
You will need this for building C libraries as well as fixing the broken Perl that shipping with 10.6.
MySQL >= 5.1
MySQL is not expressly required, but you do need it for building DBD::mysql.
bash> cd src bash> curl -O http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.39.tar.gz bash> tar xzf mysql-5.1.41.tar.gz bash> cd mysql-5.1.41 bash> ./configure --prefix=/usr/local/mysql \ --with-extra-charsets=complex \ --enable-thread-safe-client \ --enable-local-infile \ --enable-shared \ --with-plugins=innobase bash> make bash> sudo make install bash> cd /usr/local/mysql bash> sudo ./bin/mysql_install_db --user=mysql bash> sudo chown -R mysql ./var
libpng
bash> curl -O http://softlayer.dl.sourceforge.net/project/libpng/00-libpng-stable/1.2.40/libpng-1.2.40.tar.gz bash> tar xzf libpng-* bash> ccd libpng* bash> ./configure bash> make bash> sudo make install
libgd
bash> curl -O http://www.libgd.org/releases/gd-2.0.35.tar.gz bash> tar xzf gd* bash> cd gd* bash> ./configure bash> make bash> sudo make install
Appendix 2: mod_perl and Apache installation
WormBase no longer runs off apache and mod_perl, although it is entirely possible to do so. Here's how to install these components.
I prefer to build httpd and mod_perl from source; it seems easier than waiting for package repositories to be updated. Besides, I really despise Debian's apache layout. It's horrid.
Installation
Assuming you have already fetched the source into ~/src:
# Build httpd 2.2.11 cd ~/src tar xzf httpd-2.2.11.tar.gz cd httpd-2.2.11 ./configure --enable-mods-shared=all --enable-proxy make sudo make install
# Build mod_perl2-0.4 cd ~/src cd mod_perl-2.0.4 perl -I/usr/local/wormbase/website-classic/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/website-classic/extlib // When prompted for the apxs path, enter: /usr/local/apache2/bin/apxs make sudo make install
Configuration
We typically set up services as individual virtual servers. Here's an example. See "Special Services" below for additional details.
Modify the primary httpd.conf file (/usr/local/apache2/conf/httpd.conf) as follows:
#User daemon User nobody
Set up a virtual host on your port. Adjust PerlSwitches as appropriate for your architecture, and the Listen directive of the primary apaache httpd.conf file as appropriate.
<VirtualHost *:80> Include /usr/local/wormbase/website-classic/conf/httpd.conf PerlOptions +Parent PerlSwitches -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5 \ -Mlib=/usr/local/wormbase/website-classic/extlib/lib/perl5/x86_64-linux-gnu-thread-multi # PerlInterpStart 2 # PerlInterpMax 2 <Directory "/usr/local/wormbase/website-classic/html"> DirectoryIndex index.html # Order deny,allow # Deny from all </Directory> # Redirect requests for the old movable type atom feeds to feedburner RewriteEngine ON # RewriteRule ^/rss/wormbase-live-atom\.xml$ http://feeds.feedburner.com/wormbase [R=301,L] RewriteRule ^/rss/wormbase-live-atom\.xml$ http://feeds.feedburner.com/wormbase [R=301,L] RewriteRule ^/news\.xml$ http://feeds.feedburner.com/wormbase [R=301,L] RewriteRule ^/rss/wormbase-live\.xml$ http://feeds.feedburner.com/wormbase [R=301,L] </VirtualHost>
Enable virtual hosts in httpd.conf by uncommenting the following line:
# Include conf/extra/httpd-hosts.conf
Set up httpd to run under inet.d
cd /etc/init.d sudo ln -s /usr/local/apache2/bin/apachectl httpd
Edit apachectl to make it "LSB" compliant (ie Debian compliant). Add the following after the she-bang:
### BEGIN INIT INFO # Provides: httpd # Required-Start: $all # Required-Stop: $remote_fs $syslog # Default-Start: 3 5 # Default-Stop: 0 1 2 4 6 # Short-Description: Start daemon at boot time # Description: Enable service provided by daemon. ### END INIT INFO
cd /etc/rc3.d sudo ln -s ../init.d/apache2 S99apache2 cd ../rc5.d sudo ln -s ../init.d/apache2 S99apache2
Or via update-rc.d
sudo update-rc.d apache2 start 99 3 5 . stop 0 1 2 4 6 .
PHP
cd ~/src tar xzf php-5.2.4.tar.gz ./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql make sudo make install
AUTHOR
Todd Harris (todd@wormbase.org)