Administration:WormBase Production Environment
Contents
Overview
The WormBase production environment consists of a series of partially redundant web and database servers, most sitting behind a load-balancing reverse-proxy server running nginx.
Reverse proxy node
Web server nodes
- Each web cluster node runs the lightweight HTTP server starman listening on port 5000. This http server is glued via PSGI/Plack/Starman to our Catalyst web application.
- Currently, each node is -- with the exception of GBrowse -- almost entirely independent, with its own AceDB and MySQL databases.
- Web cluster nodes are accessible ONLY to the front end proxy.
Data mining nodes
Social feature node
To resolve
- How is the back end node hosting the user database specified?
- Where are log paths specified? These need to be consolidated.
- image caching
- memcache
- differences in configuration files.
- set up starman on beta.wormbase.org
nginx
- ssl
- proxy caching
- serving up of static content
- memcache
- to test
- logging in
- browser compatibility
- set up starman on dev
- set automatic updates of code and restarting of services.
Paper:
- Unified paper interface
- Longer term:
- Paper
- Overivew
Laboratory: strain designation Fetch all strains for a given lab
http://en.wikipedia.org/wiki/VCard
Webserver nodes as described in the Installing WormBase document.
Reverse Proxy Node
Installing nginx
We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.
1. Install prerequisites
# Perl Compatabile Regular Expression libaray sudo apt-get install libpcre3 libpcre3-dev
# Fetch and unpack openssel wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz tar -zxf openssl-0.9.8p.tar.gz
2. Get the nginx cache-purge module
cd src/ curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz tar xzf ngx_cache_purge-1.2.tar.gz
3. Build and install nginx
curl -O http://nginx.org/download/nginx-0.8.53.tar.gz tar xzf nginx* ./configure \ --prefix=/usr/local/wormbase/services/nginx-0.8.53 \ --error-log-path=/usr/local/wormbase/logs/nginx-error.log \ --http-log-path=/usr/local/wormbase/logs/nginx-access.log \ --with-http_stub_status_module \ --with-http_ssl_module \ --with-ipv6 \ --with-http_realip_module \ --with-http_addition_module \ --with-http_image_filter_module \ --with-http_sub_module \ --with-http_dav_module \ --with-http_flv_module \ --with-http_gzip_static_module \ --with-http_secure_link_module \ --with-openssl=../openssl-0.9.8p \ --add-module=../ngx_cache_purge-1.2 make make install cd /usr/local/wormbase/services ln -s nginx-0.8.53 nginx
4. Symlink the configuration directory
cd /usr/local/wormbase/services/nginx mv conf conf.original ln -s /usr/local/wormbase/admin/conf/nginx conf
5. Test the configuration file syntax by:
$ nginx -t
Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching
Load Balancing
nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:
http://wiki.nginx.org/NginxHttpUpstreamModule
There are a number of other interesting load-balancing modules that might be of use:
http://wiki.nginx.org/3rdPartyModules
Starting the Server
Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:
$ /etc/init.d/nginx restart
Set nginx to start at server launch
sudo /usr/sbin/update-rc.d -f nginx defaults
The output will be similar to this:
Adding system startup for /etc/init.d/nginx ... /etc/rc0.d/K20nginx -> ../init.d/nginx /etc/rc1.d/K20nginx -> ../init.d/nginx /etc/rc6.d/K20nginx -> ../init.d/nginx /etc/rc2.d/S20nginx -> ../init.d/nginx /etc/rc3.d/S20nginx -> ../init.d/nginx /etc/rc4.d/S20nginx -> ../init.d/nginx /etc/rc5.d/S20nginx -> ../init.d/nginx
Launch services on the front end machine
# nginx /etc/init.d/nginx start
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Build the user preferences database
The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on the same server providing the reverse proxy.
mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';
# All nodes currently use the same session database. mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';
Webserver Nodes
HTTP server: PSGI/Plack + Starman
PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).
http://plackperl.org/
Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.
https://github.com/miyagawa/Starman
Install Plack and Starman
- Catalyst::Controller::Metal
- Catalyst::Engine::PSGI
- Catalyst::Helper::PSGI
- Plack::Test::Adopt::Catalyst
- cpanm Task::Plack
- cpanm Starman
Configuration
The PSGI glue resides at:
script/wormbase_psgi.psgi
Starting Starman
starman script/wormbase_psgi.psgi OR starman -MFindBin script/wormbase_psgi.psgi
Set up starman to launch automatically
Copy the stub init script from wormbase/conf/starman/starman.init:
cp /usr/local/wormbase/admin/conf/starman/starman.init
Memached/libmemcached
sudo apt-get install memcached wget http://download.tangent.org/libmemcached-0.44.tar.gz tar -zxf libmemcached-0.44.tar.gz cd libmemcached-0.4 ./configure –prefix=/usr make make install
MISC:
Crypt: ssl sudo apt-get install libssl-dev
libc6 sudo apt-get install libc6-dev
Install the Webapp
The web app and all Perl libraries will be installed automatically by the deploy_wormbase_webapp.sh script.
/usr/local/wormbase/website/production -> WSXXXX-YYYY.MM.DD-X.XX-XXXX /usr/local/wormbase/website/WSXXX-YYYY.MM.DD-X.XX-XXXX
For details on installation of the web app itself, see the Install The Webapp section of the main Installing WormBase guide.
Open appropriate ports
nginx is listening on port web1:2011. Open this port via iptables.
# The new website front-end proxy, accessible to the world $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT
Launch services on back end machines
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Data mining nodes
The data mining and BLAST/BLAT server replaces the old aceserver. Because it handles requests for the AQL and WB pages, it runs the full website and has all mysql and acedb databases.
Adjust iptables
We need to open port 8000, which should only be to the squid.
sudo emacs /etc/init.d/iptables
Add the following:
# The new website runs on port 8000. It SHOULD only be accessible by squid $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT # ... or only accessible by proxy # $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT
Then
/etc/init.d/iptables.local restart
Servers
Configuring iptables
We don't want end users to directly access back end machines. Instead, we want to force them to pass through the caching proxy. For now, we will only allow the reverse proxy access to the server on port 80.
We only want the front end proxy (currently at CSHL) to be able to access the machine directly on port 80. See conf/iptables.local
in the wormbase-admin module for details.
Logs
All relevant logs can be found at:
ls /usr/local/wormbase/logs nginx-error.log // The reverse proxy error log nginx-access.log // The reverse proxy access log nginx-cache.log // The reverse proxy cache log catalyst_error.log // The catalyst error log
EVERYTHING BELOW HERE IS DEPRECATED
FastCGI, FCGI, Apache, and mod_perl
Originally, WormBase ran under apache + mod_perl.
We also experimented with fcgi and fcgid +apache.
Installing fastcgi
curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz tar xzf mod_fastcgi* cd mod_fastcgi* cp Makefile.AP2 Makefile make top_dir=/usr/local/apache2 sudo make top_dir=/usr/local/apache2 install
If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:
sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c
Add an entry to httpd.conf like this:
LoadModule fastcgi_module modules/mod_fastcgi.so // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file. // Edit it to read exactly as above.
Launch the fastcgi server
// as a socket server in daemon mode /usr/local/wormbase/website/script/wormbase_fastcgi.pl \ -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d // as a deamon bound to a specific port script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d
Set up the fastcgi server to launch at boot
Symlink the webapp-fastcgi.init script to /etc/init.d
cd /etc/init.d sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi
Set up symlinks in runlevels:
cd ../rc3.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi cd ../rc5.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
Add a cron job that keeps FCGI under control
The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).
sudo crontab -e */30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \ `cat /tmp/wormbase.pid` 104857600
mod_fcgid
mod_fcgid is an alternative to fcgi
cd src/ wget http://www.carfab.com/apachesoftware/httpd/mod_fcgid/mod_fcgid-2.3.5.tar.gz tar xzf mod_fcgid-2.3.5.tar.gz cd mod_fcgid-2.3.5 APXS=/usr/local/apache2/bin/apxs ./configure.apxs make sudo make install
Apache
Configure Apache to connect to the fastcgi server
Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf
<VirtualHost *:8000> # ServerName beta.wormbase.org ErrorLog /usr/local/wormbase/logs/wormbase2.error_log TransferLog /usr/local/wormbase/logs/wormbase2.access_log # 502 is a Bad Gateway error, and will occur if the backend server is down # This allows us to display a friendly static page that says "down for # maintenance" Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages ErrorDocument 502 /_errors/502.html # Map dynamic images to the file system # static images are located at img Alias /images /tmp/wormbase/images/ # <Directory /filesystem/path/to/MyApp/root/static> # allow from all # </Directory> # <Location /myapp/static> # SetHandler default-handler # </Location> # Static content served directly by Apache DocumentRoot /usr/local/wormbase/website/root # Alias /static /usr/local/wormbase/website-2.0/root # Approach 1: Running as a static server (Apache handles spawning of the webapp) # <IfModule fastcgi_module> # FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3 # Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/ # </IfModule> # Approach 2: External Process (via mod_fcgi ONLY) <IfModule mod_fastcgi.c> # This says to connect to the Catalyst fcgi server running on localhost, port 777 # FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777 # Or to use the socket FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock # Place the app at root... Alias / /tmp/wormbase.fcgi/ # ...or somewhere else Alias /wormbase/ /tmp/wormbase.fcgi/ </IfModule> # fcgid configuration # <IfModule mod_fcgid> # # This should point at your myapp/root # DocumentRoot /usr/local/wormbase/beta.wormbase.org/root # Alias /static /usr/local/wormbase/beta.wormbase.org/root/static # <Location /static> # SetHandler default-handler # </Location> # # Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/ # AddType application/x-httpd-php .php # <Location /> # Options ExecCGI # Order allow,deny # Allow from all # AddHandler fcgid-script .pl # </Location> # </IfModule> </VirtualHost>
Edit /usr/local/apache2/conf/httpd.conf
Add the appropriate Listen PORT directive.