Administration:WormBase Production Environment
Contents
Overview
The WormBase production environment consists of a series of partially redundant web and database servers, most sitting behind a load-balancing reverse-proxy server running nginx.
Reverse proxy node
Web server nodes
Each web cluster node runs the lightweight HTTP server starman listening on port 5000. This http server is glued via PSGI/Plack/Starman to our Catalyst web application.
Currently, each node is -- with the exception of GBrowse -- almost entirely independent, with its own AceDB and MySQL databases.
Web cluster nodes are accessible ONLY to the front end proxy.
Data mining nodes
Social feature node
To resolve
- How is the back end node hosting the user database specified?
- Where are log paths specified? These need to be consolidated.
- image caching
- memcache
- differences in configuration files.
- set up starman on beta.wormbase.org
nginx
- ssl
- proxy caching
- serving up of static content
- memcache
- to test
- logging in
- browser compatibility
- set up starman on dev
- set automatic updates of code and restarting of services.
Paper:
- Unified paper interface
- Longer term:
- Paper
- Overivew
Laboratory: strain designation Fetch all strains for a given lab
http://en.wikipedia.org/wiki/VCard
Webserver nodes as described in the Installing WormBase document.
Logs
All relevant logs can be found at:
ls /usr/local/wormbase/logs nginx-error.log // The reverse proxy error log nginx-access.log // The reverse proxy access log nginx-cache.log // The reverse proxy cache log catalyst_error.log // The catalyst error log
Reverse Proxy Node
Installing nginx
We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.
1. Install prerequisites
# Perl Compatabile Regular Expression libaray sudo apt-get install libpcre3 libpcre3-dev
# Fetch and unpack openssel wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz tar -zxf openssl-0.9.8p.tar.gz
2. Get the nginx cache-purge module
cd src/ curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz tar xzf ngx_cache_purge-1.2.tar.gz
3. Build and install nginx
curl -O http://nginx.org/download/nginx-0.8.53.tar.gz tar xzf nginx* ./configure \ --prefix=/usr/local/wormbase/services/nginx-0.8.53 \ --error-log-path=/usr/local/wormbase/logs/nginx-error.log \ --http-log-path=/usr/local/wormbase/logs/nginx-access.log \ --with-http_stub_status_module \ --with-http_ssl_module \ --with-ipv6 \ --with-http_realip_module \ --with-http_addition_module \ --with-http_image_filter_module \ --with-http_sub_module \ --with-http_dav_module \ --with-http_flv_module \ --with-http_gzip_static_module \ --with-http_secure_link_module \ --with-openssl=../openssl-0.9.8p \ --add-module=../ngx_cache_purge-1.2 make make install cd /usr/local/wormbase/services ln -s nginx-0.8.53 nginx
4. Symlink the configuration directory
cd /usr/local/wormbase/services/nginx mv conf conf.original ln -s /usr/local/wormbase/admin/conf/nginx conf
5. Test the configuration file syntax by:
$ nginx -t
Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching
Load Balancing
nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:
http://wiki.nginx.org/NginxHttpUpstreamModule
There are a number of other interesting load-balancing modules that might be of use:
http://wiki.nginx.org/3rdPartyModules
Starting the Server
Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:
$ /etc/init.d/nginx restart
Set nginx to start at server launch
sudo /usr/sbin/update-rc.d -f nginx defaults
The output will be similar to this:
Adding system startup for /etc/init.d/nginx ... /etc/rc0.d/K20nginx -> ../init.d/nginx /etc/rc1.d/K20nginx -> ../init.d/nginx /etc/rc6.d/K20nginx -> ../init.d/nginx /etc/rc2.d/S20nginx -> ../init.d/nginx /etc/rc3.d/S20nginx -> ../init.d/nginx /etc/rc4.d/S20nginx -> ../init.d/nginx /etc/rc5.d/S20nginx -> ../init.d/nginx
Webserver Nodes
PSGI/Plack + Starman
PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).
http://plackperl.org/
Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.
https://github.com/miyagawa/Starman
Install Plack:
- Catalyst::Controller::Metal
- Catalyst::Engine::PSGI
- Catalyst::Helper::PSGI
- Plack::Test::Adopt::Catalyst
Install Plack:
- cpanm Task::Plack
- cpanm Starman
Configuration
The PSGI glue resides at:
script/wormbase_psgi.psgi
Starting Starman
starman script/wormbase_psgi.psgi OR starman -MFindBin script/wormbase_psgi.psgi
Set up starman to launch automatically
Copy the stub init script from wormbase/conf/starman/starman.init:
cp /usr/local/wormbase/admin/conf/starman/starman.init
Memached/libmemcached
sudo apt-get install memcached wget http://download.tangent.org/libmemcached-0.44.tar.gz tar -zxf libmemcached-0.44.tar.gz cd libmemcached-0.4 ./configure –prefix=/usr make make install
MISC:
Crypt: ssl sudo apt-get install libssl-dev
libc6 sudo apt-get install libc6-dev
The Web App
Checking out and building the code
For a new major release:
ssh wb-dev cd /usr/local/wormbase/website // Anonymously checkout the code. You will not be able to commit back... hg clone ssh://hg@bitbucket.org/tharris/wormbase mv wormbase staging
Build dependencies.
cd staging mkdir extlib cd extlib perl -Mlocal::lib=./ eval $(perl -Mlocal::lib=./) cd ../ perl Makefile.PL make installdeps
Update wormbase.env to read "APPNAME=production", then
source wormbase.env
For an updated but minor release:
cd /usr/local/wormbase/website/staging hg incoming hg pull -u perl Makefile.PL make installdeps // make sure we are up-to-date
Build the user schema
The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on web1.
ssh web1 mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';
# All nodes currently use the same session database. mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';
Open appropriate ports
nginx is listening on port web1:2011. Open this port via iptables.
# The new website front-end proxy, accessible to the world $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT
Launch services on the front end machine
# nginx /etc/init.d/nginx start
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Launch services on back end machines
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Adjust iptables
We need to open port 8000, which should only be to the squid.
sudo emacs /etc/init.d/iptables
Add the following:
# The new website runs on port 8000. It SHOULD only be accessible by squid $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT # ... or only accessible by proxy # $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT
Then
/etc/init.d/iptables.local restart
EVERYTHING BELOW HERE IS DEPRECATED
FastCGI, FCGI, Apache, and mod_perl
Originally, WormBase ran under apache + mod_perl.
We also experimented with fcgi and fcgid +apache.
Installing fastcgi
curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz tar xzf mod_fastcgi* cd mod_fastcgi* cp Makefile.AP2 Makefile make top_dir=/usr/local/apache2 sudo make top_dir=/usr/local/apache2 install
If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:
sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c
Add an entry to httpd.conf like this:
LoadModule fastcgi_module modules/mod_fastcgi.so // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file. // Edit it to read exactly as above.
Launch the fastcgi server
// as a socket server in daemon mode /usr/local/wormbase/website/script/wormbase_fastcgi.pl \ -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d // as a deamon bound to a specific port script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d
Set up the fastcgi server to launch at boot
Symlink the webapp-fastcgi.init script to /etc/init.d
cd /etc/init.d sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi
Set up symlinks in runlevels:
cd ../rc3.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi cd ../rc5.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
Add a cron job that keeps FCGI under control
The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).
sudo crontab -e */30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \ `cat /tmp/wormbase.pid` 104857600
mod_fcgid
mod_fcgid is an alternative to fcgi
cd src/ wget http://www.carfab.com/apachesoftware/httpd/mod_fcgid/mod_fcgid-2.3.5.tar.gz tar xzf mod_fcgid-2.3.5.tar.gz cd mod_fcgid-2.3.5 APXS=/usr/local/apache2/bin/apxs ./configure.apxs make sudo make install
Apache
Configure Apache to connect to the fastcgi server
Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf
<VirtualHost *:8000> # ServerName beta.wormbase.org ErrorLog /usr/local/wormbase/logs/wormbase2.error_log TransferLog /usr/local/wormbase/logs/wormbase2.access_log # 502 is a Bad Gateway error, and will occur if the backend server is down # This allows us to display a friendly static page that says "down for # maintenance" Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages ErrorDocument 502 /_errors/502.html # Map dynamic images to the file system # static images are located at img Alias /images /tmp/wormbase/images/ # <Directory /filesystem/path/to/MyApp/root/static> # allow from all # </Directory> # <Location /myapp/static> # SetHandler default-handler # </Location> # Static content served directly by Apache DocumentRoot /usr/local/wormbase/website/root # Alias /static /usr/local/wormbase/website-2.0/root # Approach 1: Running as a static server (Apache handles spawning of the webapp) # <IfModule fastcgi_module> # FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3 # Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/ # </IfModule> # Approach 2: External Process (via mod_fcgi ONLY) <IfModule mod_fastcgi.c> # This says to connect to the Catalyst fcgi server running on localhost, port 777 # FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777 # Or to use the socket FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock # Place the app at root... Alias / /tmp/wormbase.fcgi/ # ...or somewhere else Alias /wormbase/ /tmp/wormbase.fcgi/ </IfModule> # fcgid configuration # <IfModule mod_fcgid> # # This should point at your myapp/root # DocumentRoot /usr/local/wormbase/beta.wormbase.org/root # Alias /static /usr/local/wormbase/beta.wormbase.org/root/static # <Location /static> # SetHandler default-handler # </Location> # # Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/ # AddType application/x-httpd-php .php # <Location /> # Options ExecCGI # Order allow,deny # Allow from all # AddHandler fcgid-script .pl # </Location> # </IfModule> </VirtualHost>
Edit /usr/local/apache2/conf/httpd.conf
Add the appropriate Listen PORT directive.
Servers
Configuring iptables
We don't want end users to directly access back end machines. Instead, we want to force them to pass through the caching proxy. For now, we will only allow the reverse proxy access to the server on port 80.
We only want the front end proxy (currently at CSHL) to be able to access the machine directly on port 80. See conf/iptables.local
in the wormbase-admin module for details.
The Wiki, Blog, and Forums Server
The WormBase Blog, the WormBase Wiki, and the Worm Community Forums all rely on third party software. To make it easy to update this software, each of these components is maintained as a separate virtual host running on its own port.
Iptables
The blog, wiki, and forums are all virtual hosts. However, we don't want the server to respond on port 80 for requests to the machine's IP.
The WormBase Blog
The WormBase blog is a subdomain of wormbase.org: blog.wormbase.org. If it's moved, the DNS entry *must* be updated!
Host/Port : wb-acedb1.oicr.on.ca:80 Alias: blog.wormbase.org MySQL database : wormbase_wordpress_blog Document root : /usr/local/wormbase/website-blog/current Logs : /usr/local/wormbase/blogs-access_log, /usr/local/wormbase/logs/blogs-error_log
Blog files are stored in /usr/local/wormbase/website-blog/current:
current -> wordpress-2.92
Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf
<VirtualHost *:80> ServerName blog.wormbase.org DocumentRoot /usr/local/wormbase/website-blog <Directory "/usr/local/wormbase/website-blog"> DirectoryIndex index.php index.html AddType application/x-httpd-php .php Order Deny,Allow Allow from all </Directory> LogFormat "%h %l %u %t \"%r\" %s %b" common LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous ErrorLog /usr/local/wormbase/logs/blog-error_log CustomLog /usr/local/wormbase/logs/blog-access_log combined_format </VirtualHost>
NOTE: when upgrading, be sure to copy the wp-config.php file and entire wp-content/ directory.
The WormBase Wiki
The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!
Host/Port : wb-acedb1.oicr.on.ca:80 Alias: wiki.wormbase.org MySQL database : wormbase_wiki Document root : /usr/local/wormbase/website-wiki/current Logs : /usr/local/wormbase/wiki-access_log, /usr/local/wormbase/logs/wiki-error_log
Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf
<VirtualHost *:80> ServerName wiki.wormbase.org # Current is a symlink to the current installation. DocumentRoot /usr/local/wormbase/website-wiki/current <Directory "/usr/local/wormbase/website-wiki/current"> DirectoryIndex index.php index.html AddType application/x-httpd-php .php Order Deny,Allow Allow from all </Directory> LogFormat "%h %l %u %t \"%r\" %s %b" common LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous ErrorLog /usr/local/wormbase/logs/wiki-error_log CustomLog /usr/local/wormbase/logs/wiki-access_log combined_format </VirtualHost>
The Worm Community Forums
The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!
Host/Port : wb-acedb1.oicr.on.ca:80 Alias: forums.wormbase.org MySQL database : wormbaseforumssmf Document root : /usr/local/wormbase/website-forums Logs : /usr/local/wormbase/forums-access_log, /usr/local/wormbase/logs/forums-error_log
Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf
<VirtualHost *:80> ServerName forums.wormbase.org # Current is a symlink to the current version of SMF DocumentRoot /usr/local/wormbase/website-forums/current <Directory "/usr/local/wormbase/website-forums/current"> DirectoryIndex index.php index.html AddType application/x-httpd-php .php Order Deny,Allow Allow from all </Directory> LogFormat "%h %l %u %t \"%r\" %s %b" common LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous ErrorLog /usr/local/wormbase/logs/forums-error_log CustomLog /usr/local/wormbase/logs/forums-access_log combined_format </VirtualHost>
Add "Listen 8081" to the primary httpd.conf file.
Note: If the forum is moved, it is also necessary to update Settings.php and the paths to the Sources and Themes directories in the forum Administration Panel > Configuration > Server Settings.
The Datamining and BLAST server
The data mining and BLAST/BLAT server replaces the old aceserver. Because it handles requests for the AQL and WB pages, it runs the full website and has all mysql and acedb databases.