Difference between revisions of "Administration:WormBase Production Environment"
Line 7: | Line 7: | ||
This document describes the configuration of individual web nodes and the reverse proxy server. | This document describes the configuration of individual web nodes and the reverse proxy server. | ||
+ | |||
+ | |||
+ | |||
+ | TODO: Merge this document with the general website installation documents | ||
= To resolve = | = To resolve = |
Revision as of 01:36, 24 December 2010
Contents
Overview
The WormBase production environment consists of a series of a http servers glued to our webapp, all sitting behind a load-balancing reverse-proxy server (nginx).
Each cluster node runs the lightweight HTTP server starman on port 5000. These nodes are accessible ONLY to the front end proxy.
This document describes the configuration of individual web nodes and the reverse proxy server.
TODO: Merge this document with the general website installation documents
To resolve
- How is the back end node hosting the user database specified?
- Where are log paths specified? These need to be consolidated.
- image caching
- memcache
- differences in configuration files.
- set up starman on beta.wormbase.org
nginx
- ssl
- proxy caching
- serving up of static content
- memcache
- to test
- logging in
- browser compatibility
- set up starman on dev
- set automatic updates of code and restarting of services.
Paper:
- Unified paper interface
- Longer term:
- Paper
- Overivew
Laboratory: strain designation Fetch all strains for a given lab
http://en.wikipedia.org/wiki/VCard
Logs
All relevant logs can be found at:
ls /usr/local/wormbase/logs nginx-error.log // The reverse proxy error log nginx-access.log // The reverse proxy access log nginx-cache.log // The reverse proxy cache log catalyst_error.log // The catalyst error log
Reverse Proxy and Load Balancing via nginx
Installation
We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.
1. Install prerequisites
# Perl Compatabile Regular Expression libaray sudo apt-get install libpcre3 libpcre3-dev
# Fetch and unpack openssel wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz tar -zxf openssl-0.9.8p.tar.gz
2. Get the nginx cache-purge module
cd src/ curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz tar xzf ngx_cache_purge-1.2.tar.gz
3. Build and install nginx
curl -O http://nginx.org/download/nginx-0.8.53.tar.gz tar xzf nginx* ./configure \ --prefix=/usr/local/wormbase/services/nginx-0.8.53 \ --error-log-path=/usr/local/wormbase/logs/nginx-error.log \ --http-log-path=/usr/local/wormbase/logs/nginx-access.log \ --with-http_stub_status_module \ --with-http_ssl_module \ --with-ipv6 \ --with-http_realip_module \ --with-http_addition_module \ --with-http_image_filter_module \ --with-http_sub_module \ --with-http_dav_module \ --with-http_flv_module \ --with-http_gzip_static_module \ --with-http_secure_link_module \ --with-openssl=../openssl-0.9.8p \ --add-module=../ngx_cache_purge-1.2 make make install cd /usr/local/wormbase/services ln -s nginx-0.8.53 nginx
4. Symlink the configuration directory
cd /usr/local/wormbase/services/nginx mv conf conf.original ln -s /usr/local/wormbase/admin/conf/nginx conf
5. Test the configuration file syntax by:
$ nginx -t
Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching
Load Balancing
nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:
http://wiki.nginx.org/NginxHttpUpstreamModule
There are a number of other interesting load-balancing modules that might be of use:
http://wiki.nginx.org/3rdPartyModules
Starting the Server
Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:
$ /etc/init.d/nginx restart
Set nginx to start at server launch
sudo /usr/sbin/update-rc.d -f nginx defaults
The output will be similar to this:
Adding system startup for /etc/init.d/nginx ... /etc/rc0.d/K20nginx -> ../init.d/nginx /etc/rc1.d/K20nginx -> ../init.d/nginx /etc/rc6.d/K20nginx -> ../init.d/nginx /etc/rc2.d/S20nginx -> ../init.d/nginx /etc/rc3.d/S20nginx -> ../init.d/nginx /etc/rc4.d/S20nginx -> ../init.d/nginx /etc/rc5.d/S20nginx -> ../init.d/nginx
Webserver Nodes
PSGI/Plack + Starman
PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).
http://plackperl.org/
Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.
https://github.com/miyagawa/Starman
Install Plack:
- Catalyst::Controller::Metal
- Catalyst::Engine::PSGI
- Catalyst::Helper::PSGI
- Plack::Test::Adopt::Catalyst
Install Plack:
- cpanm Task::Plack
- cpanm Starman
Configuration
The PSGI glue resides at:
script/wormbase_psgi.psgi
Starting Starman
starman script/wormbase_psgi.psgi OR starman -MFindBin script/wormbase_psgi.psgi
Set up starman to launch automatically
Copy the stub init script from wormbase/conf/starman/starman.init:
cp /usr/local/wormbase/admin/conf/starman/starman.init
Memached/libmemcached
sudo apt-get install memcached wget http://download.tangent.org/libmemcached-0.44.tar.gz tar -zxf libmemcached-0.44.tar.gz cd libmemcached-0.4 ./configure –prefix=/usr make make install
MISC:
Crypt: ssl sudo apt-get install libssl-dev
libc6 sudo apt-get install libc6-dev
The Web App
Checking out and building the code
For a new major release:
ssh wb-dev cd /usr/local/wormbase/website // Anonymously checkout the code. You will not be able to commit back... hg clone ssh://hg@bitbucket.org/tharris/wormbase mv wormbase staging
Build dependencies.
cd staging mkdir extlib cd extlib perl -Mlocal::lib=./ eval $(perl -Mlocal::lib=./) cd ../ perl Makefile.PL make installdeps
Update wormbase.env to read "APPNAME=production", then
source wormbase.env
For an updated but minor release:
cd /usr/local/wormbase/website/staging hg incoming hg pull -u perl Makefile.PL make installdeps // make sure we are up-to-date
Build the user schema
The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on web1.
ssh web1 mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';
# All nodes currently use the same session database. mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';
Open appropriate ports
nginx is listening on port web1:2011. Open this port via iptables.
# The new website front-end proxy, accessible to the world $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT
Launch services on the front end machine
# nginx /etc/init.d/nginx start
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Launch services on back end machines
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Adjust iptables
We need to open port 8000, which should only be to the squid.
sudo emacs /etc/init.d/iptables
Add the following:
# The new website runs on port 8000. It SHOULD only be accessible by squid $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT # ... or only accessible by proxy # $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT
Then
/etc/init.d/iptables.local restart
EVERYTHING BELOW HERE IS DEPRECATED
FastCGI
Installing fastcgi
curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz tar xzf mod_fastcgi* cd mod_fastcgi* cp Makefile.AP2 Makefile make top_dir=/usr/local/apache2 sudo make top_dir=/usr/local/apache2 install
If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:
sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c
Add an entry to httpd.conf like this:
LoadModule fastcgi_module modules/mod_fastcgi.so // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file. // Edit it to read exactly as above.
Launch the fastcgi server
// as a socket server in daemon mode /usr/local/wormbase/website/script/wormbase_fastcgi.pl \ -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d // as a deamon bound to a specific port script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d
Set up the fastcgi server to launch at boot
Symlink the webapp-fastcgi.init script to /etc/init.d
cd /etc/init.d sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi
Set up symlinks in runlevels:
cd ../rc3.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi cd ../rc5.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
Add a cron job that keeps FCGI under control
The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).
sudo crontab -e */30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \ `cat /tmp/wormbase.pid` 104857600
FastCGID
cd src tar xzf mod_fcid* cd mod_fcgid* APXS=/usr/local/apache2/bin/apxs ./configure.apxs make sudo make install
Apache
Configure Apache to connect to the fastcgi server
Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf
<VirtualHost *:8000> # ServerName beta.wormbase.org ErrorLog /usr/local/wormbase/logs/wormbase2.error_log TransferLog /usr/local/wormbase/logs/wormbase2.access_log # 502 is a Bad Gateway error, and will occur if the backend server is down # This allows us to display a friendly static page that says "down for # maintenance" Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages ErrorDocument 502 /_errors/502.html # Map dynamic images to the file system # static images are located at img Alias /images /tmp/wormbase/images/ # <Directory /filesystem/path/to/MyApp/root/static> # allow from all # </Directory> # <Location /myapp/static> # SetHandler default-handler # </Location> # Static content served directly by Apache DocumentRoot /usr/local/wormbase/website/root # Alias /static /usr/local/wormbase/website-2.0/root # Approach 1: Running as a static server (Apache handles spawning of the webapp) # <IfModule fastcgi_module> # FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3 # Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/ # </IfModule> # Approach 2: External Process (via mod_fcgi ONLY) <IfModule mod_fastcgi.c> # This says to connect to the Catalyst fcgi server running on localhost, port 777 # FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777 # Or to use the socket FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock # Place the app at root... Alias / /tmp/wormbase.fcgi/ # ...or somewhere else Alias /wormbase/ /tmp/wormbase.fcgi/ </IfModule> # fcgid configuration # <IfModule mod_fcgid> # # This should point at your myapp/root # DocumentRoot /usr/local/wormbase/beta.wormbase.org/root # Alias /static /usr/local/wormbase/beta.wormbase.org/root/static # <Location /static> # SetHandler default-handler # </Location> # # Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/ # AddType application/x-httpd-php .php # <Location /> # Options ExecCGI # Order allow,deny # Allow from all # AddHandler fcgid-script .pl # </Location> # </IfModule> </VirtualHost>
Edit /usr/local/apache2/conf/httpd.conf
Add the appropriate Listen PORT directive.