Administration:WormBase Production Environment

From WormBaseWiki
Jump to navigationJump to search

Overview

The WormBase production environment consists of a series of a http servers glued to our webapp, all sitting behind a load-balancing reverse-proxy server (nginx).

Each cluster node runs the lightweight HTTP server starman on port 5000. These nodes are accessible ONLY to the front end proxy.

This document describes the configuration of individual web nodes and the reverse proxy server.


To resolve

  • How is the back end node hosting the user database specified?
  • Where are log paths specified? These need to be consolidated.
  • image caching
  • memcache
  • differences in configuration files.
  • set up starman on beta.wormbase.org


nginx

  • ssl
  • proxy caching
  • serving up of static content
  • memcache


  • to test
  • logging in
  • browser compatibility
  • set up starman on dev
  • set automatic updates of code and restarting of services.



Paper:

  • Unified paper interface
  • Longer term:
  • Paper
  • Overivew
   Laboratory: strain designation
   Fetch all strains for a given lab


http://en.wikipedia.org/wiki/VCard

Logs

All relevant logs can be found at:

ls /usr/local/wormbase/logs
nginx-error.log    // The reverse proxy error log
nginx-access.log   // The reverse proxy access log
nginx-cache.log    // The reverse proxy cache log
catalyst_error.log // The catalyst error log

Reverse Proxy and Load Balancing via nginx

Installation

We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.

1. Install prerequisites

  # Perl Compatabile Regular Expression libaray
  sudo apt-get install libpcre3 libpcre3-dev
  # Fetch and unpack openssel
 wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz
 tar -zxf openssl-0.9.8p.tar.gz

2. Get the nginx cache-purge module

  cd src/
  curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz
  tar xzf ngx_cache_purge-1.2.tar.gz

3. Build and install nginx

  curl -O http://nginx.org/download/nginx-0.8.53.tar.gz
  tar xzf nginx*
  ./configure \
   --prefix=/usr/local/wormbase/services/nginx-0.8.53 \
   --error-log-path=/usr/local/wormbase/logs/nginx-error.log \
   --http-log-path=/usr/local/wormbase/logs/nginx-access.log \
   --with-http_stub_status_module \
   --with-http_ssl_module \
   --with-ipv6 \
   --with-http_realip_module \
    --with-http_addition_module \
    --with-http_image_filter_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_gzip_static_module \
    --with-http_secure_link_module \
    --with-openssl=../openssl-0.9.8p \
   --add-module=../ngx_cache_purge-1.2
   make
   make install
   cd /usr/local/wormbase/services
   ln -s nginx-0.8.53 nginx

4. Symlink the configuration directory

cd /usr/local/wormbase/services/nginx
mv conf conf.original
ln -s /usr/local/wormbase/admin/conf/nginx conf

5. Test the configuration file syntax by:

$ nginx -t

Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching

Load Balancing

nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:

 http://wiki.nginx.org/NginxHttpUpstreamModule

There are a number of other interesting load-balancing modules that might be of use:

 http://wiki.nginx.org/3rdPartyModules

Starting the Server

Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:

$ /etc/init.d/nginx restart

Set nginx to start at server launch

 sudo /usr/sbin/update-rc.d -f nginx defaults

The output will be similar to this:

Adding system startup for /etc/init.d/nginx ...
  /etc/rc0.d/K20nginx -> ../init.d/nginx
  /etc/rc1.d/K20nginx -> ../init.d/nginx
  /etc/rc6.d/K20nginx -> ../init.d/nginx
  /etc/rc2.d/S20nginx -> ../init.d/nginx
  /etc/rc3.d/S20nginx -> ../init.d/nginx
  /etc/rc4.d/S20nginx -> ../init.d/nginx
  /etc/rc5.d/S20nginx -> ../init.d/nginx

Webserver Nodes

PSGI/Plack + Starman

PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).

  http://plackperl.org/

Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.

 https://github.com/miyagawa/Starman

Install Plack:

Catalyst::Controller::Metal
Catalyst::Engine::PSGI
Catalyst::Helper::PSGI
Plack::Test::Adopt::Catalyst

Install Plack:

cpanm Task::Plack
cpanm Starman

Configuration

The PSGI glue resides at:

script/wormbase_psgi.psgi

Starting Starman

starman script/wormbase_psgi.psgi
OR
starman -MFindBin script/wormbase_psgi.psgi


Set up starman to launch automatically

Copy the stub init script from wormbase/conf/starman/starman.init:

 cp /usr/local/wormbase/admin/conf/starman/starman.init

Memached/libmemcached

sudo apt-get install memcached

wget http://download.tangent.org/libmemcached-0.44.tar.gz
tar -zxf libmemcached-0.44.tar.gz
cd libmemcached-0.4
./configure –prefix=/usr
make
make install


MISC:


Crypt:
ssl
sudo apt-get install libssl-dev
 libc6
sudo apt-get install libc6-dev


The Web App

Checking out and building the code

For a new major release:

ssh wb-dev
cd /usr/local/wormbase/website
// Anonymously checkout the code. You will not be able to commit back...
hg clone ssh://hg@bitbucket.org/tharris/wormbase
mv wormbase staging

Build dependencies.

cd staging
mkdir extlib
cd extlib
perl -Mlocal::lib=./
eval $(perl -Mlocal::lib=./)
cd ../
perl Makefile.PL
make installdeps

Update wormbase.env to read "APPNAME=production", then

source wormbase.env

For an updated but minor release:

cd /usr/local/wormbase/website/staging
hg incoming
hg pull -u
perl Makefile.PL
make installdeps    // make sure we are up-to-date


Build the user schema

The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on web1.

ssh web1
mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';
# All nodes currently use the same session database.
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';

Open appropriate ports

nginx is listening on port web1:2011. Open this port via iptables.

  # The new website front-end proxy, accessible to the world                                                          
  $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT

Launch services on the front end machine

# nginx
/etc/init.d/nginx start
# starman
cd /usr/local/wormbase/website/production/bin
./starman-production.sh start

Launch services on back end machines

# starman
cd /usr/local/wormbase/website/production/bin
./starman-production.sh start

Adjust iptables

We need to open port 8000, which should only be to the squid.

sudo emacs /etc/init.d/iptables

Add the following:

  # The new website runs on port 8000. It SHOULD only be accessible by squid                                                              
  $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT
  # ... or only accessible by proxy                                                                                                       
  #  $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT

Then

/etc/init.d/iptables.local restart



EVERYTHING BELOW HERE IS DEPRECATED


FastCGI

Installing fastcgi

curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz
tar xzf mod_fastcgi*
cd mod_fastcgi*
cp Makefile.AP2 Makefile
make top_dir=/usr/local/apache2
sudo make top_dir=/usr/local/apache2 install

If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:

sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c

Add an entry to httpd.conf like this:

 LoadModule fastcgi_module modules/mod_fastcgi.so

 // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file.
 // Edit it to read exactly as above.

Launch the fastcgi server

   // as a socket server in daemon mode
  /usr/local/wormbase/website/script/wormbase_fastcgi.pl \
       -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d

    // as a deamon bound to a specific port
    script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d

Set up the fastcgi server to launch at boot

Symlink the webapp-fastcgi.init script to /etc/init.d

cd /etc/init.d
sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi

Set up symlinks in runlevels:

cd ../rc3.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
cd ../rc5.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi

Add a cron job that keeps FCGI under control

The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).

sudo crontab -e
*/30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \
                `cat /tmp/wormbase.pid` 104857600

FastCGID

cd src
tar xzf mod_fcid*
cd mod_fcgid*
APXS=/usr/local/apache2/bin/apxs ./configure.apxs
make
sudo make install

Apache

Configure Apache to connect to the fastcgi server

Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf

<VirtualHost *:8000>
     #    ServerName beta.wormbase.org                                                                                     
     ErrorLog /usr/local/wormbase/logs/wormbase2.error_log
     TransferLog /usr/local/wormbase/logs/wormbase2.access_log


     # 502 is a Bad Gateway error, and will occur if the backend server is down
     # This allows us to display a friendly static page that says "down for
     # maintenance"
     Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages
     ErrorDocument 502 /_errors/502.html

     # Map dynamic images to the file system 
     # static images are located at img
     Alias /images       /tmp/wormbase/images/
 
  #  <Directory /filesystem/path/to/MyApp/root/static>
  #      allow from all
  #  </Directory>
  #  <Location /myapp/static>
  #      SetHandler default-handler
  #  </Location>

     # Static content served directly by Apache
     DocumentRoot /usr/local/wormbase/website/root
     #     Alias /static /usr/local/wormbase/website-2.0/root



     # Approach 1: Running as a static server (Apache handles spawning of the webapp)       
     # <IfModule fastcgi_module>
     #    FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3                      
     #    Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/
     # </IfModule>
                                   

     # Approach 2: External Process (via mod_fcgi ONLY)
     <IfModule mod_fastcgi.c>
         # This says to connect to the Catalyst fcgi server running on localhost, port 777
         #  FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777
         # Or to use the socket      
         FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock

         # Place the app at root...
         Alias /    /tmp/wormbase.fcgi/
  
         # ...or somewhere else
         Alias /wormbase/ /tmp/wormbase.fcgi/
      </IfModule>

     # fcgid configuration
     #     <IfModule mod_fcgid>
     #         # This should point at your myapp/root
     #          DocumentRoot /usr/local/wormbase/beta.wormbase.org/root
     #         Alias /static /usr/local/wormbase/beta.wormbase.org/root/static
     #         <Location /static>
     #                   SetHandler default-handler
     #          </Location>
     #
     #         Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/
     #         AddType application/x-httpd-php .php
     #         <Location />
     #                   Options ExecCGI
     #                   Order allow,deny
     #                   Allow from all
     #                   AddHandler fcgid-script .pl
     #          </Location>
     #     </IfModule>

   </VirtualHost>

Edit /usr/local/apache2/conf/httpd.conf

Add the appropriate Listen PORT directive.