Administration:WormBase Production Environment

Overview

The WormBase production environment consists of a series of a http servers glued to our webapp, all sitting behind a load-balancing reverse-proxy server (nginx).

This document describes the configuration of individual web nodes and the reverse proxy server.

Reverse Proxy and Load Balancing via nginx

Installation

We'll place nginx entirely within the wormbase root directory.

1. Install prerequisites

  # Perl Compatabile Regular Expression libaray
  sudo apt-get install libpcre3 libpcre3-dev

  # Fetch and unpack openssel
 wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz
 tar -zxf openssl-0.9.8p.tar.gz

2. Get the nginx cache-purge module

  cd src/
  curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz
  tar xzf ngx_cache_purge-1.2.tar.gz

3. Build and install nginx

  curl -O http://nginx.org/download/nginx-0.8.53.tar.gz
  tar xzf nginx*
  ./configure \
   --prefix=/usr/local/wormbase/nginx \
   --error-log-path=/usr/local/wormbase/logs/nginx-error.log \
   --http-log-path=/usr/local/wormbase/logs/nginx-access.log \
   --with-http_stub_status_module \
   --with-http_ssl_module \
   --with-ipv6 \
   --with-http_realip_module \
    --with-http_addition_module \
    --with-http_image_filter_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_gzip_static_module \
    --with-http_secure_link_module \
    --with-openssl=../openssl-0.9.8p \
   --add-module=../ngx_cache_purge-1.2
   make
   make install

Get rid of the original configuration file and symlink to that in our source code repository:

cd /usr/local/wormbase/nginx
mv conf conf.original
ln -s /usr/local/wormbase/admin/conf/nginx conf

Configuration

Configuration files are stored in the wormbase-admin source code repository, under conf/nginx and symlinked as:

 > cd /usr/local/wormbase/nginx ; ls conf
 conf -> /home/tharris/projects/wormbase/wormbase-admin/conf/nginx

  emacs /usr/local/wormbase/nginx/conf/nginx.conf 
  upstream backend  {
     server 123.123.123.123;
     server 123.123.123.124;   # or hostname
     # server george.constantshift.com weight=3  # eg; weighting
  }

  server {
      server_name beta.wormbase.org;
      location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass  http://backend;

       }
  }

Test the configuration file syntax by:

$ nginx -t

Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching

Load Balancing

nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:

 http://wiki.nginx.org/NginxHttpUpstreamModule

There are a number of other interesting load-balancing modules that might be of use:

 http://wiki.nginx.org/3rdPartyModules

Starting the Server

Copy wormbase-admin/init/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:

$ /etc/init.d/nginx restart

Set nginx to start at server launch

 sudo /usr/sbin/update-rc.d -f nginx defaults

The output will be similar to this:

Adding system startup for /etc/init.d/nginx ...
  /etc/rc0.d/K20nginx -> ../init.d/nginx
  /etc/rc1.d/K20nginx -> ../init.d/nginx
  /etc/rc6.d/K20nginx -> ../init.d/nginx
  /etc/rc2.d/S20nginx -> ../init.d/nginx
  /etc/rc3.d/S20nginx -> ../init.d/nginx
  /etc/rc4.d/S20nginx -> ../init.d/nginx
  /etc/rc5.d/S20nginx -> ../init.d/nginx

Cluster Nodes

memcached

PSGI/Plack + Starman

PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).

  http://plackperl.org/

Starman is a high performance pre-forking Perl PSGI server:

 https://github.com/miyagawa/Starman

Install Plack:

Catalyst::Controller::Metal

Catalyst::Engine::PSGI

Catalyst::Helper::PSGI

Plack::Test::Adopt::Catalyst

Instal CPANMinus

App::cpanminus

Install Plack:

cpanm Task::Plack

cpanm Starman

Configuration

script/wormbase_psgi.psgi

Starting Starman

starman script/wormbase_psgi.psgi
OR
starman -MFindBin script/wormbase_psgi.psgi

FastCGI

Installing fastcgi

curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz
tar xzf mod_fastcgi*
cd mod_fastcgi*
cp Makefile.AP2 Makefile
make top_dir=/usr/local/apache2
sudo make top_dir=/usr/local/apache2 install

If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:

sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c

Add an entry to httpd.conf like this:

 LoadModule fastcgi_module modules/mod_fastcgi.so

 // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file.
 // Edit it to read exactly as above.

Launch the fastcgi server

   // as a socket server in daemon mode
  /usr/local/wormbase/website/script/wormbase_fastcgi.pl \
       -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d

    // as a deamon bound to a specific port
    script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d

Set up the fastcgi server to launch at boot

Symlink the webapp-fastcgi.init script to /etc/init.d

cd /etc/init.d
sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi

Set up symlinks in runlevels:

cd ../rc3.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
cd ../rc5.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi

Add a cron job that keeps FCGI under control

The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).

sudo crontab -e
*/30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \
                `cat /tmp/wormbase.pid` 104857600

FastCGID

cd src
tar xzf mod_fcid*
cd mod_fcgid*
APXS=/usr/local/apache2/bin/apxs ./configure.apxs
make
sudo make install

Apache

Configure Apache to connect to the fastcgi server

Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf

<VirtualHost *:8000>
     #    ServerName beta.wormbase.org                                                                                     
     ErrorLog /usr/local/wormbase/logs/wormbase2.error_log
     TransferLog /usr/local/wormbase/logs/wormbase2.access_log


     # 502 is a Bad Gateway error, and will occur if the backend server is down
     # This allows us to display a friendly static page that says "down for
     # maintenance"
     Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages
     ErrorDocument 502 /_errors/502.html

     # Map dynamic images to the file system 
     # static images are located at img
     Alias /images       /tmp/wormbase/images/
 
  #  <Directory /filesystem/path/to/MyApp/root/static>
  #      allow from all
  #  </Directory>
  #  <Location /myapp/static>
  #      SetHandler default-handler
  #  </Location>

     # Static content served directly by Apache
     DocumentRoot /usr/local/wormbase/website/root
     #     Alias /static /usr/local/wormbase/website-2.0/root



     # Approach 1: Running as a static server (Apache handles spawning of the webapp)       
     # <IfModule fastcgi_module>
     #    FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3                      
     #    Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/
     # </IfModule>
                                   

     # Approach 2: External Process (via mod_fcgi ONLY)
     <IfModule mod_fastcgi.c>
         # This says to connect to the Catalyst fcgi server running on localhost, port 777
         #  FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777
         # Or to use the socket      
         FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock

         # Place the app at root...
         Alias /    /tmp/wormbase.fcgi/
  
         # ...or somewhere else
         Alias /wormbase/ /tmp/wormbase.fcgi/
      </IfModule>

     # fcgid configuration
     #     <IfModule mod_fcgid>
     #         # This should point at your myapp/root
     #          DocumentRoot /usr/local/wormbase/beta.wormbase.org/root
     #         Alias /static /usr/local/wormbase/beta.wormbase.org/root/static
     #         <Location /static>
     #                   SetHandler default-handler
     #          </Location>
     #
     #         Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/
     #         AddType application/x-httpd-php .php
     #         <Location />
     #                   Options ExecCGI
     #                   Order allow,deny
     #                   Allow from all
     #                   AddHandler fcgid-script .pl
     #          </Location>
     #     </IfModule>

   </VirtualHost>

Edit /usr/local/apache2/conf/httpd.conf

Add the appropriate Listen PORT directive.

Adjust iptables

We need to open port 8000, which should only be to the squid.

sudo emacs /etc/init.d/iptables

Add the following:

  # The new website runs on port 8000. It SHOULD only be accessible by squid                                                              
  $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT
  # ... or only accessible by proxy                                                                                                       
  #  $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT

Then

/etc/init.d/iptables.local restart

PRIVATE BETA TEST:

• Open iptables of web1 for our nginx instance:

# The new website runs on port 8000. It SHOULD only be accessible by squid                                                              
  $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT
  # ... or only accessible by proxy                                                                                                       
  #  $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT