Administration:WormBase Production Environment

Overview

The WormBase production environment consists of a series of partially redundant web and database servers, most sitting behind a load-balancing reverse-proxy server running nginx.

Reverse proxy node

Web server nodes

Each web cluster node runs the lightweight HTTP server starman listening on port 5000. This http server is glued via PSGI/Plack/Starman to our Catalyst web application.

Currently, each node is -- with the exception of GBrowse -- almost entirely independent, with its own AceDB and MySQL databases.

Web cluster nodes are accessible ONLY to the front end proxy.

Data mining nodes

Social feature node

To resolve

• How is the back end node hosting the user database specified?

• Where are log paths specified? These need to be consolidated.
• image caching
• memcache
• differences in configuration files.
• set up starman on beta.wormbase.org

nginx

• ssl
• proxy caching
• serving up of static content
• memcache

• to test
• logging in
• browser compatibility

• set up starman on dev

• set automatic updates of code and restarting of services.

Paper:

• Unified paper interface
• Longer term:
• Paper
• Overivew

   Laboratory: strain designation
   Fetch all strains for a given lab

http://en.wikipedia.org/wiki/VCard

Webserver nodes as described in the Installing WormBase document.

Reverse Proxy Node

Installing nginx

We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.

1. Install prerequisites

  # Perl Compatabile Regular Expression libaray
  sudo apt-get install libpcre3 libpcre3-dev

  # Fetch and unpack openssel
 wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz
 tar -zxf openssl-0.9.8p.tar.gz

2. Get the nginx cache-purge module

  cd src/
  curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz
  tar xzf ngx_cache_purge-1.2.tar.gz

3. Build and install nginx

  curl -O http://nginx.org/download/nginx-0.8.53.tar.gz
  tar xzf nginx*
  ./configure \
   --prefix=/usr/local/wormbase/services/nginx-0.8.53 \
   --error-log-path=/usr/local/wormbase/logs/nginx-error.log \
   --http-log-path=/usr/local/wormbase/logs/nginx-access.log \
   --with-http_stub_status_module \
   --with-http_ssl_module \
   --with-ipv6 \
   --with-http_realip_module \
    --with-http_addition_module \
    --with-http_image_filter_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_gzip_static_module \
    --with-http_secure_link_module \
    --with-openssl=../openssl-0.9.8p \
   --add-module=../ngx_cache_purge-1.2
   make
   make install
   cd /usr/local/wormbase/services
   ln -s nginx-0.8.53 nginx

4. Symlink the configuration directory

cd /usr/local/wormbase/services/nginx
mv conf conf.original
ln -s /usr/local/wormbase/admin/conf/nginx conf

5. Test the configuration file syntax by:

$ nginx -t

Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching

Load Balancing

nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:

 http://wiki.nginx.org/NginxHttpUpstreamModule

There are a number of other interesting load-balancing modules that might be of use:

 http://wiki.nginx.org/3rdPartyModules

Starting the Server

Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:

$ /etc/init.d/nginx restart

Set nginx to start at server launch

 sudo /usr/sbin/update-rc.d -f nginx defaults

The output will be similar to this:

Adding system startup for /etc/init.d/nginx ...
  /etc/rc0.d/K20nginx -> ../init.d/nginx
  /etc/rc1.d/K20nginx -> ../init.d/nginx
  /etc/rc6.d/K20nginx -> ../init.d/nginx
  /etc/rc2.d/S20nginx -> ../init.d/nginx
  /etc/rc3.d/S20nginx -> ../init.d/nginx
  /etc/rc4.d/S20nginx -> ../init.d/nginx
  /etc/rc5.d/S20nginx -> ../init.d/nginx

Launch services on the front end machine

# nginx
/etc/init.d/nginx start

# starman
cd /usr/local/wormbase/website/production/bin
./starman-production.sh start

Webserver Nodes

HTTP server: PSGI/Plack + Starman

PSGI: specification for Perl superglue between frameworks and servers. Plack is an implementation of PSGI. Compare to Rack (Ruby) or Jack (Javascript).

  http://plackperl.org/

Starman is a high performance pre-forking Perl PSGI server. We're using it in place of Apache+fastcgi.

 https://github.com/miyagawa/Starman

Install Plack and Starman

Catalyst::Controller::Metal

Catalyst::Engine::PSGI

Catalyst::Helper::PSGI

Plack::Test::Adopt::Catalyst

cpanm Task::Plack

cpanm Starman

Configuration

The PSGI glue resides at:

script/wormbase_psgi.psgi

Starting Starman

starman script/wormbase_psgi.psgi
OR
starman -MFindBin script/wormbase_psgi.psgi

Set up starman to launch automatically

Copy the stub init script from wormbase/conf/starman/starman.init:

 cp /usr/local/wormbase/admin/conf/starman/starman.init

Memached/libmemcached

sudo apt-get install memcached

wget http://download.tangent.org/libmemcached-0.44.tar.gz
tar -zxf libmemcached-0.44.tar.gz
cd libmemcached-0.4
./configure –prefix=/usr
make
make install

MISC:

Crypt:
ssl
sudo apt-get install libssl-dev

 libc6
sudo apt-get install libc6-dev

Install the Webapp

The web app and all Perl libraries will be installed automatically by the deploy_wormbase_webapp.sh script.

 /usr/local/wormbase/website/production -> WSXXXX-YYYY.MM.DD-X.XX-XXXX
 /usr/local/wormbase/website/WSXXX-YYYY.MM.DD-X.XX-XXXX

For details on installation of the web app itself, see the Install The Webapp section of the main Installing WormBase guide.

Build the user schema

The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on web1.

ssh web1
mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';

# All nodes currently use the same session database.
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';

Open appropriate ports

nginx is listening on port web1:2011. Open this port via iptables.

  # The new website front-end proxy, accessible to the world                                                          
  $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT

Launch services on back end machines

# starman
cd /usr/local/wormbase/website/production/bin
./starman-production.sh start

Data mining nodes

The data mining and BLAST/BLAT server replaces the old aceserver. Because it handles requests for the AQL and WB pages, it runs the full website and has all mysql and acedb databases.

Adjust iptables

We need to open port 8000, which should only be to the squid.

sudo emacs /etc/init.d/iptables

Add the following:

  # The new website runs on port 8000. It SHOULD only be accessible by squid                                                              
  $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT
  # ... or only accessible by proxy                                                                                                       
  #  $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT

Then

/etc/init.d/iptables.local restart

---

EVERYTHING BELOW HERE IS DEPRECATED

FastCGI, FCGI, Apache, and mod_perl

Originally, WormBase ran under apache + mod_perl.

We also experimented with fcgi and fcgid +apache.

Installing fastcgi

curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz
tar xzf mod_fastcgi*
cd mod_fastcgi*
cp Makefile.AP2 Makefile
make top_dir=/usr/local/apache2
sudo make top_dir=/usr/local/apache2 install

If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:

sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c

Add an entry to httpd.conf like this:

 LoadModule fastcgi_module modules/mod_fastcgi.so

 // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file.
 // Edit it to read exactly as above.

Launch the fastcgi server

   // as a socket server in daemon mode
  /usr/local/wormbase/website/script/wormbase_fastcgi.pl \
       -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d

    // as a deamon bound to a specific port
    script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d

Set up the fastcgi server to launch at boot

Symlink the webapp-fastcgi.init script to /etc/init.d

cd /etc/init.d
sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi

Set up symlinks in runlevels:

cd ../rc3.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
cd ../rc5.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi

Add a cron job that keeps FCGI under control

The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).

sudo crontab -e
*/30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \
                `cat /tmp/wormbase.pid` 104857600

mod_fcgid

mod_fcgid is an alternative to fcgi

cd src/
wget http://www.carfab.com/apachesoftware/httpd/mod_fcgid/mod_fcgid-2.3.5.tar.gz
tar xzf mod_fcgid-2.3.5.tar.gz 
cd mod_fcgid-2.3.5   
APXS=/usr/local/apache2/bin/apxs ./configure.apxs  
make
sudo make install

Apache

Configure Apache to connect to the fastcgi server

Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf

<VirtualHost *:8000>
     #    ServerName beta.wormbase.org                                                                                     
     ErrorLog /usr/local/wormbase/logs/wormbase2.error_log
     TransferLog /usr/local/wormbase/logs/wormbase2.access_log


     # 502 is a Bad Gateway error, and will occur if the backend server is down
     # This allows us to display a friendly static page that says "down for
     # maintenance"
     Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages
     ErrorDocument 502 /_errors/502.html

     # Map dynamic images to the file system 
     # static images are located at img
     Alias /images       /tmp/wormbase/images/
 
  #  <Directory /filesystem/path/to/MyApp/root/static>
  #      allow from all
  #  </Directory>
  #  <Location /myapp/static>
  #      SetHandler default-handler
  #  </Location>

     # Static content served directly by Apache
     DocumentRoot /usr/local/wormbase/website/root
     #     Alias /static /usr/local/wormbase/website-2.0/root



     # Approach 1: Running as a static server (Apache handles spawning of the webapp)       
     # <IfModule fastcgi_module>
     #    FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3                      
     #    Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/
     # </IfModule>
                                   

     # Approach 2: External Process (via mod_fcgi ONLY)
     <IfModule mod_fastcgi.c>
         # This says to connect to the Catalyst fcgi server running on localhost, port 777
         #  FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777
         # Or to use the socket      
         FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock

         # Place the app at root...
         Alias /    /tmp/wormbase.fcgi/
  
         # ...or somewhere else
         Alias /wormbase/ /tmp/wormbase.fcgi/
      </IfModule>

     # fcgid configuration
     #     <IfModule mod_fcgid>
     #         # This should point at your myapp/root
     #          DocumentRoot /usr/local/wormbase/beta.wormbase.org/root
     #         Alias /static /usr/local/wormbase/beta.wormbase.org/root/static
     #         <Location /static>
     #                   SetHandler default-handler
     #          </Location>
     #
     #         Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/
     #         AddType application/x-httpd-php .php
     #         <Location />
     #                   Options ExecCGI
     #                   Order allow,deny
     #                   Allow from all
     #                   AddHandler fcgid-script .pl
     #          </Location>
     #     </IfModule>

   </VirtualHost>

Edit /usr/local/apache2/conf/httpd.conf

Add the appropriate Listen PORT directive.

Servers

Configuring iptables

We don't want end users to directly access back end machines. Instead, we want to force them to pass through the caching proxy. For now, we will only allow the reverse proxy access to the server on port 80.

We only want the front end proxy (currently at CSHL) to be able to access the machine directly on port 80. See conf/iptables.local in the wormbase-admin module for details.

The Wiki, Blog, and Forums Server

The WormBase Blog, the WormBase Wiki, and the Worm Community Forums all rely on third party software. To make it easy to update this software, each of these components is maintained as a separate virtual host running on its own port.

Iptables

The blog, wiki, and forums are all virtual hosts. However, we don't want the server to respond on port 80 for requests to the machine's IP.

The WormBase Blog

The WormBase blog is a subdomain of wormbase.org: blog.wormbase.org. If it's moved, the DNS entry *must* be updated!

     Host/Port : wb-acedb1.oicr.on.ca:80
     Alias: blog.wormbase.org
MySQL database : wormbase_wordpress_blog
 Document root : /usr/local/wormbase/website-blog/current
        Logs   : /usr/local/wormbase/blogs-access_log, /usr/local/wormbase/logs/blogs-error_log

Blog files are stored in /usr/local/wormbase/website-blog/current:

 current -> wordpress-2.92

Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf

<VirtualHost *:80>
   ServerName blog.wormbase.org
   DocumentRoot /usr/local/wormbase/website-blog

    <Directory "/usr/local/wormbase/website-blog">
       DirectoryIndex index.php index.html
       AddType application/x-httpd-php .php
       Order Deny,Allow
       Allow from all
   </Directory>

   LogFormat "%h %l %u %t \"%r\" %s %b" common
   LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
   LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous

   ErrorLog     /usr/local/wormbase/logs/blog-error_log
   CustomLog    /usr/local/wormbase/logs/blog-access_log combined_format
</VirtualHost>

NOTE: when upgrading, be sure to copy the wp-config.php file and entire wp-content/ directory.

The WormBase Wiki

The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!

     Host/Port : wb-acedb1.oicr.on.ca:80
     Alias: wiki.wormbase.org
MySQL database : wormbase_wiki
 Document root : /usr/local/wormbase/website-wiki/current
        Logs   : /usr/local/wormbase/wiki-access_log, /usr/local/wormbase/logs/wiki-error_log

Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf

<VirtualHost *:80>
   ServerName wiki.wormbase.org

   # Current is a symlink to the current installation.
   DocumentRoot /usr/local/wormbase/website-wiki/current

    <Directory "/usr/local/wormbase/website-wiki/current">
       DirectoryIndex index.php index.html
       AddType application/x-httpd-php .php
       Order Deny,Allow
       Allow from all
   </Directory>

   LogFormat "%h %l %u %t \"%r\" %s %b" common
   LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
   LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous

   ErrorLog     /usr/local/wormbase/logs/wiki-error_log
   CustomLog    /usr/local/wormbase/logs/wiki-access_log combined_format
</VirtualHost>

The Worm Community Forums

The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!

     Host/Port : wb-acedb1.oicr.on.ca:80
     Alias: forums.wormbase.org
MySQL database : wormbaseforumssmf
 Document root : /usr/local/wormbase/website-forums
        Logs   : /usr/local/wormbase/forums-access_log, /usr/local/wormbase/logs/forums-error_log

Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf

<VirtualHost *:80>
   ServerName forums.wormbase.org
   # Current is a symlink to the current version of SMF
   DocumentRoot /usr/local/wormbase/website-forums/current

    <Directory "/usr/local/wormbase/website-forums/current">
       DirectoryIndex index.php index.html
       AddType application/x-httpd-php .php
       Order Deny,Allow
       Allow from all
   </Directory>

   LogFormat "%h %l %u %t \"%r\" %s %b" common
   LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
   LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous

   ErrorLog     /usr/local/wormbase/logs/forums-error_log
   CustomLog    /usr/local/wormbase/logs/forums-access_log combined_format

</VirtualHost>

Add "Listen 8081" to the primary httpd.conf file.

Note: If the forum is moved, it is also necessary to update Settings.php and the paths to the Sources and Themes directories in the forum Administration Panel > Configuration > Server Settings.

Logs

All relevant logs can be found at:

ls /usr/local/wormbase/logs
nginx-error.log    // The reverse proxy error log
nginx-access.log   // The reverse proxy access log
nginx-cache.log    // The reverse proxy cache log
catalyst_error.log // The catalyst error log