Administration:WormBase Production Environment

Overview

The WormBase production environment consists of a series of partially redundant web and database servers, most sitting behind a load-balancing reverse-proxy server running nginx. This document describes the basic setup and configuration of this environment.

Reverse proxy node

Two servers each running nginx as a load-balancing reverse proxy. Built in memcached support establishes a memory cache amongst all back end web server nodes. Requests are distributed in round-robin fashion.

Web server nodes

Each web cluster node runs the lightweight HTTP server starman listening on port 5000. This http server is glued via PSGI/Plack/Starman to our Catalyst web application.

Currently, each node is -- with the exception of GBrowse -- almost entirely independent, with its own AceDB and MySQL databases.

Web cluster nodes are accessible ONLY to the front end proxy.

Data mining nodes

Social feature node

To resolve

• How is the back end node hosting the user database specified?

• Where are log paths specified? These need to be consolidated.
• image caching
• memcache
• differences in configuration files.
• set up starman on beta.wormbase.org

nginx

• ssl
• proxy caching
• serving up of static content
• memcache

• to test
• logging in
• browser compatibility

• set up starman on dev

• set automatic updates of code and restarting of services.

Paper:

• Unified paper interface
• Longer term:
• Paper
• Overivew

   Laboratory: strain designation
   Fetch all strains for a given lab

http://en.wikipedia.org/wiki/VCard

Reverse Proxy Node

Installing nginx

We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.

1. Install prerequisites

  # Perl Compatabile Regular Expression libaray
  sudo apt-get install libpcre3 libpcre3-dev

  # Fetch and unpack openssel
 wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz
 tar -zxf openssl-0.9.8p.tar.gz

2. Get the nginx cache-purge module

  cd src/
  curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz
  tar xzf ngx_cache_purge-1.2.tar.gz

3. Build and install nginx

  curl -O http://nginx.org/download/nginx-0.8.53.tar.gz
  tar xzf nginx*
  ./configure \
   --prefix=/usr/local/wormbase/services/nginx-0.8.53 \
   --error-log-path=/usr/local/wormbase/logs/nginx-error.log \
   --http-log-path=/usr/local/wormbase/logs/nginx-access.log \
   --with-http_stub_status_module \
   --with-http_ssl_module \
   --with-ipv6 \
   --with-http_realip_module \
    --with-http_addition_module \
    --with-http_image_filter_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_gzip_static_module \
    --with-http_secure_link_module \
    --with-openssl=../openssl-0.9.8p \
   --add-module=../ngx_cache_purge-1.2
   make
   make install
   cd /usr/local/wormbase/services
   ln -s nginx-0.8.53 nginx

4. Symlink the configuration directory

cd /usr/local/wormbase/services/nginx
mv conf conf.original
ln -s /usr/local/wormbase/admin/conf/nginx conf

5. Test the configuration file syntax by:

$ nginx -t

Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching

About Load Balancing

nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:

 http://wiki.nginx.org/NginxHttpUpstreamModule

There are a number of other interesting load-balancing modules that might be of use:

 http://wiki.nginx.org/3rdPartyModules

Starting the Server

Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:

$ /etc/init.d/nginx restart

Set nginx to start at server launch

 sudo /usr/sbin/update-rc.d -f nginx defaults

The output will be similar to this:

Adding system startup for /etc/init.d/nginx ...
  /etc/rc0.d/K20nginx -> ../init.d/nginx
  /etc/rc1.d/K20nginx -> ../init.d/nginx
  /etc/rc6.d/K20nginx -> ../init.d/nginx
  /etc/rc2.d/S20nginx -> ../init.d/nginx
  /etc/rc3.d/S20nginx -> ../init.d/nginx
  /etc/rc4.d/S20nginx -> ../init.d/nginx
  /etc/rc5.d/S20nginx -> ../init.d/nginx

Launch services on the front end machine

# nginx
/etc/init.d/nginx start

# starman
cd /usr/local/wormbase/website/production/bin
./starman-production.sh start

Open appropriate ports

nginx is listening on port web1:2011. Open this port via iptables.

  # The new website front-end proxy, accessible to the world                                                          
  $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT

Webserver Nodes

Individual webserver nodes should be configured essentially as described in the Installing WormBase documentation, except that they do not require nginx.

HTTP server: PSGI/Plack + Starman

See Starman: the lightweight http server section in the Installing WormBase documentation.

Memached/libmemcached

sudo apt-get install memcached

wget http://download.tangent.org/libmemcached-0.44.tar.gz
tar -zxf libmemcached-0.44.tar.gz
cd libmemcached-0.4
./configure –prefix=/usr
make
make install

MISC:

Crypt:
ssl
sudo apt-get install libssl-dev

 libc6
sudo apt-get install libc6-dev

The Webapp

The web app and all Perl libraries will be installed automatically by the deploy_wormbase_webapp.sh script.

 /usr/local/wormbase/website/production -> WSXXXX-YYYY.MM.DD-X.XX-XXXX
 /usr/local/wormbase/website/WSXXX-YYYY.MM.DD-X.XX-XXXX

For details on installation of the web app itself, see the Install The Webapp section of the main Installing WormBase guide.

Launch services on back end machines

# starman
cd /usr/local/wormbase/website/production/bin
./starman-production.sh start

Data mining nodes

The data mining and BLAST/BLAT server replaces the old aceserver. Because it handles requests for the AQL and WB pages, it runs the full website and has all mysql and acedb databases.

Social feature node

The WormBase Blog, the WormBase Wiki, and the Worm Community Forums all rely on third party software. To make it easy to update this software, each of these components is maintained as a separate name-based virtual host running on the same server: wb-social.oicr.on.ca.

The WormBase Blog

The WormBase blog is a subdomain of wormbase.org: blog.wormbase.org. If it's moved, the DNS entry *must* be updated!

     Host/Port : wb-social.oicr.on.ca:80
     Alias: blog.wormbase.org
MySQL database : wormbase_wordpress_blog
 Document root : /usr/local/wormbase/website-blog/current
        Logs   : /usr/local/wormbase/blogs-access_log, /usr/local/wormbase/logs/blogs-error_log

Blog files are stored in /usr/local/wormbase/website-blog/current:

 current -> wordpress-2.92

Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf

<VirtualHost *:80>
   ServerName blog.wormbase.org
   DocumentRoot /usr/local/wormbase/website-blog

    <Directory "/usr/local/wormbase/website-blog">
       DirectoryIndex index.php index.html
       AddType application/x-httpd-php .php
       Order Deny,Allow
       Allow from all
   </Directory>

   LogFormat "%h %l %u %t \"%r\" %s %b" common
   LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
   LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous

   ErrorLog     /usr/local/wormbase/logs/blog-error_log
   CustomLog    /usr/local/wormbase/logs/blog-access_log combined_format
</VirtualHost>

NOTE: when upgrading, be sure to copy the wp-config.php file and entire wp-content/ directory.

The WormBase Wiki

The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!

     Host/Port : wb-social.oicr.on.ca:80
     Alias: wiki.wormbase.org
MySQL database : wormbase_wiki
 Document root : /usr/local/wormbase/website-wiki/current
        Logs   : /usr/local/wormbase/wiki-access_log, /usr/local/wormbase/logs/wiki-error_log

Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf

<VirtualHost *:80>
   ServerName wiki.wormbase.org

   # Current is a symlink to the current installation.
   DocumentRoot /usr/local/wormbase/website-wiki/current

    <Directory "/usr/local/wormbase/website-wiki/current">
       DirectoryIndex index.php index.html
       AddType application/x-httpd-php .php
       Order Deny,Allow
       Allow from all
   </Directory>

   LogFormat "%h %l %u %t \"%r\" %s %b" common
   LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
   LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous

   ErrorLog     /usr/local/wormbase/logs/wiki-error_log
   CustomLog    /usr/local/wormbase/logs/wiki-access_log combined_format
</VirtualHost>

The Worm Community Forums

The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!

     Host/Port : wb-social.oicr.on.ca:80
     Alias: forums.wormbase.org
MySQL database : wormbaseforumssmf
 Document root : /usr/local/wormbase/website-forums
        Logs   : /usr/local/wormbase/forums-access_log, /usr/local/wormbase/logs/forums-error_log

Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf

<VirtualHost *:80>
   ServerName forums.wormbase.org
   # Current is a symlink to the current version of SMF
   DocumentRoot /usr/local/wormbase/website-forums/current

    <Directory "/usr/local/wormbase/website-forums/current">
       DirectoryIndex index.php index.html
       AddType application/x-httpd-php .php
       Order Deny,Allow
       Allow from all
   </Directory>

   LogFormat "%h %l %u %t \"%r\" %s %b" common
   LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
   LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous

   ErrorLog     /usr/local/wormbase/logs/forums-error_log
   CustomLog    /usr/local/wormbase/logs/forums-access_log combined_format

</VirtualHost>

Add "Listen 8081" to the primary httpd.conf file.

Note: If the forum is moved, it is also necessary to update Settings.php and the paths to the Sources and Themes directories in the forum Administration Panel > Configuration > Server Settings.

Miscellaneous

Build the user preferences database

The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on the same server providing the reverse proxy.

mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';

# All nodes currently use the same session database.
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca';
mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';

Q: How/Where do I configure the location of the wormbase_user database in the application?

Adjust iptables NOT DONE

We need to open port 8000, which should only be to the squid.

sudo emacs /etc/init.d/iptables

Add the following:

  # The new website runs on port 8000. It SHOULD only be accessible by squid                                                              
  $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT
  # ... or only accessible by proxy                                                                                                       
  #  $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT

Then

/etc/init.d/iptables.local restart

---

EVERYTHING BELOW HERE IS DEPRECATED

Monitoring

See the monitoring services document? Nagios requires apache and fcgi

Should I preserve the fastcgi,fcgi configuration just in case?

FastCGI, FCGI, Apache, and mod_perl

Originally, WormBase ran under apache + mod_perl.

We also experimented with fcgi and fcgid +apache.

Installing fastcgi

curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz
tar xzf mod_fastcgi*
cd mod_fastcgi*
cp Makefile.AP2 Makefile
make top_dir=/usr/local/apache2
sudo make top_dir=/usr/local/apache2 install

If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:

sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c

Add an entry to httpd.conf like this:

 LoadModule fastcgi_module modules/mod_fastcgi.so

 // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file.
 // Edit it to read exactly as above.

Launch the fastcgi server

   // as a socket server in daemon mode
  /usr/local/wormbase/website/script/wormbase_fastcgi.pl \
       -l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d

    // as a deamon bound to a specific port
    script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d

Set up the fastcgi server to launch at boot

Symlink the webapp-fastcgi.init script to /etc/init.d

cd /etc/init.d
sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi

Set up symlinks in runlevels:

cd ../rc3.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
cd ../rc5.d
sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi

Add a cron job that keeps FCGI under control

The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).

sudo crontab -e
*/30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \
                `cat /tmp/wormbase.pid` 104857600

mod_fcgid

mod_fcgid is an alternative to fcgi

cd src/
wget http://www.carfab.com/apachesoftware/httpd/mod_fcgid/mod_fcgid-2.3.5.tar.gz
tar xzf mod_fcgid-2.3.5.tar.gz 
cd mod_fcgid-2.3.5   
APXS=/usr/local/apache2/bin/apxs ./configure.apxs  
make
sudo make install

Apache

Configure Apache to connect to the fastcgi server

Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf

<VirtualHost *:8000>
     #    ServerName beta.wormbase.org                                                                                     
     ErrorLog /usr/local/wormbase/logs/wormbase2.error_log
     TransferLog /usr/local/wormbase/logs/wormbase2.access_log


     # 502 is a Bad Gateway error, and will occur if the backend server is down
     # This allows us to display a friendly static page that says "down for
     # maintenance"
     Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages
     ErrorDocument 502 /_errors/502.html

     # Map dynamic images to the file system 
     # static images are located at img
     Alias /images       /tmp/wormbase/images/
 
  #  <Directory /filesystem/path/to/MyApp/root/static>
  #      allow from all
  #  </Directory>
  #  <Location /myapp/static>
  #      SetHandler default-handler
  #  </Location>

     # Static content served directly by Apache
     DocumentRoot /usr/local/wormbase/website/root
     #     Alias /static /usr/local/wormbase/website-2.0/root



     # Approach 1: Running as a static server (Apache handles spawning of the webapp)       
     # <IfModule fastcgi_module>
     #    FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3                      
     #    Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/
     # </IfModule>
                                   

     # Approach 2: External Process (via mod_fcgi ONLY)
     <IfModule mod_fastcgi.c>
         # This says to connect to the Catalyst fcgi server running on localhost, port 777
         #  FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777
         # Or to use the socket      
         FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock

         # Place the app at root...
         Alias /    /tmp/wormbase.fcgi/
  
         # ...or somewhere else
         Alias /wormbase/ /tmp/wormbase.fcgi/
      </IfModule>

     # fcgid configuration
     #     <IfModule mod_fcgid>
     #         # This should point at your myapp/root
     #          DocumentRoot /usr/local/wormbase/beta.wormbase.org/root
     #         Alias /static /usr/local/wormbase/beta.wormbase.org/root/static
     #         <Location /static>
     #                   SetHandler default-handler
     #          </Location>
     #
     #         Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/
     #         AddType application/x-httpd-php .php
     #         <Location />
     #                   Options ExecCGI
     #                   Order allow,deny
     #                   Allow from all
     #                   AddHandler fcgid-script .pl
     #          </Location>
     #     </IfModule>

   </VirtualHost>

Edit /usr/local/apache2/conf/httpd.conf

Add the appropriate Listen PORT directive.