Difference between revisions of "Administration:WormBase Production Environment"
(→MISC) |
|||
| Line 54: | Line 54: | ||
# Perl Compatabile Regular Expression libaray | # Perl Compatabile Regular Expression libaray | ||
| − | sudo apt-get install libpcre3 libpcre3-dev | + | sudo apt-get install libpcre3 libpcre3-dev {libssl-dev libc6-dev : necessary?} |
# Fetch and unpack openssel | # Fetch and unpack openssel | ||
Revision as of 16:33, 10 January 2011
Contents
- 1 Overview
- 2 To resolve
- 3 Nodes
- 4 NFS
- 5 Miscellaneous
- 6 See Also
- 7 Monitoring
- 8 Apache
Overview
The WormBase production environment consists of a series of partially redundant web and database servers, most sitting behind a load-balancing reverse-proxy server running nginx. This document describes the basic setup and configuration of this environment.
Reverse proxy node
- Two servers each running nginx as a load-balancing reverse proxy. Built in memcached support establishes a memory cache amongst all back end web server nodes. Requests are distributed in round-robin fashion.
nginx acedb starman_webapp mysql userdb
Web server nodes
- Each web cluster node runs the lightweight HTTP server starman listening on port 5000. This http server is glued via PSGI/Plack/Starman to our Catalyst web application.
- Currently, each node is -- with the exception of GBrowse -- almost entirely independent, with its own AceDB and MySQL databases.
- Web cluster nodes are accessible ONLY to the front end proxy.
Data mining nodes
Social feature node
To resolve
- How is the back end node hosting the user database specified?
- Where are log paths specified? These need to be consolidated.
- image caching
- differences in configuration files.
nginx
- ssl
- proxy caching
- to test
- logging in
- browser compatibility
- set automatic updates of code and restarting of services
Nodes
Reverse Proxy Node
nginx
Installation
We'll place nginx entirely within the wormbase root directory. It's configuration and init files are maintained in the wormbase-admin module.
1. Install prerequisites
# Perl Compatabile Regular Expression libaray
sudo apt-get install libpcre3 libpcre3-dev {libssl-dev libc6-dev : necessary?}
# Fetch and unpack openssel wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz tar -zxf openssl-0.9.8p.tar.gz
2. Get the nginx cache-purge module
cd src/ curl -O http://labs.frickle.com/files/ngx_cache_purge-1.2.tar.gz tar xzf ngx_cache_purge-1.2.tar.gz
3. Build and install nginx
curl -O http://nginx.org/download/nginx-0.8.53.tar.gz tar xzf nginx* ./configure \ --prefix=/usr/local/wormbase/services/nginx-0.8.53 \ --error-log-path=/usr/local/wormbase/logs/nginx-error.log \ --http-log-path=/usr/local/wormbase/logs/nginx-access.log \ --with-http_stub_status_module \ --with-http_ssl_module \ --with-ipv6 \ --with-http_realip_module \ --with-http_addition_module \ --with-http_image_filter_module \ --with-http_sub_module \ --with-http_dav_module \ --with-http_flv_module \ --with-http_gzip_static_module \ --with-http_secure_link_module \ --with-openssl=../openssl-0.9.8p \ --add-module=../ngx_cache_purge-1.2 make make install cd /usr/local/wormbase/services ln -s nginx-0.8.53 nginx
4. Symlink the configuration directory
cd /usr/local/wormbase/services/nginx mv conf conf.original ln -s /usr/local/wormbase/admin/conf/nginx conf
5. Test the configuration file syntax by:
$ nginx -t
Here's a more complicated example demonstrating caching and load balancing: http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching
About Load Balancing
nginx relies on the NginxHttpUpstreamModule for load balancing. It's built-in by default. The documentation contains a number of possibly useful configuration directives:
http://wiki.nginx.org/NginxHttpUpstreamModule
There are a number of other interesting load-balancing modules that might be of use:
http://wiki.nginx.org/3rdPartyModules
Starting the Server
Copy wormbase-admin/conf/nginx.init to /etc/init.d/nginx to setup a suitable init script. (Re)start the server by:
$ /etc/init.d/nginx restart
Set nginx to start at server launch
sudo /usr/sbin/update-rc.d -f nginx defaults
The output will be similar to this:
Adding system startup for /etc/init.d/nginx ... /etc/rc0.d/K20nginx -> ../init.d/nginx /etc/rc1.d/K20nginx -> ../init.d/nginx /etc/rc6.d/K20nginx -> ../init.d/nginx /etc/rc2.d/S20nginx -> ../init.d/nginx /etc/rc3.d/S20nginx -> ../init.d/nginx /etc/rc4.d/S20nginx -> ../init.d/nginx /etc/rc5.d/S20nginx -> ../init.d/nginx
memcached
Install
$ sudo apt-get install memcached
Configure
Make memcached listen to all IP addresses, not just requests from localhost:'
$ emacs /etc/memcached.conf -l 127.0.0.1 <--- comment out this line
Lock down access to memcached via iptables as described below.
memcached installed memcached configed iptables config web1 web2 configured web3 web4 mining
Configure iptables
################################################## # New architecture # # Proxy nodes: # service port accessibility # nginx 2011, later on 80 all # memcached 11211 cluster # starman 5000 localhost # mysql 3306 cluster, dev # # Backend nodes # service port accessibility # starman 5000 proxy nodes # memcached 11211 cluster # ##################################################
# Proxy # The new website front-end proxy, accessible to the world $BIN -A INPUT -p tcp --dport 2011 -m state --state NEW -j ACCEPT
# backend # starman $BIN -A INPUT -p tcp -m iprange --src-range 206.108.125.168-206.108.125.190 --dport 5000 -j ACCEPT # memcached $BIN -A INPUT -p tcp -m iprange --src-range 206.108.125.168-206.108.125.190 --dport 11211 -j ACCEPT
Launch services on the front end machine
# nginx $ sudo /etc/init.d/nginx restart
# memcached $ sudo /etc/init.d/memcached restart
# starman cd /usr/local/wormbase/website/production/bin $ ./starman-production.sh start
Webserver Nodes
Individual webserver nodes should be configured essentially as described in the Installing WormBase documentation, except that they do not require nginx.
HTTP server: PSGI/Plack + Starman
See Starman: the lightweight http server section in the Installing WormBase documentation.
Memached/libmemcached
See above for details.
The Webapp
The web app and all Perl libraries will be installed automatically by the deploy_wormbase_webapp.sh script.
/usr/local/wormbase/website/production -> WSXXXX-YYYY.MM.DD-X.XX-XXXX /usr/local/wormbase/website/WSXXX-YYYY.MM.DD-X.XX-XXXX
For details on installation of the web app itself, see the Install The Webapp section of the main Installing WormBase guide.
Launch services on back end machines
# starman cd /usr/local/wormbase/website/production/bin ./starman-production.sh start
Data mining nodes
The data mining and BLAST/BLAT server replaces the old aceserver. Because it handles requests for the AQL and WB pages, it is configured exactly as web cluster nodes, with the addition of BLAST, BLAT, and ePCR.
To support AcePerl data mining, port 2005 needs to be open.
To support BioPerl queries of the GFF databsaes, port 3306 needs to be open.
WormMart node
Social feature node
The WormBase Blog, the WormBase Wiki, and the Worm Community Forums all rely on third party software. To make it easy to update this software, each of these components is maintained as a separate name-based virtual host running on the same server: wb-social.oicr.on.ca.
The WormBase Blog
The WormBase blog is a subdomain of wormbase.org: blog.wormbase.org. If it's moved, the DNS entry *must* be updated!
Host/Port : wb-social.oicr.on.ca:80
Alias: blog.wormbase.org
MySQL database : wormbase_wordpress_blog
Document root : /usr/local/wormbase/website-blog/current
Logs : /usr/local/wormbase/blogs-access_log, /usr/local/wormbase/logs/blogs-error_log
Blog files are stored in /usr/local/wormbase/website-blog/current:
current -> wordpress-2.92
Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf
<VirtualHost *:80>
ServerName blog.wormbase.org
DocumentRoot /usr/local/wormbase/website-blog
<Directory "/usr/local/wormbase/website-blog">
DirectoryIndex index.php index.html
AddType application/x-httpd-php .php
Order Deny,Allow
Allow from all
</Directory>
LogFormat "%h %l %u %t \"%r\" %s %b" common
LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous
ErrorLog /usr/local/wormbase/logs/blog-error_log
CustomLog /usr/local/wormbase/logs/blog-access_log combined_format
</VirtualHost>
NOTE: when upgrading, be sure to copy the wp-config.php file and entire wp-content/ directory.
The WormBase Wiki
The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!
Host/Port : wb-social.oicr.on.ca:80
Alias: wiki.wormbase.org
MySQL database : wormbase_wiki
Document root : /usr/local/wormbase/website-wiki/current
Logs : /usr/local/wormbase/wiki-access_log, /usr/local/wormbase/logs/wiki-error_log
Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf
<VirtualHost *:80>
ServerName wiki.wormbase.org
# Current is a symlink to the current installation.
DocumentRoot /usr/local/wormbase/website-wiki/current
<Directory "/usr/local/wormbase/website-wiki/current">
DirectoryIndex index.php index.html
AddType application/x-httpd-php .php
Order Deny,Allow
Allow from all
</Directory>
LogFormat "%h %l %u %t \"%r\" %s %b" common
LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous
ErrorLog /usr/local/wormbase/logs/wiki-error_log
CustomLog /usr/local/wormbase/logs/wiki-access_log combined_format
</VirtualHost>
The Worm Community Forums
The WormBase Wiki is a subdirectory of the primary WormBase domain. If it's moved, the proxy that sits in front of it must be updated!
Host/Port : wb-social.oicr.on.ca:80
Alias: forums.wormbase.org
MySQL database : wormbaseforumssmf
Document root : /usr/local/wormbase/website-forums
Logs : /usr/local/wormbase/forums-access_log, /usr/local/wormbase/logs/forums-error_log
Add the following apache configuration to /usr/local/apache2/conf/extras/httpd-vhosts.conf
<VirtualHost *:80>
ServerName forums.wormbase.org
# Current is a symlink to the current version of SMF
DocumentRoot /usr/local/wormbase/website-forums/current
<Directory "/usr/local/wormbase/website-forums/current">
DirectoryIndex index.php index.html
AddType application/x-httpd-php .php
Order Deny,Allow
Allow from all
</Directory>
LogFormat "%h %l %u %t \"%r\" %s %b" common
LogFormat "%h %l %u %t %{Referer}i \"%{User-Agent}i\" \"%r\" %s %b" combined_format
LogFormat "witheld %l %u %t \"%r\" %s %b" anonymous
ErrorLog /usr/local/wormbase/logs/forums-error_log
CustomLog /usr/local/wormbase/logs/forums-access_log combined_format
</VirtualHost>
Add "Listen 8081" to the primary httpd.conf file.
Note: If the forum is moved, it is also necessary to update Settings.php and the paths to the Sources and Themes directories in the forum Administration Panel > Configuration > Server Settings.
NFS
We use NFS at WormBase to consolidate logs, simplify maintainence of temporary and static files, and provide access to common (file-system) databases.
NFS is served from wb-web1. NOTE: This may be a I/O performance bottleneck - we may need a separate machine
/usr/local/wormbase/shared
This directory will be mounted at the same path for each client.
Contents:
logs/ // Server logs databases // File-based databases for various searches tmp // tmp images static // static files
Note that logs and databases are symlinks in each production node:
/usr/local/wormbase/databases -> shared/databases
Configuring the NFS server
Install NFS
On Debian, the NFS server needs:
sudo apt-get install nfs-kernel-server nfs-common portmap
Share mounts are configured through /etc/exports
# WormBase NFS for temporary and static content /usr/local/wormbase/shared 206.108.125.177(rw,no_root_squash) 206.108.125.190(rw,no_root_squash) 206.108.125.168(rw,no_root_squash) 206.108.125.191(rw,no_root_squash)
After making changes to /etc/exports, load them by:
sudo exportfs -a
Lock down access to NFS from most hosts
We can use /etc/hosts.deny to quickly lock down NFS:
portmap:ALL lockd:ALL mountd:ALL rquotad:ALL statd:ALL
Allow access to select hosts
We can use /etc/hosts.allow to specifically allow access to NFS:
# WormBase NFS services portmap: ***.***.***.*** lockd: ***.***.***.*** rquotad: ***.***.***.*** mountd: ***.***.***.*** statd: ***.***.***.***
Configure NFS clients
Install NFS
On Debian, NFS clients require:
sudo apt-get install nfs-common portmap
sudo mount ${NFS_SERVER}:/usr/local/wormbase/nfs /usr/local/wormbase/shared // The shared dir must already exist
sudo umount /usr/local/wormbase/shared
sudo emacs /etc/fstab
Add
wb-web1.oicr.on.ca:/usr/local/wormbase/shared /usr/local/wormbase/shared nfs rw 0 0
Miscellaneous
Build the user preferences database
The website uses a mysql backend to store user preferences, browsing history, session data. This shouldn't ever need to be recreated (at least until we have a migration path in place from an old database to a new one!), but here's how to create it for reference. For now, this database is hosted on the same server providing the reverse proxy.
mysql -u root -p < /usr/local/wormbase/website/production/util/user_login.sql mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@localhost';
# All nodes currently use the same session database. mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web1.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web2.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web3.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-web4.oicr.on.ca'; mysql -u root -p -e 'grant all privileges on wormbase_user.* to wb@wb-mining.oicr.on.ca';
Q: How/Where do I configure the location of the wormbase_user database in the application?
Adjust iptables NOT DONE
We need to open port 8000, which should only be to the squid.
sudo emacs /etc/init.d/iptables
Add the following:
# The new website runs on port 8000. It SHOULD only be accessible by squid $BIN -A INPUT -p tcp --dport 8000 -m state --state NEW -j ACCEPT # ... or only accessible by proxy # $BIN -A INPUT -p tcp -s 206.108.125.175 --dport 8000 -m state --state NEW -j ACCEPT
Then
/etc/init.d/iptables.local restart
Logs
All relevant logs can be found at:
ls /usr/local/wormbase/logs nginx-error.log // The reverse proxy error log nginx-access.log // The reverse proxy access log nginx-cache.log // The reverse proxy cache log catalyst_error.log // The catalyst error log
See Also
Monitoring
Logs
EVERYTHING BELOW HERE IS DEPRECATED
Monitoring
See the monitoring services document? Nagios requires apache and fcgi
Should I preserve the fastcgi,fcgi configuration just in case?
FastCGI, FCGI, Apache, and mod_perl
Originally, WormBase ran under apache + mod_perl.
We also experimented with fcgi and fcgid +apache.
Installing fastcgi
curl -O http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz tar xzf mod_fastcgi* cd mod_fastcgi* cp Makefile.AP2 Makefile make top_dir=/usr/local/apache2 sudo make top_dir=/usr/local/apache2 install
If you get an error on make saying it can't find special.mk (which is supposed to be distributed with httpd but isn't on CentOS and is not part of httpd-devel, either), try:
sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c
Add an entry to httpd.conf like this:
LoadModule fastcgi_module modules/mod_fastcgi.so // Note: if you use the apxs command above, it inserts an incorrect line into your httpd.conf file. // Edit it to read exactly as above.
Launch the fastcgi server
// as a socket server in daemon mode
/usr/local/wormbase/website/script/wormbase_fastcgi.pl \
-l /tmp/wormbase.sock -n 5 -p /tmp/wormbase.pid -d
// as a deamon bound to a specific port
script/wormbase_fastcgi.pl -l :3001 -n 5 -p /tmp/wormbase.pid -d
Set up the fastcgi server to launch at boot
Symlink the webapp-fastcgi.init script to /etc/init.d
cd /etc/init.d sudo ln -s /usr/local/wormbase/website/util/init/webapp-fastcgi.init wormbase-fastcgi
Set up symlinks in runlevels:
cd ../rc3.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi cd ../rc5.d sudo ln -s ../init.d/wormbase-fastcgi S99wormbase-fastcgi
Add a cron job that keeps FCGI under control
The following cron job will kill off fcgi children that exceed the specified memory limit (in bytes).
sudo crontab -e
*/30 * * * * /usr/local/wormbase/website/util/crons/fastcgi-childreaper.pl \
`cat /tmp/wormbase.pid` 104857600
mod_fcgid
mod_fcgid is an alternative to fcgi
cd src/ wget http://www.carfab.com/apachesoftware/httpd/mod_fcgid/mod_fcgid-2.3.5.tar.gz tar xzf mod_fcgid-2.3.5.tar.gz cd mod_fcgid-2.3.5 APXS=/usr/local/apache2/bin/apxs ./configure.apxs make sudo make install
Apache
Configure Apache to connect to the fastcgi server
Edit /usr/local/apache2/conf/extra/httpd-vhosts.conf
<VirtualHost *:8000>
# ServerName beta.wormbase.org
ErrorLog /usr/local/wormbase/logs/wormbase2.error_log
TransferLog /usr/local/wormbase/logs/wormbase2.access_log
# 502 is a Bad Gateway error, and will occur if the backend server is down
# This allows us to display a friendly static page that says "down for
# maintenance"
Alias /_errors /home/todd/projects/wormbase/website/trunk/root/error-pages
ErrorDocument 502 /_errors/502.html
# Map dynamic images to the file system
# static images are located at img
Alias /images /tmp/wormbase/images/
# <Directory /filesystem/path/to/MyApp/root/static>
# allow from all
# </Directory>
# <Location /myapp/static>
# SetHandler default-handler
# </Location>
# Static content served directly by Apache
DocumentRoot /usr/local/wormbase/website/root
# Alias /static /usr/local/wormbase/website-2.0/root
# Approach 1: Running as a static server (Apache handles spawning of the webapp)
# <IfModule fastcgi_module>
# FastCgiServer /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl -processes 3
# Alias / /usr/local/wormbase/website-2.0/script/wormbase_fastcgi.pl/
# </IfModule>
# Approach 2: External Process (via mod_fcgi ONLY)
<IfModule mod_fastcgi.c>
# This says to connect to the Catalyst fcgi server running on localhost, port 777
# FastCgiExternalServer /tmp/myapp.fcgi -host localhost:7777
# Or to use the socket
FastCgiExternalServer /tmp/wormbase.fcgi -socket /tmp/wormbase.sock
# Place the app at root...
Alias / /tmp/wormbase.fcgi/
# ...or somewhere else
Alias /wormbase/ /tmp/wormbase.fcgi/
</IfModule>
# fcgid configuration
# <IfModule mod_fcgid>
# # This should point at your myapp/root
# DocumentRoot /usr/local/wormbase/beta.wormbase.org/root
# Alias /static /usr/local/wormbase/beta.wormbase.org/root/static
# <Location /static>
# SetHandler default-handler
# </Location>
#
# Alias / /usr/local/wormbase/beta.wormbase.org/script/wormbase_fastcgi.pl/
# AddType application/x-httpd-php .php
# <Location />
# Options ExecCGI
# Order allow,deny
# Allow from all
# AddHandler fcgid-script .pl
# </Location>
# </IfModule>
</VirtualHost>
Edit /usr/local/apache2/conf/httpd.conf
Add the appropriate Listen PORT directive.
