WormBase Infrastructure

From WormBaseWiki
Revision as of 19:06, 17 May 2008 by Tharris (talk | contribs)
Jump to navigationJump to search

The WormBase Infrastructure

This document describes the WormBase infrastructure including hardware and software configuration.

Document conventions

Commands that require super-user privileges are prefaced with "sudo" or a "$" prompt. If the system is configured correctly, you should not need to be a root user in order to update the site.

*** Potential stumbling blocks are indented and hilighted with a
 *** preceeding triple asterisk.  Yikes!

General network topology

The network topology of the WormBase site looks like this:

                      |                                (Non-accelerated / non-cached reqeuests)
                      |                              |             |             |             |
                      |                              |             |             |             |
            |---------|---------|                    |             |             |             |
            |         |         |                    |             |             |             |
            |         |         |                    |             |             |             |
            |        \/         |                    |             |             |             |
            |       squid       |                    |             |             |             |
            |         |         | www (fe)           |             |             |             |
            |        \/         |                    |             |             |             |
            |   load balancer   |                    |             |             |             |
            |         |         |                    |             |             |             |
            |---------|---------|                    |             |             |             |
                      |                              |             |             |             |
                      |                              |             |             |             |
       |------------------------------|              |             |             |             |
       |              |               |              \/            \/            \/            \/
 |-----------|  |-----------|  |-----------|    |-----------| |-----------| |-----------| |-----------|
 |     |     |  |     |     |  |           |    |           | |           | |           | |           |
 |   httpd   |  |   httpd   |  |   httpd   |    |   httpd   | | httpd/ace | |  biomart  | |  future   |
 |    unc    |  |    vab    |  |  crestone |    |   blast   | | aceserver | |           | |           |
 -------------  -------------  -------------    ------------- ------------- ------------- -------------

The WormBase infrastructure includes:

  • a multiply redundant pool of servers running identical copies of the database and website
  • a caching/load balancing server that distributes requests to the server pool
  • agressive object and page caching across the site

All servers reside outside of the firewall but within the DMZ. All machines within the DMZ can communicate with each other on any port. Outside access requires specific firewall exceptions for the desired port.

UPDATE 2/2006

Initially, all back end servers were forced to respond to direct requests in order to server dynamically generated images from specific pages. In other words, whichever server generates the initial image must also server that image. This required that all back end servers have a firewall exception for port 80. A secondary consequence was that clients could inadvertently access back end servers without passing through the reverse proxy.

In 2/2006 I rewrote the paths of dynamically generated images to include the name of the back end machine. A simple squid redirector then directs these queries to the proper back end host based on the structure of the URL. In addition, the back end servers gene, vab, and unc no longer need to be exposed to the world directly; all listen on port 8080 preventing bypass of the squid proxy. Port 80 is automatically redirected to www.wormbase.org.


primary name aliases/other FQDNs tasks ports* mysqld server ID
fe.wormbase.org www.wormbase.org
  • primary front end Squid caching server
  • balances load to back end servers

brie6.cshl.edu, stein.cshl.edu

  • serves up most AceDB-based pages at WormBase
  • hosts wormbase and wormbook mailing lists
  • hosts WormBook (not accelerated by fe.wormbase.org)
  • hosts steinlab.cshl.edu (not accelerated by fe.wormbase.org)
22, 80 (for WormBook and Stein), 8080 for WormBase
  • serves Genome Browser pages for WormBase
none. httpd listens on 8080.
gene.wormbase.org none. httpd listens on 8080.
  • provides blast and blat searches for WormBase
22, 80
  • provides support for AQL, WB query language, and cisortho
80, 2005, 3306
  • hosts the WormBase implementation of BioMart


  • primary WormBase development machine
  • hosts the WormBase FTP site
21, 22, 80
  • secondary WormBase development machine
  • hosts the WormBaseWiki

'* Firewall exceptions in boldface'

Production nodes

"Production nodes" or "live servers" are the servers that provide all the functionality at www.wormbase.org.


This server acts as a proxy for the entire WormBase site fielding requests for www.wormbase.org on port 80 using the reverse-proxy caching server "squid". Each request is filtered through a simple load balancing script which rewrites the URL and sends the request to an appropriate back end server. Returned results are cached in memory and/or on disk before being returned to the client to accelerate subsequent requests. Internally, this server is also known as fe.wormbase.org (for "front end"). fe.wormbase.org is configured as a redundant RAID in case of catastrophic disk failure.

fe.wormbase.org currently redirects the following requests (evaluated in this order):

URL host
/.*squid\/cachemgr\.cgi/ localhost
/.*\/gbrowse\/.*/ vab
/cisortho/ aceserver
/wiki/ crestone
* unc (brie6)

unc.wormbase.org (brie6)

Formerly the main www.wormbase.org server, unc.wormbase.org is a backend origin server running the entire WormBase site. It also serves the mailing lists archives, the RSS feeds, and access log statistics. unc also hosts a number of virtual hosts that do not pass through the proxy server on fe. Port 80 for unc.wormbase.org is open in the firewall.


vab answers requests on port 80 from fe.wormbase.org with httpd. Currently, vab.wormbase.org is only serving requests to the Genome Browser although it is capable of serving any request at WormBased. It hosts no virtual hosts. Port 80 for vab.wormbase.org is open in the firewall.


aceserver is the publically accessible data mining server. Requests to aceserver.cshl.org do not pass through the proxy server on fe. Ports 80, 3360 (mysql), and 2005 (acedb) are open for aceserver.cshl.org.

aceserver.cshl.org also handles



BLAST and BLAT requests are handled by blast.wormbase.org. Requests for blast analyses at blast.wormbase.org do not pass through the proxy server on fe. blast.wormbase.org is also capable of handling requests load balancing requests but the redirector on fe.wormbase.org is not currently configured for this. Port 80 for blast.wormbase.org is open in the firewall.


biomart.wormbase.org serves WormMart. Requests to biomart.wormbase.org do not pass through the proxy server on fe.

Development nodes

WormBase also maintains two development nodes.

dev.wormbase.org (brie3)

dev.wormbase.org is the current development server at WormBase. It maintains a CVS-checked out version of the source and database tarballs that is used to automatically push software and new databases onto the production nodes and mirror servers (described below). dev.wormbase.org also hosts the WormBase FTP site, aliased to ftp.wormbase.org.


crestone hosts the WormBase rearchitecture. Please don't junk it up with files -- disk space is at a premium!

Frozen release servers

freeze1 -- https://freeze1.wormbase.org:8333/


freeze2 -- https://freeze2.wormbase.org:8333/


brie6 -- https://brie6.cshl.edu:8333/


be1.wormbase.org -- https://be1:8333/

 ws180.wormbase.org -- this is just a cname entry pointing to www.wormbase.org.
      proxy resolves and points to the private back end virtual machine:

Typical request flow / infrastructure rationale

The WormBase infrastructure and design decisions are best understood by following requests through the system.

Static pages

First, consider a request for the home page, index.html. This request is recieved at www.wormbase.org by the squid server running on port 80. If this is the first request for the home page (unlikely!), squid passes the request to an instance of the load balancer script. This script rewrites the URL and sends it to one of the back end servers, unc or vab. The back end server returns the page to fe.wormbase.org which caches it in memory and on disk and returns it to the user. Subsequent requests are served directly from the cache, bypassing the back end servers altogether.

Dynamic pages

Consider a request for a typical gene page, unc-2. This request is recieved at www.wormbase.org by the squid server. Assuming that this is the first access of the unc-2 gene page during this release cycle, the load balancer sends the request to either unc.wormbase.org or vab.wormbase.org. The returned page is cached by the squid server and returned to the client.

It is important to note that the Gene page contains dynamically generate images. These images are now generated using absolute URLs. Thus, when a cached gene page is loaded, the client can correctly request the image from the appropriate back end server. For this to work correctly, the back end servers *must* be registered with DNS and have port 80 open in the firewall.

Special cases

  • POST requests

Because squid uses URLs as identifiers in the cache, any page view that results from a POST action will not be cached.

  • GBrowse (/db/seq/gbrowse)

Requests for Gbrowse (/db/seq/gbrowse) are not cached. Images created by gbrowse_img (such as those embedded on the Gene Summary and Clone pages) *are* cached as these can be considerably more "static" than individual GBrowse views. See the section "Access Control Lists" for additional details. Currently, requests for Gbrowse are handled exclusively by vab.wormbase.org.

  • Mailing list archives (/mailarch/)
  • RSS feeds (/rss/* and /private/manage_newsfeeds)
  • Access log statistics (/stats/)
  • The Feedback form (/db/misc/feedback)

These items are all handled directly by unc.wormbase.org (redirection is handled at the level of the redirector on fe.wormbase.org).

  • The Cache Manager CGI

The Cache Manager CGI is only available from fe.wormbase.org:81. Note you must be "on localhost" in order to use this statistics viewer.