Difference between revisions of "Software Life Cycle: 1. Updating The Development Server"
Line 67: | Line 67: | ||
Usage : ./steps/create_directories.pl {WSRELEASE} | Usage : ./steps/create_directories.pl {WSRELEASE} | ||
− | === Mirror and unpack ACeDB from Sanger | + | === Mirror and unpack ACeDB from Sanger === |
Mirror and unpack the new release of the database from Sanger. Add in appropriate control files for the new acedb database: serverpasswrd.wrm, passwrd.wrm, serverconfig.wrm pulled from the checked out development source (/usr/local/wormbase/wspec). | Mirror and unpack the new release of the database from Sanger. Add in appropriate control files for the new acedb database: serverpasswrd.wrm, passwrd.wrm, serverconfig.wrm pulled from the checked out development source (/usr/local/wormbase/wspec). | ||
Line 92: | Line 92: | ||
brie3> /usr/local/acedb/elegans -> elegans_{WSRELEASE} | brie3> /usr/local/acedb/elegans -> elegans_{WSRELEASE} | ||
− | + | === Mirror ontology from Sanger === | |
− | + | ||
+ | Mirror OBO files from Sanger. | ||
+ | |||
+ | Usage: ./steps/mirror_ontology_files.pl {WSRELEASE} | ||
+ | |||
+ | === Compile ontology resources for the site === | ||
+ | |||
+ | Take the mirrored files and compile them into the databases for the ontology searches. | ||
+ | |||
+ | Usage: ./steps/compile_ontology_resources.pl {WSRELEASE} | ||
+ | |||
+ | |||
+ | |||
*Create BLAST databases for available species | *Create BLAST databases for available species | ||
*Create BLAT database for available species | *Create BLAT database for available species |
Revision as of 20:26, 6 January 2009
This is a quick description of how to update the development server with a new release of the database. In general, the development process involves mirroring a large number of files from Sanger, unpacking, and in many cases, massaging into a format suitable for driving the website.
Contents
Development Server
The current development server is
brie3: brie3.cshl.org / dev.wormbase.org
Update Pipeline Code
The update pipeline code is available in the WormBase admin module:
brie3> svn co svn+ssh://brie3.cshl.org/usr/local/svn/admin
Change into the development directory:
brie3> cd admin/update/development
The contents are:
logs/ -- the logs directory for each step/update README.txt -- nothing steps/ -- Perl scripts that launch each step Update.pm -- the top level module for the update process Update/ -- Perl modules corresponding to each step updatelog.conf -- a configuration file for the update process update.sh -- master script that fires off each step of the pipeline util/ -- various helper scripts for the update process
A single shell script fires off all steps of the process.
brie3> ./update.sh
The steps it performs are:
- Purge disk space
- Create necessary directories
- Mirror and unpack ACeDB from Sanger
- Mirror ontology files from Sanger
- Compile ontology resources for the site
- Create BLAST databases for available species
- Create BLAT database for available species
- Create ePCR databases for select species
- Load genomic GFF databases for available species
- Build and load GFF patches
- Convert GFF3 into GFF3
- Create a GBrowse-driven genetic map
- Create a GBrowse-drive physical map
- Create dump files of common datasets
- Load the CLUSTALW database
- Mirror annotation files from Sanger to the FTP site
Each step is described below.
Update Steps
Purge Disk Space
Remove obsolete files from the (staging) FTP site. These have already been mirrored to the production FTP site and do not need to be maintained on the development server.
Usage : ./steps/purge_disk_space.pl
Create necessary directories
Create staging directories for the update process.
Usage : ./steps/create_directories.pl {WSRELEASE}
Mirror and unpack ACeDB from Sanger
Mirror and unpack the new release of the database from Sanger. Add in appropriate control files for the new acedb database: serverpasswrd.wrm, passwrd.wrm, serverconfig.wrm pulled from the checked out development source (/usr/local/wormbase/wspec).
Files will be mirrored and unpacked to /usr/local. Please make sure that there is sufficient space in this directory! You will most likely need approximately 25 GB of disk space. Possible places to free up disk space:
/usr/local/mysq/data /usr/local/acedb/tmp ~{you}/mp3s
Usage : ./steps/create_directories.pl {WSRELEASE}
Note: This can take a *long* time. You might to run this in a screen:
brie3> screen brie3> ./steps/mirror_acedb.pl WS198 (to disconnect your screen) brie3> ^a ^d (to resume your screen) brie3> screen -r
When complete, check to make sure that the following directory and symlink exist:
brie3> /usr/local/acedb/elegans -> elegans_{WSRELEASE}
Mirror ontology from Sanger
Mirror OBO files from Sanger.
Usage: ./steps/mirror_ontology_files.pl {WSRELEASE}
Compile ontology resources for the site
Take the mirrored files and compile them into the databases for the ontology searches.
Usage: ./steps/compile_ontology_resources.pl {WSRELEASE}
- Create BLAST databases for available species
- Create BLAT database for available species
- Create ePCR databases for select species
- Load genomic GFF databases for available species
- Build and load GFF patches
- Convert GFF3 into GFF3
- Create a GBrowse-driven genetic map
- Create a GBrowse-drive physical map
- Create dump files of common datasets
- Load the CLUSTALW database
- Mirror annotation files from Sanger to the FTP site