Difference between revisions of "Software Life Cycle: 1. Updating The Development Server"

From WormBaseWiki
Jump to navigationJump to search
(New page: This is a quick description of how to update the development server with a new release of the database. == Development Server == The current development server is brie3: brie3.cshl.o...)
 
Line 1: Line 1:
This is a quick description of how to update the development server with a new release of the database.
+
This is a quick description of how to update the development server with a new release of the database.  In general, the development process involves mirroring a large number of files from Sanger, unpacking, and in many cases, massaging into a format suitable for driving the website.
  
== Development Server ==
+
== Development Server ==
  
The current development server is
+
The current development server is  
  
 
   brie3: brie3.cshl.org / dev.wormbase.org
 
   brie3: brie3.cshl.org / dev.wormbase.org
 +
 +
== Update Pipeline Code  ==
  
== Update Pipeline Code ==
+
The update pipeline code is available in the WormBase admin module:
  
The update pipeline code is available in the WormBase admin module:
+
  brie3> svn co svn+ssh://brie3.cshl.org/usr/local/svn/admin
 +
 +
Change into the development directory:  
  
   todd> svn co svn+ssh://brie3.cshl.org/usr/local/svn/admin
+
   brie3> cd admin/update/development
 
+
== Overview of Process ==
+
The contents are:  
 
 
Change into the development directory:
 
 
 
  todd> cd admin/update/development
 
 
 
The contents are:
 
  
 
   logs/      -- the logs directory for each step/update
 
   logs/      -- the logs directory for each step/update
Line 28: Line 26:
 
   updatelog.conf  -- a configuration file for the update process
 
   updatelog.conf  -- a configuration file for the update process
 
   update.sh  -- master script that fires off each step of the pipeline
 
   update.sh  -- master script that fires off each step of the pipeline
   util      -- various helper scripts for the update process
+
   util/       -- various helper scripts for the update process
 +
 +
A single shell script fires off all steps of the process.
  
A single shell script fires off all steps of the process.
+
  brie3> ./update.sh
 +
 +
The steps it performs are:
  
  todd> ./update.sh
+
* Purge disk space
 +
* Create necessary directories
 +
* Mirror and unpack ACeDB from Sanger
 +
* Mirror ontology files from Sanger
 +
* Compile ontology resources for the site
 +
* Create BLAST databases for available species
 +
* Create BLAT database for available species
 +
* Create ePCR databases for select species
 +
* Load genomic GFF databases for available species
 +
* Build and load GFF patches
 +
* Convert GFF3 into GFF3
 +
* Create a GBrowse-driven genetic map
 +
* Create a GBrowse-drive physical map
 +
* Create dump files of common datasets
 +
* Load the CLUSTALW database
 +
* Mirror annotation files from Sanger to the FTP site
  
 
Each step is described below.
 
Each step is described below.
 +
 +
== Update Steps ==
 +
 +
=== Purge Disk Space ===
 +
 +
Remove obsolete files from the (staging) FTP site.  These have already been mirrored to the production FTP site and do not need to be maintained on the development server.
 +
 +
  Usage : ./steps/purge_disk_space.pl
 +
 +
=== Create necessary directories ==
 +
 +
Create staging directories for the update process.
 +
 +
  Usage : ./steps/create_directories.pl {WSRELEASE}
 +
 +
=== Mirror and unpack ACeDB from Sanger ====
 +
 +
Mirror and unpack the new release of the database from Sanger.
 +
 +
  Usage : ./steps/create_directories.pl {WSRELEASE}
 +
 +
''Note:  This can take a *long* time.  You might to run this in a screen:''
 +
 +
  brie3> screen
 +
  brie3> ./steps/mirror_acedb.pl WS198
 +
  brie3> ^D
 +
  (to resume)
 +
  brie3> screen -r 
 +
 +
* Mirror ontology files from Sanger
 +
* Compile ontology resources for the site
 +
* Create BLAST databases for available species
 +
* Create BLAT database for available species
 +
* Create ePCR databases for select species
 +
* Load genomic GFF databases for available species
 +
* Build and load GFF patches
 +
* Convert GFF3 into GFF3
 +
* Create a GBrowse-driven genetic map
 +
* Create a GBrowse-drive physical map
 +
* Create dump files of common datasets
 +
* Load the CLUSTALW database
 +
* Mirror annotation files from Sanger to the FTP site

Revision as of 19:20, 6 January 2009

This is a quick description of how to update the development server with a new release of the database. In general, the development process involves mirroring a large number of files from Sanger, unpacking, and in many cases, massaging into a format suitable for driving the website.

Development Server

The current development server is

 brie3: brie3.cshl.org / dev.wormbase.org

Update Pipeline Code

The update pipeline code is available in the WormBase admin module:

 brie3> svn co svn+ssh://brie3.cshl.org/usr/local/svn/admin

Change into the development directory:

 brie3> cd admin/update/development

The contents are:

  logs/      -- the logs directory for each step/update
  README.txt -- nothing 
  steps/     -- Perl scripts that launch each step
  Update.pm  -- the top level module for the update process
  Update/    -- Perl modules corresponding to each step
  updatelog.conf  -- a configuration file for the update process
  update.sh  -- master script that fires off each step of the pipeline
  util/       -- various helper scripts for the update process

A single shell script fires off all steps of the process.

 brie3> ./update.sh

The steps it performs are:

  • Purge disk space
  • Create necessary directories
  • Mirror and unpack ACeDB from Sanger
  • Mirror ontology files from Sanger
  • Compile ontology resources for the site
  • Create BLAST databases for available species
  • Create BLAT database for available species
  • Create ePCR databases for select species
  • Load genomic GFF databases for available species
  • Build and load GFF patches
  • Convert GFF3 into GFF3
  • Create a GBrowse-driven genetic map
  • Create a GBrowse-drive physical map
  • Create dump files of common datasets
  • Load the CLUSTALW database
  • Mirror annotation files from Sanger to the FTP site

Each step is described below.

Update Steps

Purge Disk Space

Remove obsolete files from the (staging) FTP site. These have already been mirrored to the production FTP site and do not need to be maintained on the development server.

 Usage : ./steps/purge_disk_space.pl 

= Create necessary directories

Create staging directories for the update process.

 Usage : ./steps/create_directories.pl {WSRELEASE}

Mirror and unpack ACeDB from Sanger =

Mirror and unpack the new release of the database from Sanger.

 Usage : ./steps/create_directories.pl {WSRELEASE}

Note: This can take a *long* time. You might to run this in a screen:

 brie3> screen
 brie3> ./steps/mirror_acedb.pl WS198
 brie3> ^D
 (to resume)
 brie3> screen -r   
  • Mirror ontology files from Sanger
  • Compile ontology resources for the site
  • Create BLAST databases for available species
  • Create BLAT database for available species
  • Create ePCR databases for select species
  • Load genomic GFF databases for available species
  • Build and load GFF patches
  • Convert GFF3 into GFF3
  • Create a GBrowse-driven genetic map
  • Create a GBrowse-drive physical map
  • Create dump files of common datasets
  • Load the CLUSTALW database
  • Mirror annotation files from Sanger to the FTP site