Difference between revisions of "Software Life Cycle: 1. Updating The Development Server"
Line 97: | Line 97: | ||
Usage : ./steps/create_directories.pl ${RELEASE} | Usage : ./steps/create_directories.pl ${RELEASE} | ||
Input : Files mirrored from Sanger to ${ACEDB}/tmp | Input : Files mirrored from Sanger to ${ACEDB}/tmp | ||
− | Output : Unpacked AceDB files in ${ACEDB}/ | + | Output : Unpacked AceDB files in ${ACEDB}/wormbase_${RELEASE} |
''Note: This can take a *long* time. You might to run this in a screen:'' | ''Note: This can take a *long* time. You might to run this in a screen:'' | ||
Line 112: | Line 112: | ||
When complete, check to make sure that the following directory and symlink exist: | When complete, check to make sure that the following directory and symlink exist: | ||
− | brie3> ${ACEDB}/ | + | brie3> ${ACEDB}/wormbase -> wormbase_${RELEASE} |
Trouble shooting notes: | Trouble shooting notes: |
Revision as of 13:46, 20 July 2009
This is a quick description of how to update the development server with a new release of the database. In general, the development process involves mirroring a large number of files from Sanger, unpacking, and in many cases, massaging into a format suitable for driving the website.
Contents
- 1 Development Server
- 2 Update Pipeline Code
- 3 Check logs and resolve issues
- 4 Update Steps
- 4.1 Purge Disk Space
- 4.2 Create necessary directories
- 4.3 Mirror and unpack ACeDB from Sanger
- 4.4 Mirror ontology from Sanger
- 4.5 Compile ontology resources for the site
- 4.6 Create BLAST databases for available species
- 4.7 Create BLAT databases for available species
- 4.8 Create ePCR databases for available species
- 4.9 Load genomic GFF DBs for available species
- 4.10 Build and Load GFF patches
- 4.11 Convert GFF2 into GFF3
- 4.12 Create a GBrowse-driven genetic map
- 4.13 Create a GBrowse-driven physical map
- 4.14 Create dump files of common datasets
- 4.15 Load the CLUSTALW database
- 4.16 Mirror annotation files from Sanger to the FTP site
Development Server
The current development server is
wb-dev: wb-dev.oicr.on.ca / dev.wormbase.org
When indicated, substitute WSXXX for ${RELEASE}.
System paths referred to in this document:
FTP : ~ftp/pub/wormbase (on brie3 this is /usr/local/) WORMBASE : /usr/local/wormbase ACEDB : /usr/local/wormbase/acedb
Update Pipeline Code
The update pipeline code is available in the WormBase admin module:
> svn co svn+ssh://brie3.cshl.org/usr/local/svn/admin
Change into the development directory:
> cd admin/update/development
The contents are:
logs/ -- the logs directory for each step/update README.txt -- nothing steps/ -- Perl scripts that launch each step Update.pm -- the top level module for the update process Update/ -- Perl modules corresponding to each step updatelog.conf -- a configuration file for the update process update.sh -- master script that fires off each step of the pipeline util/ -- various helper scripts for the update process
A single shell script fires off all steps of the process.
> ./update.sh
However, is best to run each step individually; periodically checking the log to monitor progress.
> ./steps/<individual_update_script>.pl
The steps to perform are:
- Purge disk space
- Create necessary directories
- Mirror and unpack ACeDB from Sanger
- Mirror ontology files from Sanger
- Compile ontology resources for the site
- Create BLAST databases for available species
- Create BLAT database for available species
- Create ePCR databases for select species
- Load genomic GFF databases for available species
- Build and load GFF patches
- Convert GFF3 into GFF3
- Create a GBrowse-driven genetic map
- Create a GBrowse-drive physical map
- Create dump files of common datasets
- Mirror annotation files from Sanger to the FTP site
Each step is described below.
Check logs and resolve issues
Log files are under admin/development/logs and the logging levels are set to record ERRORS and WARNINGS
NB: While logs provide excellent records of the loading process. It is important to check proper functioning of the web pages that use the data; links to which are provided with the description of each step.
Update Steps
Purge Disk Space
Remove obsolete files from the (staging) FTP site on the development site. These have already been mirrored to the production FTP site and do not need to be maintained on the development server.
Usage : ./steps/purge_disk_space.pl
Create necessary directories
Create staging directories for the update process.
Usage : ./steps/create_directories.pl ${RELEASE} Output : Directories in ${WORMBASE}/databases (primarily)
Mirror and unpack ACeDB from Sanger
Mirror and unpack the new release of the database from Sanger. Add in appropriate control files for the new acedb database: serverpasswrd.wrm, passwrd.wrm, serverconfig.wrm pulled from the checked out development source (/usr/local/wormbase/wspec).
Files will be mirrored and unpacked to /usr/local. Please make sure that there is sufficient space in this directory! You will most likely need approximately 25 GB of disk space. Possible places to free up disk space:
/usr/local/mysq/data /usr/local/acedb/tmp ~{you}/mp3s
Usage : ./steps/create_directories.pl ${RELEASE} Input : Files mirrored from Sanger to ${ACEDB}/tmp Output : Unpacked AceDB files in ${ACEDB}/wormbase_${RELEASE}
Note: This can take a *long* time. You might to run this in a screen:
brie3> screen brie3> ./steps/mirror_acedb.pl WS198 (to disconnect your screen) brie3> ^a ^d (to resume your screen) brie3> screen -r screen command reference
When complete, check to make sure that the following directory and symlink exist:
brie3> ${ACEDB}/wormbase -> wormbase_${RELEASE}
Trouble shooting notes:
- gzip version incompatibilies may break the step.
Mirror ontology from Sanger
Mirror OBO files from Sanger. These are necessary for the ontology searches.
Usage : ./steps/mirror_ontology_files.pl ${RELEASE} Input : none Output : Files mirrored to ${WORMBASE}/databases/${RELEASE}/ontology
Compile ontology resources for the site
Take the mirrored files and compile them into the databases for the ontology searches.
Usage : ./steps/compile_ontology_resources.pl ${RELEASE} Input : OBO files mirrored earlier in ${WORMBASE}/databases/${RELEASE}/ontology Output : to ${WORMBASE}/database/${RELEASE}/ontology:
*anatomy_association.RELEASE.wb *gene_association.RELEASE.wb.ce *gene_ontology.RELEASE.obo *name2id.txt *search_data.txt *anatomy_ontology.RELEASE.obo *gene_association.RELEASE.wb.cjp *id2association_counts.txt *parent2ids.txt *gene_association.RELEASE.wb *gene_association.RELEASE.wb.ppa *id2name.txt *phenotype_association.RELEASE.wb *gene_association.RELEASE.wb.cb *gene_association.RELEASE.wb.rem *id2parents.txt *phenotype_ontology.RELEASE.obo
Create BLAST databases for available species
Build BLAST databases for available species. For some species, this includes databases for genes, ests, proteins, and genomic sequence. For others, only genomic sequence and protein databases are constructed.
Usage : ./steps/create_blast_databases.pl ${RELEASE} Input : Genomic sequence and protein FASTA files mirrored from Sanger to ${FTP}/genomes/${SPECIES}/sequences/dna/${SPECIES}.dna.fa.gz; Gene and EST sequences derived from AceDB Output : BLAST databases in ${WORMBASE}/databases/${RELEASE}/blast
Create BLAT databases for available species
Build BLAT databases of genomic sequence for each available species.
Usage : ./steps/create_blat_databases.pl ${RELEASE} Input : Genomic sequence FASTA files mirrored from Sanger to ${FTP}/genomes/${SPECIES}/sequences/dna/${SPECIES}.dna.fa.gz Output : BLAT .nib files in ${WORMBASE}/databases/${RELEASE}/blat
Create ePCR databases for available species
Build ePCR databases for each species.
Usage : ./steps/create_epcr_databases.pl ${RELEASE} Input : Mirrored genomic sequence files from Sanger to ${FTP}/genomes/${SPECIES}/sequences/dna/${SPECIES}.dna.fa.gz Output : ePCR databases to ${WORMBASE}/databases/${RELEASE}/epcr
Load genomic GFF DBs for available species
Get genomic gff files from Sanger and load into the DBs
Usage : ./steps/load_genomic_gffdb.pl ${RELEASE} Input : GFF and FASTA files mirrored from Sanger to GFF : ${FTP}/${SPECIES}/genome_feature_tables/GFF2/${SPECIES}.${VERSION}.gff.gz DNA : ${FTP}/${SPECIES}/sequences/dna/${SPECIES}.${VERSION}.dna.fa.gz Output: (This script both creates/mirrors and uses the files above).
Trouble shooting notes:
- File and directory names need to be consistent with format specified in Update.pm circa line 36
- If necessary, e.g. files were incorrectly named, they should be manually downloaded from the source site, uncompressed and renamed correctly
- Progress can be monitored by checking the log file ..admin/development/logs/{WSRELEASE}/load genomic feature gff databases.log and the building of the mysql db files in /usr/local/mysql/data
Build and Load GFF patches
Create and load number of patches for the c_elegans GFF database, including protein motifs and genetic limits.
Usage : ./steps/load_gff_patches.pl ${RELEASE} Input : Files created to ${FTP}/genomes/c_elegans/genome_feature_tables/GFF2 Output : Files created above.
Convert GFF2 into GFF3
Notes...
Usage: ./steps/convert_gff2_to_gff3.pl ${RELEASE}
Create a GBrowse-driven genetic map
Notes...
Usage: ./steps/load_gmap_gffdb.pl ${RELEASE}
Create a GBrowse-driven physical map
Notes...
Usage: ./steps/load_pmap_gffdb.pl {WSRELEASE}
Create dump files of common datasets
Notes...
Load the CLUSTALW database
Notes...
Usage: ./steps/load_clustal_db.pl {WSRELEASE}
Mirror annotation files from Sanger to the FTP site
Notes...
Usage: ./steps/mirror_annotations.pl {WSRELEASE}
- Convert GFF2 into GFF3
- Create a GBrowse-driven genetic map
- Create a GBrowse-driven physical map
- Create dump files of common datasets
- Load the CLUSTALW database
- Mirror annotation files from Sanger to the FTP site