Difference between revisions of "Developer documentation"
From WormBaseWiki
Jump to navigationJump to search| (200 intermediate revisions by 10 users not shown) | |||
| Line 1: | Line 1: | ||
This page contains information pertinent to employees of WormBase. | This page contains information pertinent to employees of WormBase. | ||
| − | = | + | = Navigating and Editing the Developer Documentation = |
| − | + | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | == [[:Category:Web Dev| Web Dev Category Hierarchy (Under Construction)]] == | |
| − | |||
| − | |||
| − | + | '''You will be able to navigate the Developer Documentation by browsing through the [[:Category:Web Dev| Web Dev Category Hierarchy (Under Construction)]] above, which is under construction right now. In the mean time, please browse through this page or search for information that you need.''' | |
| − | === | + | <categorytree mode=pages showcount="on" hideroot="on" depth=4 >Web Dev</categorytree> |
| − | |||
| − | |||
| − | '' | + | == How to Contribute to the Documentation == |
| − | + | If you'd like to help out adding to the documentation, that's great! Here is some general good practice that will help keeping the documentation organized, navigable and up to date. | |
| − | + | * Ensure a page is tagged with a category, when you add or edit a page. You can browse through our [[:Category:Web Dev| high level categories]] and locate the '''most specific category''' that describes your page. | |
| − | + | <blockquote> For example, include the following code at the bottom of a page will make the page appear under the category [[:Category: Getting Started (Web Dev)]] : | |
| − | * | ||
| − | |||
| − | |||
| − | |||
| − | + | <code><nowiki> | |
| − | + | [[Category: Getting Started (Web Dev)]] | |
| − | + | </nowiki></code> | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | </blockquote> | |
| − | |||
| − | |||
| − | ''' | + | * '''Update''', deprecate or remove old documentation when appropriate. |
| − | |||
| − | + | * Any documents that contain potentially '''sensitive information''' belong on Google Docs (Contact Todd for details). This might reasonably include anything that reveals information about '''our systems or architecture'''. | |
| − | + | * Additional guidelines on [[Documentation and Communication Standards]] | |
| − | ''' | + | * '''Other ideas?''' Please update the wiki here! |
| − | |||
| − | |||
| − | |||
| − | '' | + | = New Employee Orientation = |
| − | * | + | * [https://docs.google.com/document/d/1E37IgK1gsaw6ZXa3cYFz2HPVPuB__KmNs5tErO6VXDY/pub New Employee Orientation] -- ''Getting started at WormBase, for employees at any location. Read this first.'' |
| − | + | * [https://docs.google.com/document/d/1Xm1aQTm7zXapfirgD1PP7KWNjPlXHNw_iaLLAVvZ-Gg/pub Webdev team/OICR Orientation] -- ''For new employees of the web development team at OICR.'' | |
| − | + | *[[Co-op Documentation]] | |
| − | + | = Project Management = | |
| − | + | *[[Project Calendar|Project Calendar (Google)]] | |
| − | |||
| − | |||
| − | + | ==== Conference call agendas/minutes ==== | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | *[[Conference calls| Conference call agendas and minutes (All WormBase)]] | |
| − | * | + | *[[Conference call agendas and minutes| Conference call agendas and minutes (WebDev team)]] |
| − | * | + | *[[Project meeting agendas and minutes]] |
| − | * | + | *[[InterMine Strategy Meetings]] |
| − | + | ==== Source Code Repositories ==== | |
| − | |||
| − | |||
| − | |||
| − | '' | + | ''We use the distributed source code management system git. Projects are hosted on [http://github.org/WormBase Github].'' |
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | *[http://github.org/WormBase/website website] - ''The WormBase web application (public)'' | |
| − | + | *[http://github.org/WormBase/website-admin website-admin] - ''Administrative code (private)'' | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Recommended reading (learning git): | |
| − | *[ | + | * [http://try.github.io/levels/1/challenges/1 try.gihub.io] |
| − | |||
| − | |||
| − | |||
| − | = | + | = General Documentation = |
| + | *[[Co-op Documentation]] | ||
| + | *[[SOPs|Internal documentation and standard operating procedures]] | ||
| + | *[[Feature WishList]] | ||
| − | = | + | = The Web Application = |
| − | * [[Website:Application overview|Application overview]] - '' | + | * [[Website:Application overview|Application overview]] - ''An overview of the web application, including MVC structure, request flow, caching, etc'' |
* [[Website:Configuration|Configuration]] -- ''Application and view configuration: formatting and standards'' | * [[Website:Configuration|Configuration]] -- ''Application and view configuration: formatting and standards'' | ||
* [[Website:WormBase::API|WormBase::API]] -- ''The WormBase data model, maintained external to the web application'' | * [[Website:WormBase::API|WormBase::API]] -- ''The WormBase data model, maintained external to the web application'' | ||
| Line 121: | Line 68: | ||
* [[Website:Model| Model]] -- ''The web application model; thin, glued to the external Model'' | * [[Website:Model| Model]] -- ''The web application model; thin, glued to the external Model'' | ||
* [[Website:Controller| Controller]] -- ''Overview of controller files and actions'' | * [[Website:Controller| Controller]] -- ''Overview of controller files and actions'' | ||
| − | * [[Website: | + | * Genome Browsing |
| − | *[[ | + | ** [[Website:GBrowse Set-up on dev.wormbase.org|GBrowse Set-up on dev.wormbase.org]] -- "Updating/rolling-back GBrowse installations on a per user basis" |
| + | ** [[Website:Genome Browser Survey|Genome Browser Survey]] -- "Community survey for evaluating use cases and required functionality of a WormBase genome browser" | ||
| + | * Visualization | ||
| + | ** [[Website:Visualization Library Assessment]] -- "Assessment of several visualization options for integration in the Website" | ||
| + | ** [[Website:Visualization with ggplot2 of the R-Project]] -- "Installation, set-up, and other information about the R-project based visualization" | ||
| − | + | * [[Website:WormMine| WormMine]] -- ''How to install, update, and maintain the InterMine instance at WormBase'' | |
| + | ** [[Website:WormMine Builds]] -- Summary stats of WormMine builds | ||
| + | *[[Website:Contribute_from_GitHub|Contribute from Github]] | ||
| − | *[[ | + | == The Mobile Website == |
| − | *[[ | + | * [[Website:Mobile| Mobile]] -- ''Overview of the mobile website'' |
| − | *[[ | + | * [[Website:MobileRouter| Router]] -- ''A definition of the possible routes of the mobile website'' |
| + | * [[Website:MobileModels| Models]] -- ''Details about the models'' | ||
| + | * [[Website:MobileCollections| Collections]] -- ''Details about the collections'' | ||
| + | * [[Website:MobileViews| Views]] -- ''Details about the views'' | ||
| + | * [[Website:MobileWidgets| Widgets Porting]] -- ''Details about the porting of widgets in the mobile website/app'' | ||
| + | = Data Model and Data Development = | ||
| − | == | + | *[[WormBase Models]] |
| + | *[[Genome Standards]] -- ''What criteria should a genome meet before we accept it for integration?'' | ||
| + | *[[Suggested pre-release data checks|Suggested pre-release data checks]] | ||
| + | *[[modENCODE Analysis & metadata discussion|modENCODE Analysis & metadata discussion]] | ||
| + | *[[modENCODE Integration status|modENCODE Integration status]] | ||
| + | *[[Genome sequence changes]] | ||
| + | *[[WormBase Genomes]] | ||
| + | *[[GFF Release Data and Changes]] | ||
| + | |||
| + | = Development Guidelines = | ||
| + | * [https://docs.google.com/a/wormbase.org/document/d/1jps4_bTrsR-Km0QTKk3ENRckbKyK3y8jZwSvq3rh2lo/pub Setting up your development environment] -- ''How to set up your development environment on dev.wormbase.org'' | ||
| + | *[https://docs.google.com/a/wormbase.org/document/d/1ONNKWe82OL5oZoPBmQ3nOZ_EZuzDfjbVpYvMkjnfULc/edit#heading=h.3t2png3rdgj8 Developer Resources at WormBase] -- ''Available developer resources, including the shared development space'' | ||
| + | *[https://docs.google.com/document/d/1zIOm95tV9A14n9xnkPkMgiOTGcOIzx161y9UbDZvfW4/pub WormBase Release Workflow] -- ''Project-wide schedule that documents what is happening when and where. ([[WormBase Release Workflow|original wiki version)]]'' | ||
| + | *[[Development_workflow_-_webdev|Formal Development Strategy]] -- ''An overview of the development process at WormBase'' | ||
| + | *[[Documentation and Communication Standards|Documentation and Communication Standards]] | ||
| + | *[[Coding standards|Coding standards]] | ||
| + | *[[Developer_notes|Developer notes]] -- ''Best practices, conventions, etc'' | ||
| + | *[[Browser Support Statement]] -- ''official graded browser support statement'' | ||
| − | === | + | == Testing == |
| − | + | *[[Unit Testing]] | |
| − | |||
| − | * [[ | ||
| − | == | + | == Help us maintain the site == |
| − | + | *[[Website:Contribute from GitHub|Contribute from GitHub]] | |
| − | * [[Website: | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | = Information for third party developers = | |
| + | *[[Linking To WormBase|Linking To WormBase]] | ||
| + | = Administration = | ||
| + | == Architecture == | ||
| + | '''Software Platform''' | ||
| + | *[[Administration:Installing WormBase|Installing WormBase]] - ''How to get a basic installation of WormBase up and running'' | ||
| + | *[[Administration:Installing WormMine|Installing WormMine]] - ''How to get a basic installation of WormMine up and running'' | ||
| + | *[[Administration:WormBase Production Environment|Production Environment]] - ''Building and Maintaining the WormBase production environment; extends the "Installing WormBase" document'' | ||
| + | * [[GBrowse Administration]] -- ''building, configuring, and maintaining production GBrowse nodes.'' | ||
* [[Building WormMart|Building WormMart]] - ''How to convert a WormBase AceDB database into WormMart'' | * [[Building WormMart|Building WormMart]] - ''How to convert a WormBase AceDB database into WormMart'' | ||
| − | + | '''Hardware Platform''' | |
| − | *[[ | + | *[[WormBase_Infrastructure|Hardware Platform]] - ''A complete accounting of WormBase compute resources. |
| − | == | + | == Software Life Cycle == |
| − | *[ | + | * [https://docs.google.com/a/wormbase.org/document/d/1oPpj8d5gibUc-gpUZorl6ETT5baE6mp-v2bMedKauiA/ Software Life Cycle: 1. Build] -- ''Preparing a new release for the website'' |
| − | * [[ | + | * [[Software Life Cycle: 2. Software development period|Software development period]] - ''Once a new database has been staged, the development phase begins in earnest on private development sites'' |
| − | *[[ | + | * [[Software Life Cycle: 3. QA/QC and Performance Benchmarking|QA/QC and Performance Benchmarking]] - ''Integrating changes and staging the code for QA/QC and performance testing prior to deployment'' |
| − | *[ | + | * [https://docs.google.com/a/wormbase.org/document/d/1KlweWzfxc9-uKxR1dJwuPfR4NPJs-H1S8Yxg_V1Ha8U/edit#heading=h.e2m5ps3gnzcs Software Life Cycle: 4. Production] -- ''Deploying a new release to production'' |
| − | === | + | == Measuring Application Performance == |
| − | |||
| − | + | * [[Website:Testing_And_Profiling|Unit Testing and Profiling]] - ''Testing and profiling protocol for the web application'' | |
| + | * System Testing | ||
| + | * [[Website:Validation|Standards Validation]] - ''Validation of HTML, CSS, XML'' | ||
| + | * Security Assessment | ||
| + | * Accessibility | ||
| + | * [[Website:Usability Testing|Usability Testing]] - ''usability testing strategies and evaluation'' | ||
| + | * [[Website:Browser Compatibility Statement|Browser Compatibility Testing]] - ''browser support matrix and testing protocol'' | ||
| + | * [[Website:Load Testing|Load Testing]] - ''Performance and load testing; historical benchmarks'' | ||
| + | ---- | ||
| − | '' | + | == Routine Administration Tasks == |
| + | * [[Cronjobs|Cronjobs]] - ''a full accounting of all cronjobs at WB'' | ||
| + | * [[Creating_a_Google_Sitemap|Creating a Google Sitemap]] - ''how to tell Google which URLs to index'' | ||
| + | * [[Managing Perl Libraries]] - ''Managing Perl libraries across projects and architectures'' | ||
| + | * [[Log file management and analysis|Log file management and analysis]] - ''rotating and analyzing logs.'' | ||
| + | * [[Site_statistics|Site statistics]] | ||
| − | [[ | + | == Monitoring and Crises Management == |
| + | *[https://docs.google.com/a/wormbase.org/document/d/18oRqTtC-q1P1bPyLAjo4XkqMGzepPi5Be1w7CfLuzg8/edit?hl=en_US Crises Recovery Plan] - ''How to diagnose and recover from service interruptions.'' | ||
| + | *[https://docs.google.com/a/wormbase.org/document/d/18G-lemZ0zKXraQlzXqNmsUCWwZmjuUHIipF6uRd0R0k/edit Monitoring services and servers] - ''Keeping tabs on the production architecture'' | ||
| − | *[ | + | == Frozen Releases == |
| − | + | *[[Building Frozen Releases]] - ''The most current documentation on how to build a frozen release'' | |
| − | + | = Deprecated Documentation = | |
''The good word gone bad. Cruft. Lint. Maybe useful. Probably not.'' | ''The good word gone bad. Cruft. Lint. Maybe useful. Probably not.'' | ||
| + | |||
| + | ==== Rearchitecture ==== | ||
| + | *[[Website:Development Milestones|Development Milestones]] | ||
| + | *[[Website:Design Specifications|Requirements analysis / Design Specifications]] | ||
| + | *[[Website:Rearchitecture Development Duties|Rearchitecture Development Duties]] | ||
| + | * AceDB classes -> API progress matrix (index of classes; see [https://spreadsheets.google.com/a/wormbase.org/ccc?key=tygSYCz49YW5hyCcYMHUZ1Q#gid=2 official matrix]) | ||
| + | *[[HOWTO migrate a CGI|HOWTO migrate a CGI]] - ''quick overview of how to migrate an existing CGI into a Model, Controller, and View. Sparse'' | ||
| + | |||
| + | ==== Load balancing and Failover ==== | ||
| + | *[[Building Squid|Building Squid]] - ''How to build, install and configure Squid'' | ||
| + | |||
| + | ==== Administration ==== | ||
| + | *[[Formal_development_strategy|Formal development strategy]] - ''An overview of the development process at WormBase'' -- THE OLD VERSION | ||
*[[MySQL database replication|MySQL database replication]] - ''Mirroring mysql databases across production nodes'' | *[[MySQL database replication|MySQL database replication]] - ''Mirroring mysql databases across production nodes'' | ||
*[[Trac installation notes|Trac installation notes]] | *[[Trac installation notes|Trac installation notes]] | ||
| Line 182: | Line 184: | ||
* [[Establishing the SVN repository]] - ''Setting up the subversion repository'' | * [[Establishing the SVN repository]] - ''Setting up the subversion repository'' | ||
* [[Managing GBrowse]] - ''Quick HowTo for managing GBrowse in development and production environments'' | * [[Managing GBrowse]] - ''Quick HowTo for managing GBrowse in development and production environments'' | ||
| + | *[[Website:Monitoring services|Monitoring services and servers]] - ''Monitoring services and servers'' | ||
==== Frozen Releases as Virtual Machines ==== | ==== Frozen Releases as Virtual Machines ==== | ||
Latest revision as of 18:25, 18 June 2014
This page contains information pertinent to employees of WormBase.
Contents
Web Dev Category Hierarchy (Under Construction)
You will be able to navigate the Developer Documentation by browsing through the Web Dev Category Hierarchy (Under Construction) above, which is under construction right now. In the mean time, please browse through this page or search for information that you need.
<categorytree mode=pages showcount="on" hideroot="on" depth=4 >Web Dev</categorytree>
How to Contribute to the Documentation
If you'd like to help out adding to the documentation, that's great! Here is some general good practice that will help keeping the documentation organized, navigable and up to date.
- Ensure a page is tagged with a category, when you add or edit a page. You can browse through our high level categories and locate the most specific category that describes your page.
For example, include the following code at the bottom of a page will make the page appear under the category Category: Getting Started (Web Dev) :
[[Category: Getting Started (Web Dev)]]
- Update, deprecate or remove old documentation when appropriate.
- Any documents that contain potentially sensitive information belong on Google Docs (Contact Todd for details). This might reasonably include anything that reveals information about our systems or architecture.
- Additional guidelines on Documentation and Communication Standards
- Other ideas? Please update the wiki here!
New Employee Orientation
- New Employee Orientation -- Getting started at WormBase, for employees at any location. Read this first.
- Webdev team/OICR Orientation -- For new employees of the web development team at OICR.
- Co-op Documentation
Project Management
Conference call agendas/minutes
- Conference call agendas and minutes (All WormBase)
- Conference call agendas and minutes (WebDev team)
- Project meeting agendas and minutes
- InterMine Strategy Meetings
Source Code Repositories
We use the distributed source code management system git. Projects are hosted on Github.
- website - The WormBase web application (public)
- website-admin - Administrative code (private)
Recommended reading (learning git):
General Documentation
The Web Application
- Application overview - An overview of the web application, including MVC structure, request flow, caching, etc
- Configuration -- Application and view configuration: formatting and standards
- WormBase::API -- The WormBase data model, maintained external to the web application
- View -- Templating system, commom template elements, javascript, and CSS
- Model -- The web application model; thin, glued to the external Model
- Controller -- Overview of controller files and actions
- Genome Browsing
- GBrowse Set-up on dev.wormbase.org -- "Updating/rolling-back GBrowse installations on a per user basis"
- Genome Browser Survey -- "Community survey for evaluating use cases and required functionality of a WormBase genome browser"
- Visualization
- Website:Visualization Library Assessment -- "Assessment of several visualization options for integration in the Website"
- Website:Visualization with ggplot2 of the R-Project -- "Installation, set-up, and other information about the R-project based visualization"
- WormMine -- How to install, update, and maintain the InterMine instance at WormBase
- Website:WormMine Builds -- Summary stats of WormMine builds
- Contribute from Github
The Mobile Website
- Mobile -- Overview of the mobile website
- Router -- A definition of the possible routes of the mobile website
- Models -- Details about the models
- Collections -- Details about the collections
- Views -- Details about the views
- Widgets Porting -- Details about the porting of widgets in the mobile website/app
Data Model and Data Development
- WormBase Models
- Genome Standards -- What criteria should a genome meet before we accept it for integration?
- Suggested pre-release data checks
- modENCODE Analysis & metadata discussion
- modENCODE Integration status
- Genome sequence changes
- WormBase Genomes
- GFF Release Data and Changes
Development Guidelines
- Setting up your development environment -- How to set up your development environment on dev.wormbase.org
- Developer Resources at WormBase -- Available developer resources, including the shared development space
- WormBase Release Workflow -- Project-wide schedule that documents what is happening when and where. (original wiki version)
- Formal Development Strategy -- An overview of the development process at WormBase
- Documentation and Communication Standards
- Coding standards
- Developer notes -- Best practices, conventions, etc
- Browser Support Statement -- official graded browser support statement
Testing
Help us maintain the site
Information for third party developers
Administration
Architecture
Software Platform
- Installing WormBase - How to get a basic installation of WormBase up and running
- Installing WormMine - How to get a basic installation of WormMine up and running
- Production Environment - Building and Maintaining the WormBase production environment; extends the "Installing WormBase" document
- GBrowse Administration -- building, configuring, and maintaining production GBrowse nodes.
- Building WormMart - How to convert a WormBase AceDB database into WormMart
Hardware Platform
- Hardware Platform - A complete accounting of WormBase compute resources.
Software Life Cycle
- Software Life Cycle: 1. Build -- Preparing a new release for the website
- Software development period - Once a new database has been staged, the development phase begins in earnest on private development sites
- QA/QC and Performance Benchmarking - Integrating changes and staging the code for QA/QC and performance testing prior to deployment
- Software Life Cycle: 4. Production -- Deploying a new release to production
Measuring Application Performance
- Unit Testing and Profiling - Testing and profiling protocol for the web application
- System Testing
- Standards Validation - Validation of HTML, CSS, XML
- Security Assessment
- Accessibility
- Usability Testing - usability testing strategies and evaluation
- Browser Compatibility Testing - browser support matrix and testing protocol
- Load Testing - Performance and load testing; historical benchmarks
Routine Administration Tasks
- Cronjobs - a full accounting of all cronjobs at WB
- Creating a Google Sitemap - how to tell Google which URLs to index
- Managing Perl Libraries - Managing Perl libraries across projects and architectures
- Log file management and analysis - rotating and analyzing logs.
- Site statistics
Monitoring and Crises Management
- Crises Recovery Plan - How to diagnose and recover from service interruptions.
- Monitoring services and servers - Keeping tabs on the production architecture
Frozen Releases
- Building Frozen Releases - The most current documentation on how to build a frozen release
Deprecated Documentation
The good word gone bad. Cruft. Lint. Maybe useful. Probably not.
Rearchitecture
- Development Milestones
- Requirements analysis / Design Specifications
- Rearchitecture Development Duties
- AceDB classes -> API progress matrix (index of classes; see official matrix)
- HOWTO migrate a CGI - quick overview of how to migrate an existing CGI into a Model, Controller, and View. Sparse
Load balancing and Failover
- Building Squid - How to build, install and configure Squid
Administration
- Formal development strategy - An overview of the development process at WormBase -- THE OLD VERSION
- MySQL database replication - Mirroring mysql databases across production nodes
- Trac installation notes
- Trac Feature Tracker / Subversion repository
- Establishing the SVN repository - Setting up the subversion repository
- Managing GBrowse - Quick HowTo for managing GBrowse in development and production environments
- Monitoring services and servers - Monitoring services and servers
Frozen Releases as Virtual Machines
- How to build a WormBase Virtual Machine -- also describes how to create a VMX for a frozen release
- Configuring a server to host virtual machines -- quick guide to setting up a frozen release server
- Managing servers hosting virtual machines
- How to build a frozen release -- how to configure a WormBase VM as a frozen release server
- Creating Virtual Disks - How to create VMDKs for unlimited portable storage
