Difference between revisions of "Developer documentation"

From WormBaseWiki
Jump to navigationJump to search
 
(239 intermediate revisions by 11 users not shown)
Line 1: Line 1:
 
This page contains information pertinent to employees of WormBase.  
 
This page contains information pertinent to employees of WormBase.  
  
= Project Management =
+
= Navigating and Editing the Developer Documentation =
*[[Project Calendar|Project Calendar]]
 
*[[Conference call agendas and minutes]]
 
*[[Project meeting agendas and minutes]]
 
*[[Linking To WormBase|Linking To WormBase]]
 
*[[Suggested pre-release data checks|Suggested pre-release data checks]]
 
*[[SOPs|Internal documentation and standard operating procedures]]
 
*[[modENCODE Analysis & metadata discussion|modENCODE Analysis & metadata discussion]]
 
*[[Genome Standards|What criteria should a genome meet before we accept it for integration?]]
 
*[[WormBase Models]]
 
*[[Orthologs: use cases for searches]]
 
*[[Genome sequence changes]]
 
  
= Web Site Documentation =
 
  
* [[Using_private_development_sites|Installing the Web Application (legacy code) ]] - ''Here's how to install the classic WormBase web site for development purposes''
 
*[[Installing the Web Application|Installing the Web Application]] - ''Here's how to install the WormBase web application -- including Catalyst -- on your system''
 
*[[HOWTO migrate a CGI|HOWTO migrate a CGI]] - ''quick overview of how to migrate an existing CGI into a Model, Controller, and View. Sparse''
 
*[[Website:Design Specifications|Requirements analysis / Design Specifications]]
 
  
== Development Milestones ==
+
== [[:Category:Web Dev| Web Dev Category Hierarchy (Under Construction)]] ==
  
=== Measuring Progress ===
+
'''You will be able to navigate the Developer Documentation by browsing through the [[:Category:Web Dev| Web Dev Category Hierarchy (Under Construction)]] above, which is under construction right now. In the mean time, please browse through this page or search for information that you need.'''
* [[AceOjbect_Redesign_Page|AceDB classes -> API progress matrix]]
 
*[http://spreadsheets.google.com/ccc?key=pHvlNLMCZIv1KBoy8ZhbZXA&hl=en Migration matrix (GoogleDoc)]
 
  
'''July 14 - July 30th'''
+
<categorytree mode=pages showcount="on" hideroot="on" depth=4 >Web Dev</categorytree>
* Coronate Czars
 
* Coding standards and best practices for CSS, Javascript, API in place
 
* Release new pages to production powered by the new architecture (classic view only)
 
** Variation, Protein, Sequence -- and maybe -- Gene
 
** '''Todd''' to develop deployment strategy, hosting environment, and proxy configuration
 
** '''Abby''' and '''Xiaoqi''' to work up templates and markup specific to the classic view
 
** '''Norie''' to make requisite API changes as needed
 
  
'''July 30th - August 31st'''
+
== How to Contribute to the Documentation ==
* [[AceOjbect_Redesign_Page|Finish API development of critical classes]]
+
If you'd like to help out adding to the documentation, that's great! Here is some general good practice that will help keeping the documentation organized, navigable and up to date.
* Finish migration of prominent CGI-based pages to new architecture and release into production (classic view only)
+
* Ensure a page is tagged with a category, when you add or edit a page. You can browse through our [[:Category:Web Dev| high level categories]] and locate the '''most specific category''' that describes your page.
* Solidify look-and-feel of new site: site map, browsing paths, design patterns, navigation
+
<blockquote> For example, include the following code at the bottom of a page will make the page appear under the category [[:Category: Getting Started (Web Dev)]] :
* UI features to address/develop
 
** Enhanced, high-performance search
 
** External URL handling
 
** Error trapping and reporting
 
** Authorization/Authentication
 
** Customizability
 
** Configurable two-panel selector
 
** Incorporation of old site elements like BLAST/BLAT
 
  
'''September 1st'''
+
<code><nowiki>
* Begin beta testing of new site to invited parties
+
[[Category: Getting Started (Web Dev)]]
 +
</nowiki></code>
  
'''October 1st - November 1st'''
+
</blockquote>
* Begin public beta testing of new site
 
* Launch surveys on UI
 
  
'''November 1st - December 1st'''
+
* '''Update''', deprecate or remove old documentation when appropriate.
* Address survey issues
 
* Test, test, test
 
  
'''December 1st'''
+
* Any documents that contain potentially '''sensitive information''' belong on Google Docs (Contact Todd for details). This might reasonably include anything that reveals information about '''our systems or architecture'''.
* Go Live with new site
 
  
== Division of Labor ==
+
* Additional guidelines on [[Documentation and Communication Standards]]
  
''Duties, current as of July 2010.''
+
* '''Other ideas?''' Please update the wiki here!
  
'''Configuration Czar''' -- ''View and application configuration'' -- '''Norie'''
+
= New Employee Orientation =
* Keep configuration files neat, orderly, and documented
+
* [https://docs.google.com/document/d/1E37IgK1gsaw6ZXa3cYFz2HPVPuB__KmNs5tErO6VXDY/pub New Employee Orientation] -- ''Getting started at WormBase, for employees at any location. Read this first.''
* Create new configuration for hard-coded variables and UI elements when needed.
+
* [https://docs.google.com/document/d/1Xm1aQTm7zXapfirgD1PP7KWNjPlXHNw_iaLLAVvZ-Gg/pub Webdev team/OICR Orientation] -- ''For new employees of the web development team at OICR.''
* Enforce/evangelize configuration standards and formatting
+
*[[Co-op Documentation]]
  
'''CSS Czar''' -- ''Responsible for the maintenance of CSS files''
+
= Project Management =
* Develop and document CSS best practices
+
*[[Project Calendar|Project Calendar (Google)]]
* Keep CSS files tidy and organized  function
 
* Enforce/evangelize standard CSS markup in templates
 
  
'''Javascript Czar''' -- ''[[web2.0 Javascript use and maintenance|Javascript use and maintenance]]''
+
==== Conference call agendas/minutes ====
* Develop and document Javascript best practices
 
* Explore and advise on javascript plugins
 
* Enforce/evangelize javascript standards and formatting
 
  
'''Template Czar''' -- ''Oversee template development''
+
*[[Conference calls| Conference call agendas and minutes (All WormBase)]]
* Develop and document template standards
+
*[[Conference call agendas and minutes| Conference call agendas and minutes (WebDev team)]]
* Define and document shared template elements (ie tables, fields, widgets)
+
*[[Project meeting agendas and minutes]]
* Keep template directory organized
+
*[[InterMine Strategy Meetings]]
* Enforce/evangelize template standards
 
  
'''Controller Czar''' -- ''Maintain and develop new controllers as needed''
+
==== Source Code Repositories ====
* Document existing controllers
 
* Document logic flow through application
 
* Develop new controller actions as requested/needed
 
  
'''WormBase::API Czar''' -- ''Oversee development of the external Model'' -- '''Norie'''
+
''We use the distributed source code management system git.  Projects are hosted on [http://github.org/WormBase Github].''
* Develop and document API coding standards and best practices
 
* Document shared API methods
 
* Enforce/evangelize coding standards
 
  
'''Testing Czar''' -- ''Oversee testing of the web app'' -- '''Todd'''
+
*[http://github.org/WormBase/website website] - ''The WormBase web application (public)''
* Develop and document test requirements
+
*[http://github.org/WormBase/website-admin website-admin] - ''Administrative code (private)''
* Ensure adequate test coverage and performance
 
* Develop global View and API tests
 
* Stress Test app
 
  
'''Profiling Czar''' -- ''Track progress and performance of the web app'' -- '''Todd'''
+
Recommended reading (learning git):
* Explore profiling options and discuss
+
* [http://try.github.io/levels/1/challenges/1 try.gihub.io]
** Built-in server and Catalyst::Stats
 
** -d profilers: Devel::NYTProf
 
* Add dynamically configurable profiling
 
* Identify and report bottlenecks
 
* Develop caching mechanism of ajax-requested widgets
 
  
== Coding standards and Best Practices ==
+
= General Documentation =
*[[Developer_notes|Developer notes]] - ''Best practices, conventions, etc''
+
*[[Co-op Documentation]]
*[[Coding standards|Coding standards]]  
+
*[[SOPs|Internal documentation and standard operating procedures]]
*[[Formal_development_strategy|Formal development strategy]] - ''An overview of the development process at WormBase''
+
*[[Feature WishList]]
  
== Software platform  ==
+
= The Web Application =
  
* [[Application overview|Application overview]] - ''A quick tour through the software platform, request flow, caching, etc''  
+
* [[Website:Application overview|Application overview]] - ''An overview of the web application, including MVC structure, request flow, caching, etc''  
 
* [[Website:Configuration|Configuration]] -- ''Application and view configuration: formatting and standards''
 
* [[Website:Configuration|Configuration]] -- ''Application and view configuration: formatting and standards''
 
* [[Website:WormBase::API|WormBase::API]] -- ''The WormBase data model, maintained external to the web application''
 
* [[Website:WormBase::API|WormBase::API]] -- ''The WormBase data model, maintained external to the web application''
Line 125: Line 68:
 
* [[Website:Model| Model]] -- ''The web application model; thin, glued to the external Model''
 
* [[Website:Model| Model]] -- ''The web application model; thin, glued to the external Model''
 
* [[Website:Controller| Controller]] -- ''Overview of controller files and actions''
 
* [[Website:Controller| Controller]] -- ''Overview of controller files and actions''
 +
* Genome Browsing
 +
** [[Website:GBrowse Set-up on dev.wormbase.org|GBrowse Set-up on dev.wormbase.org]] -- "Updating/rolling-back GBrowse installations on a per user basis"
 +
** [[Website:Genome Browser Survey|Genome Browser Survey]] -- "Community survey for evaluating use cases and required functionality of a WormBase genome browser"
 +
* Visualization
 +
** [[Website:Visualization Library Assessment]] -- "Assessment of several visualization options for integration in the Website"
 +
** [[Website:Visualization with ggplot2 of the R-Project]] -- "Installation, set-up, and other information about the R-project based visualization"
  
== Hardware platform  ==
+
* [[Website:WormMine| WormMine]] -- ''How to install, update, and maintain the InterMine instance at WormBase''
 +
** [[Website:WormMine Builds]] -- Summary stats of WormMine builds
 +
*[[Website:Contribute_from_GitHub|Contribute from Github]]
  
*[[MySQL database replication|MySQL database replication]] - ''Mirroring mysql databases across production nodes''  
+
== The Mobile Website ==
*[[Performance optimization|Performance optimization]] - ''Memory, Disk, and Network I/O troubleshooting''  
+
* [[Website:Mobile| Mobile]] -- ''Overview of the mobile website''
*[[WormBase_Infrastructure|WormBase Infrastructure]] - ''A complete accounting of WormBase compute resources and topology.
+
* [[Website:MobileRouter| Router]] -- ''A definition of the possible routes of the mobile website''
*[[WormBase Development Environment|WormBase Development Environment]] - ''Building, Configuring, and Administering the WormBase development server''
+
* [[Website:MobileModels| Models]] -- ''Details about the models''
 +
* [[Website:MobileCollections| Collections]] -- ''Details about the collections''
 +
* [[Website:MobileViews| Views]] -- ''Details about the views''
 +
* [[Website:MobileWidgets| Widgets Porting]] -- ''Details about the porting of widgets in the mobile website/app''
 +
= Data Model and Data Development =
 +
 
 +
*[[WormBase Models]]
 +
*[[Genome Standards]] -- ''What criteria should a genome meet before we accept it for integration?''
 +
*[[Suggested pre-release data checks|Suggested pre-release data checks]]
 +
*[[modENCODE Analysis & metadata discussion|modENCODE Analysis & metadata discussion]]
 +
*[[modENCODE Integration status|modENCODE Integration status]]
 +
*[[Genome sequence changes]]
 +
*[[WormBase Genomes]]
 +
*[[GFF Release Data and Changes]]
  
--
+
= Development Guidelines =
 +
* [https://docs.google.com/a/wormbase.org/document/d/1jps4_bTrsR-Km0QTKk3ENRckbKyK3y8jZwSvq3rh2lo/pub Setting up your development environment] -- ''How to set up your development environment on dev.wormbase.org''
 +
*[https://docs.google.com/a/wormbase.org/document/d/1ONNKWe82OL5oZoPBmQ3nOZ_EZuzDfjbVpYvMkjnfULc/edit#heading=h.3t2png3rdgj8 Developer Resources at WormBase] -- ''Available developer resources, including the shared development space''
 +
*[https://docs.google.com/document/d/1zIOm95tV9A14n9xnkPkMgiOTGcOIzx161y9UbDZvfW4/pub WormBase Release Workflow] -- ''Project-wide schedule that documents what is happening when and where. ([[WormBase Release Workflow|original wiki version)]]''
 +
*[[Development_workflow_-_webdev|Formal Development Strategy]] -- ''An overview of the development process at WormBase''
 +
*[[Documentation and Communication Standards|Documentation and Communication Standards]]
 +
*[[Coding standards|Coding standards]]
 +
*[[Developer_notes|Developer notes]] -- ''Best practices, conventions, etc''
 +
*[[Browser Support Statement]] -- ''official graded browser support statement''
  
* [[GBrowse Administration]] -- ''building, configuring, and maintaining production GBrowse nodes.''
+
== Testing ==
* [[Webserver Administration]] -- ''building, configuring, and maintaining non-GBrowse production nodes.''
+
*[[Unit Testing]]
  
== Administration  ==
+
== Help us maintain the site ==
 +
*[[Website:Contribute from GitHub|Contribute from GitHub]]
  
=== Routine Administration Tasks ===
+
= Information for third party developers =
* [[Testing|Testing]] - ''Testing guidelines for the web application and hardware platform''
 
* [[Profiling|Profiling]] - ''Profiling the web application (and historical benchmarks)''
 
* [[Deployment|Deployment]] - ''How to deploy the web app for production use''
 
* [[Cronjobs|Cronjobs]]
 
* [[Creating_a_Google_Sitemap|Creating a Google Sitemap]]
 
* [[Managing Perl Libraries]] - ''Managing Perl libraries across projects and architectures''
 
* [[Managing GBrowse]] - ''Quick HowTo for managing GBrowse in development and production environments''
 
* [[Log file management and analysis|Log file management and analysis]] - ''rotating and analyzing logs.''
 
* [[Site_statistics|Site statistics]]
 
  
=== Updates ===
+
*[[Linking To WormBase|Linking To WormBase]]  
* [[Updating The Development Server]] - ''Updating the development server with a new release of the database''
 
* [[Updating The Production Servers]] - ''Updating production environments with a new release of the database''
 
* [[Building WormMart|Building WormMart]] - ''How to convert a WormBase AceDB database into WormMart''
 
  
=== Load balancing and Failover ===
+
= Administration =
*[[Building Squid|Building Squid]] - ''How to build and install Squid''
 
*[[Configuring Squid|Configuring Squid]] - ''basic squid configuration at WormBase''
 
  
=== Monitoring and Crises Management ===
+
== Architecture ==
*[[Development Server Administration]] - ''Monitoring and fixing the development server''
 
* [[General_administration|General administration and monitoring notes]] - ''Partially redundant, sorry''
 
*[[Monitoring services and servers|Monitoring services and servers]]
 
*[[Crises Recovery Plan|Crises Recovery Plan]] - ''Everything breaks. Here's how to fix it.''
 
  
=== Frozen Releases ===
+
'''Software Platform'''
*[[Building Frozen Releases]] - ''The most current documentation on how to build a frozen release''
+
*[[Administration:Installing WormBase|Installing WormBase]] - ''How to get a basic installation of WormBase up and running''
 +
*[[Administration:Installing WormMine|Installing WormMine]] - ''How to get a basic installation of  WormMine up and running''
 +
*[[Administration:WormBase Production Environment|Production Environment]] - ''Building and Maintaining the WormBase production environment; extends the "Installing WormBase" document''
 +
* [[GBrowse Administration]] -- ''building, configuring, and maintaining production GBrowse nodes.''
 +
* [[Building WormMart|Building WormMart]] - ''How to convert a WormBase AceDB database into WormMart''
  
== Developer Tools ==
+
'''Hardware Platform'''
 +
*[[WormBase_Infrastructure|Hardware Platform]] - ''A complete accounting of WormBase compute resources.
  
*Twitter: [http://twitter.com/wormbase General announcements]
+
== Software Life Cycle ==
*FriendFeed: [http://friendfeed.com/wormbase WormBase FF room]
+
* [https://docs.google.com/a/wormbase.org/document/d/1oPpj8d5gibUc-gpUZorl6ETT5baE6mp-v2bMedKauiA/ Software Life Cycle: 1. Build] -- ''Preparing a new release for the website''
*[http://www.wormbase.org/forums/index.php?board=326.0 WormBase Forum: Developer Forum]  
+
* [[Software Life Cycle: 2. Software development period|Software development period]] - ''Once a new database has been staged, the development phase begins in earnest on private development sites''
 +
* [[Software Life Cycle: 3. QA/QC and Performance Benchmarking|QA/QC and Performance Benchmarking]] - ''Integrating changes and staging the code for QA/QC and performance testing prior to deployment''
 +
* [https://docs.google.com/a/wormbase.org/document/d/1KlweWzfxc9-uKxR1dJwuPfR4NPJs-H1S8Yxg_V1Ha8U/edit#heading=h.e2m5ps3gnzcs Software Life Cycle: 4. Production] -- ''Deploying a new release to production''
  
 +
== Measuring Application Performance ==
  
== Source Code Repositories ==
+
* [[Website:Testing_And_Profiling|Unit Testing and Profiling]] - ''Testing and profiling protocol for the web application''
 +
* System Testing
 +
* [[Website:Validation|Standards Validation]] - ''Validation of HTML, CSS, XML''
 +
* Security Assessment
 +
* Accessibility
 +
* [[Website:Usability Testing|Usability Testing]] - ''usability testing strategies and evaluation''
 +
* [[Website:Browser Compatibility Statement|Browser Compatibility Testing]] - ''browser support matrix and testing protocol''
 +
* [[Website:Load Testing|Load Testing]] - ''Performance and load testing; historical benchmarks''
 +
----
  
''We use the distributed source code management system Mercurial.  Projects are hosted on [http://bitbucket.org BitBucket].''
+
== Routine Administration Tasks ==
 +
* [[Cronjobs|Cronjobs]] - ''a full accounting of all cronjobs at WB''
 +
* [[Creating_a_Google_Sitemap|Creating a Google Sitemap]] - ''how to tell Google which URLs to index''
 +
* [[Managing Perl Libraries]] - ''Managing Perl libraries across projects and architectures''
 +
* [[Log file management and analysis|Log file management and analysis]] - ''rotating and analyzing logs.''
 +
* [[Site_statistics|Site statistics]]
  
[[WormBase Source Code|How-To create, check out, and manage our repositories]]
+
== Monitoring and Crises Management ==
 +
*[https://docs.google.com/a/wormbase.org/document/d/18oRqTtC-q1P1bPyLAjo4XkqMGzepPi5Be1w7CfLuzg8/edit?hl=en_US Crises Recovery Plan] - ''How to diagnose and recover from service interruptions.''
 +
*[https://docs.google.com/a/wormbase.org/document/d/18G-lemZ0zKXraQlzXqNmsUCWwZmjuUHIipF6uRd0R0k/edit Monitoring services and servers] - ''Keeping tabs on the production architecture''
  
*[http://bitbucket.org/tharris/wormbase WormBase] - ''The WormBase web application (public)''
+
== Frozen Releases ==
*[http://bitbucket.org/tharris/wormbase-admin WormBase-admin] - ''Administrative code (private)''
+
*[[Building Frozen Releases]] - ''The most current documentation on how to build a frozen release''
  
== Deprecated Documentation ==
+
= Deprecated Documentation =
  
 
''The good word gone bad. Cruft. Lint. Maybe useful. Probably not.''
 
''The good word gone bad. Cruft. Lint. Maybe useful. Probably not.''
*[[NFS at WormBase]]
+
 
 +
==== Rearchitecture ====
 +
*[[Website:Development Milestones|Development Milestones]]
 +
*[[Website:Design Specifications|Requirements analysis / Design Specifications]]
 +
*[[Website:Rearchitecture Development Duties|Rearchitecture Development Duties]]
 +
* AceDB classes -> API progress matrix (index of classes; see [https://spreadsheets.google.com/a/wormbase.org/ccc?key=tygSYCz49YW5hyCcYMHUZ1Q#gid=2 official matrix])
 +
*[[HOWTO migrate a CGI|HOWTO migrate a CGI]] - ''quick overview of how to migrate an existing CGI into a Model, Controller, and View. Sparse''
 +
 
 +
==== Load balancing and Failover  ====
 +
*[[Building Squid|Building Squid]] - ''How to build, install and configure Squid''
 +
 
 +
==== Administration ====
 +
*[[Formal_development_strategy|Formal development strategy]] - ''An overview of the development process at WormBase'' -- THE OLD VERSION
 +
*[[MySQL database replication|MySQL database replication]] - ''Mirroring mysql databases across production nodes''
 
*[[Trac installation notes|Trac installation notes]]
 
*[[Trac installation notes|Trac installation notes]]
 
*[http://dev.wormbase.org/trac Trac Feature Tracker / Subversion repository]
 
*[http://dev.wormbase.org/trac Trac Feature Tracker / Subversion repository]
 
* [[Establishing the SVN repository]] - ''Setting up the subversion repository''
 
* [[Establishing the SVN repository]] - ''Setting up the subversion repository''
 +
* [[Managing GBrowse]] - ''Quick HowTo for managing GBrowse in development and production environments''
 +
*[[Website:Monitoring services|Monitoring services and servers]] - ''Monitoring services and servers''
  
 
==== Frozen Releases as Virtual Machines ====
 
==== Frozen Releases as Virtual Machines ====
Line 199: Line 192:
 
* [[How_to_build_a_frozen_release|How to build a frozen release]] -- ''how to configure a WormBase VM as a frozen release server''
 
* [[How_to_build_a_frozen_release|How to build a frozen release]] -- ''how to configure a WormBase VM as a frozen release server''
 
* [[Creating Virtual Disks]] - '' How to create VMDKs for unlimited portable storage''
 
* [[Creating Virtual Disks]] - '' How to create VMDKs for unlimited portable storage''
 +
 +
 +
[[Category:Developer documentation]]

Latest revision as of 18:25, 18 June 2014

This page contains information pertinent to employees of WormBase.

Navigating and Editing the Developer Documentation

Web Dev Category Hierarchy (Under Construction)

You will be able to navigate the Developer Documentation by browsing through the Web Dev Category Hierarchy (Under Construction) above, which is under construction right now. In the mean time, please browse through this page or search for information that you need.

<categorytree mode=pages showcount="on" hideroot="on" depth=4 >Web Dev</categorytree>

How to Contribute to the Documentation

If you'd like to help out adding to the documentation, that's great! Here is some general good practice that will help keeping the documentation organized, navigable and up to date.

  • Ensure a page is tagged with a category, when you add or edit a page. You can browse through our high level categories and locate the most specific category that describes your page.

For example, include the following code at the bottom of a page will make the page appear under the category Category: Getting Started (Web Dev) :

[[Category: Getting Started (Web Dev)]]

  • Update, deprecate or remove old documentation when appropriate.
  • Any documents that contain potentially sensitive information belong on Google Docs (Contact Todd for details). This might reasonably include anything that reveals information about our systems or architecture.
  • Other ideas? Please update the wiki here!

New Employee Orientation

Project Management

Conference call agendas/minutes

Source Code Repositories

We use the distributed source code management system git. Projects are hosted on Github.

Recommended reading (learning git):

General Documentation

The Web Application

The Mobile Website

  • Mobile -- Overview of the mobile website
  • Router -- A definition of the possible routes of the mobile website
  • Models -- Details about the models
  • Collections -- Details about the collections
  • Views -- Details about the views
  • Widgets Porting -- Details about the porting of widgets in the mobile website/app

Data Model and Data Development

Development Guidelines

Testing

Help us maintain the site

Information for third party developers

Administration

Architecture

Software Platform

Hardware Platform

Software Life Cycle

Measuring Application Performance


Routine Administration Tasks

Monitoring and Crises Management

Frozen Releases

Deprecated Documentation

The good word gone bad. Cruft. Lint. Maybe useful. Probably not.

Rearchitecture

Load balancing and Failover

Administration

Frozen Releases as Virtual Machines