Difference between revisions of "Co-op Documentation"

From WormBaseWiki
Jump to navigationJump to search
 
(105 intermediate revisions by 8 users not shown)
Line 1: Line 1:
==Current Tasks==
 
===Tasks In Progress===
 
====Tool Corrections====
 
*'''Details:'''
 
**Tree Display Issues:
 
***''Note: tree display changes postponed because users are primarily dev and curators, primary focus is general correct function for ws230 release''
 
***Clicking opens tree display on separate page
 
****Create button for opening in new page and keep navigation in the widget (in an iframe)
 
****Clicking on objects with corresponding pages should actually open those pages with the tree display widget opened and focused
 
*****Allows better navigation using tree display
 
***Need to test that all types of object can be displayed (particularly links to external images that were using hack method to display)
 
*'''In Progress:'''
 
**"Can't locate object method "run" via package "WormBase::API::Service::tree" error
 
***Temporarily removing inclusion of TreeSubs at least allows loading and using
 
*'''Related/Example Links:'''
 
**[http://beta.wormbase.org/species/c_elegans/gene/WBGene00006763#f--10 unc-26] (beta) ("run method" error)
 
**[http://staging.wormbase.org/species/c_elegans/gene/WBGene00006763#f--10 unc-26] (staging) (no error)
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#f--10 unc-26] (dev)
 
*'''Completed:'''
 
**Tree Display Issues:
 
***Expand/Collapse Nodes and View schema links have invalid links (.../tools/tools/tree/run/...)
 
***Found issue where Controller passes param that is an array of array refs, but the method treats it as if they have been merged
 
****Created flatten function "sub flatten {return map { ref($_) eq 'ARRAY' ? flatten(@$_) : $_ } @_}"
 
****built-in or more efficient method in perl to handle this?
 
***Embedded links to objects were invalid. Only links to expand/collapse list tags were valid
 
****Linking was relative to current page which was assumed to be /tools/tree/run
 
****Changed to always link to tools/tree/run (for ws230 release)
 
 
====General Page Corrections====
 
*'''Details:'''
 
**Visiting all minor pages and verifying that all fields display information correctly (checking behaviour for sparsely- & densely-populated)
 
**Continuing with fatal non-compliance checks for all pages
 
**Standardization and optimization
 
***Converting methods to use standard such as _pack_obj/_get_evidence where necessary
 
***Converting templates to use macros such as build_data_table instead of building manually
 
*'''In Progress:'''
 
**Feature Page
 
***Should the SO_term in the overview be removed/changed?
 
**Gene Class
 
***If gene class uses other_name (e.g. mig->ppn) shouldn't the genes for both be accessible on the ppn page
 
****Searching 'mig' gives links to 2 separate pages, where one has mig info and the other has ppn info but the names are opposite
 
*****Both of these pages have ppn as the title when visited
 
**Picture
 
***Unable to find any pages for this class
 
****Has API methods, templates, and is defined in wormbase.conf
 
**Enabling precaching for widgets/pages with significant load times
 
***Set "precache true" in wormbase.conf under the widget/page
 
*'''Related/Example Links:'''
 
**[[Example Pages]]
 
*'''Completed:'''
 
**Analysis Page -> Overview Widget
 
***Added highlight box back (was originally empty, but now WBID is automatically added)
 
**Anatomy_function Page -> Associations Widget
 
***Added references and body parts involved for Anatomy functions
 
****Reason: No page for anatomy functions and the data should be displayed (also assists in navigation)
 
****[http://dev.wormbase.org:8023/species/c_elegans/anatomy_term/WBbt:0004522#1--10 WBbt:0004522]
 
**Expression_cluster Page -> Clustered_data, Genes, Overview(Remarks) Widgets
 
***Corrected invalid hash references, and corrected overlap issue for remarks
 
****<nowiki>http://dev.wormbase.org:8023/species/c_elegans/expression_cluster/[cgc5767]:cluster_88#01243--10</nowiki>
 
**Feature Page
 
***Corrected improperly displayed (overlapping) remarks in overview
 
***Corrected array ref issues in associations widget
 
**Gene Page
 
***Corrected improper retrieval of phenotypes_not_observed
 
**Gene Class
 
***Changed tables in 'Current Genes' and 'Previous Genes' widgets to use build_data_tables
 
****Consistent, paginated, searchable
 
**Gene Regulation
 
***Corrected reference issue for 'molecule regulator' field and modified 'regulates' field to be more clear
 
**GO_term
 
***Issue with size: If evidence is shown, GO_term cannot load if it contains > 100 motifs. Otherwise, all GO_terms can load
 
****Note: largest number of motifs is 880
 
***Classic site displays human diseases, beta doesn't
 
****Notes: Tried adding widget to conf (didn't work). Not sure where/how info is being read from for either site
 
***[http://dev.wormbase.org:8023/species/c_elegans/go_term/GO:0018996 GO:0018996] (Dev Version)
 
***[http://www.wormbase.org/db/ontology/gene?name=GO%3A0018996;class=GO_term GO:0018996] (Classic Version)
 
**Interaction
 
***Both classic and beta weren't displaying all interactions (possibly due to recent changes to model)
 
****All information shown was regarding the first(alphabetically) interaction_type (i.e. treating info as scalar instead of array)
 
***Removed numerous fields and created general summary table showing how everything is related
 
***Should the Interactors widget be removed/changed? Doesn't seem to provide much useful info
 
****Should just be replaced by Interactions Cytoscape view
 
***[http://dev.wormbase.org:8023/species/c_elegans/interaction/WBInteraction0004639 WBInteraction0004639] (Dev Version)
 
***[http://www.wormbase.org/db/gene/interaction?name=WBInteraction0004639;class=Interaction WBInteraction0004639] (Classic Version)
 
***[http://www.wormbase.org/db/misc/etree?name=WBInteraction0004639;class=Interaction WBInteraction0004639] (Tree Display)
 
***Restructured table to combine effector, effected, & non-directional into one "Interactors" column
 
****[http://dev.wormbase.org:8023/species/c_elegans/interaction/WBInteraction0500325 WBInteraction0500325]
 
**Microarray_results
 
***Only displayed one gene and one cds (both incorrectly due to tags being arrays but passed to pack_obj as single object)
 
****Changed to handle as array and use tags2link macro in template
 
***Added transcript and pseudogene fields
 
***Added "Experiments" widget containing associated experiments, range, and microarray details
 
***[http://dev.wormbase.org:8023/species/c_elegans/microarray_results/Aff_Y105C5.DD Aff_Y105C5.DD] (Dev Version)
 
**Molecule
 
***Corrected affected... fields to display data and formatted into tables
 
***Created Phenotypes Affected widget and moved relevant fields
 
***[http://dev.wormbase.org:8023/species/c_elegans/molecule/C027492 C027492] (Dev Version)
 
**Operon
 
***Added Structure widget to display gene information
 
***Added Location widget with Gene Models and Operon tracks
 
***[http://dev.wormbase.org:8023/species/c_elegans/operon/CEOP2606 CEOP2606] (Dev Version)
 
**Oligo
 
***Added In Sequences and PCR products fields to overview
 
**Structure_data
 
***Added overview template and corrected corresponding fields in API
 
***Added external links widget
 
**Standardization and optimization
 
***Combined and cleaned-up API methods for "Gene page->Phenotypes widget->Evidence column of tables"
 
====Refining Interactions widget====
 
*'''Details:'''
 
**Scale distance based from current gene based on connectedness(number of interactions)
 
***Shouldn't cluster by gene class (problems arise from classes like "let" where genes aren't necessarily related, so grouping/clustering can be misleading)
 
****Interesting example where this seems to happen automatically: [http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00012348#8--10 pptr-1]
 
*'''Progress:'''
 
**Checking cytoscapeweb documentation for how to incorporate this
 
*'''Related/Example Links:'''
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#8--10 unc-26](<del>82</del> 71)
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00000912#8--10 daf-16](<del>433</del> 181)
 
**[http://beta.wormbase.org/species/c_elegans/gene/WBGene00000912#8--10 old daf-16 (beta)](1386 w/ duplication)
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00000898#8--10 daf-2](600 no nearby)
 
*'''Completed:'''
 
**If the number of interactions is too high, only show the minimal table (hide Cytoscape?) and add "Show More"/"Show Neighbouring Interactions"? option
 
***Otherwise, show all interactions(nearby interactions included) and Cytoscape plugin
 
***NOTE: This may need to be revisited and optimized/cleaned up
 
***Looking into BLASTP field in Homology Widget on Protein page that uses similar mechanism
 
****remove duplication
 
****instead of toggle href load, load diagram with no nearby in toggle
 
****create threshold-determining algorithm based on nodes?
 
**Split interactions into 2 methods, one for those involving current gene and one for nearby interactions
 
**If local interactions exceeds <del>400</del> 100(after merging duplicates), doesn't load nearby
 
**Combined duplicate edges (duplicate implies matching source node, target node, direction, type, & phenotype)
 
**Changed table: "Details" column with single link to interaction object -> "Citations" column with list of links to duplicate interactions
 
***More than 10 results in a cell is now collapsible into "# Results"
 
**Modified tooltip
 
***Removed interaction number and link to page (misleading for multiple)
 
***Added phenotype and number of citations to tooltip
 
**Added "edge width based on number of citations" feature
 
***Set 15 as max number of citations; anything higher will be scaled down
 
**Change colour scheme to follow WormBase's
 
***Start by looking in main.css
 
 
====Gene page alterations====
 
*'''Details:'''
 
**Making corrections to Gene page interface based on curator consultation
 
*'''Progress:'''
 
**Interactions info needs to be loaded from a separate database to speed up
 
*'''Related/Example Links:'''
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#8--10 unc-26 Interactions Widget]
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#a--10 unc-26 Phenotypes Widget]
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#b--10 unc-26 Reagents Widget]
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#6--10 unc-26 Homology Widget]
 
**[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00006763#68ab--10 unc-26 All 4 Widgets]
 
*'''Completed:'''
 
**Phenotype:
 
***Added evidence information
 
****May need to be modified/re-organized (pending consultation)
 
****Gene Ontology widget uses similar mechanism
 
****_get_evidence method in Object.pm
 
**Consultation from curators. Teleconference: 11:30 am EST Wednesday, January 18th, 2012
 
**Interactions:
 
***Added edge-colour legend
 
***Hide no_interaction edges unless that option is selected/filtered for by user
 
**Phenotype:
 
***Standardize look and feel of tables (convert to data tables)
 
****Done changing Phenotypes, Phenotypes_not_observed, and RNAi tables to use build_data_table
 
****Modified corresponding API methods to return compatible data for build_data_table
 
***Combined/merged Phenotype and RNAi tables for observed and not_observed (with >10 results collapsible)
 
**Reagents:
 
***Drop the word "found" for search results
 
***Switch search results to actual list of results(collapsible if too many)
 
****See/learn how this was done for the Expression widget
 
****implemented using tags2link macro
 
**Homology:
 
***restore best blast hits (on the protein object)
 
***protein domains: alphabetize & restrict to just interpro hits
 
**Changing evidence in Phenotypes widget to be consistent
 
***Format: "name/id(paper)" with evidence drop-down when clicked for all types - Alleles, Transgenes,& RNAi(extract info)
 
 
====Sequence/Transcript/CDS page alterations====
 
*'''Details:'''
 
**Making corrections to Sequence, CDS, & Transcript pages based on curator consultation
 
**CDS
 
***Sequences Widget
 
****Show marked-up sequences for corresponding transcripts
 
**All
 
***Overview Widget
 
****Download dialog launched from highlight box: fix scrollbar and rapid-click issues
 
***Location Widget
 
****Highlight current on track or at least show gene names
 
*'''Progress:'''
 
**Split CDS and Transcript, now checking that methods do not attempt to access invalid fields
 
*'''Related/Example Links:'''
 
*[http://dev.wormbase.org:8023/species/c_elegans/cds/JC8.10c#01234--10 JC8.10c] (CDS)
 
*[http://dev.wormbase.org:8023/species/c_elegans/transcript/JC8.10c.1#01234--10 JC8.10c.1] (Transcript)
 
*[http://dev.wormbase.org:8023/species/c_elegans/sequence/yk444b8.5#01234--10 yk444b8.5] (Sequence)
 
*[http://dev.wormbase.org:8023/species/c_elegans/sequence/2RSSE#01234--10 2RSSE] (Sequence w/ Method)
 
*'''Completed:'''
 
**Consultation from curators. Teleconference: 11:30 am EST Wednesday, January 25th, 2012
 
**Separating Transcript(+CDS) and Sequence pages
 
***Separating CDS from Transcript
 
***Checking Sequence page methods to ensure that only valid fields are accessed
 
****Comparing with schema
 
****Removing eval statements
 
**Region Widget
 
***Redistribute data and remove widget
 
****Matching cDNAs moved to Reagents widget
 
****Moved "Transcripts in this region" field to Sequences widget
 
**Origin Widget
 
***Moved to Overview
 
***Changed to a curator-only field using curator_block macro
 
**Reagents Widget
 
***Add PCR products
 
****Tags/info should already be in the model
 
**Sequences Widget
 
***Predicted Exon Structure table
 
****Removed completely (Although tag exists in schema, it has not been populated for any Sequences, only Transcripts)
 
****<del>Calculate and show exon & intron sizes using "Relative to Itself"</del>
 
****<del>Remove "Relative to Superlink"</del>
 
*****<del>Start & End "Relative to Itself" are useful in some cases (keep)</del>
 
**Sequences
 
***Sequences Widget
 
****Show orthologs? (for navigation to related info, especially useful for curators)
 
*****Syntenic sequences
 
****Show corresponding gene in predicted genes and transcriptional units (e.g. unc-26 for yk444b8.5)
 
**Transcript
 
***Split into CDS and Transcript
 
***Location Widget
 
****Show protein domains for CDS
 
**All
 
***Overview Widget
 
****Make origin info available to users (not just curators)
 
****Show remarks/description associated with Method to make it more meaningful
 
*****e.g. Method for AAA27985.1 is ndb_cds which is not particularly meaningful
 
****Show affiliated gene(e.g. unc-26 for yk444b8.5)
 
*****Add navigation table (similar to table in sequences widget on gene page, e.g. Matching transcripts/CDS)
 
***Location Widget
 
****Hide YAC's, fosmids, & cosmids track
 
 
====Variation page alterations====
 
*'''Details:'''
 
**Making corrections to Variation page interface based on curator consultation
 
**Change default widgets (Overview, Genetics, Molecular Details, Phenotypes)
 
**Overview Widget
 
***Hide Status field if Live (show otherwise e.g. suppressed/dead)
 
***Remove redundant remarks
 
**Molecular Details Widget
 
***Check that the sequence is actually on the + strand?
 
***"Flanking Sequences" and "Context" should be removed
 
****Just show marked-up sequence by default (toggle if too large)
 
****Add colour legend (yellow=flanking, red=deletion/substitution/...)
 
***Features Affected field -> Predicted CDSs
 
****Add "Variation" to "Contained in:" to clarify
 
****Change Predicted CDSs and Clone to data tables
 
**Location Widget
 
***Remove contig submission track
 
**Phenotypes
 
***Remove remarks and add evidence/citations
 
***Remove phenotype description or hide by default
 
*'''Progress:'''
 
*'''Related/Example Links:'''
 
**[http://dev.wormbase.org:8023/species/c_elegans/variation/WBVar00143133#01234567--10 e345](Allele:deletion)
 
**[http://dev.wormbase.org:8023/species/c_elegans/variation/WBVar00278010#01234567--10 ok5175](Allele:substitution)
 
*'''Completed:'''
 
**Consultation from curators. Teleconference: 11:30 am EST Wednesday, February 1st, 2012
 
 
====Curator Requests====
 
*'''Details:'''
 
**Protein Page ([mailto:cgrove@caltech.edu Christian A Grove])
 
***[http://dev.wormbase.org:8023/species/c_elegans/protein/WP:CE28239#01234--10 UNC-26] (Sample Protein Page)
 
**#Overview widget: in the right-hand box, the "Status: history" is not clear. What does this mean? Do we need/want to display this?
 
**#External Links widget: "TREEFAM" links back to the same UNC-26 protein page, as does "WORMPEP". "WP:CE28239" links to the old UNC-26 protein page; is this what we want? Also, both UniProt links are dead/obsolete/outdated
 
**#We should add units to Molecular Weight wherever it is mentioned (as I said in the gene page review). The units should either be "kilodaltons" or "kD"
 
**#History widget: The "JC8.10" under "Predicted gene" links to a WormBase search result for "JC8.10". I think it would be better to direct users to the exact (unc-26) page, rather than having them have to select among many options.
 
**#The red text in the protein schematic for domain names is a bit hard on the eyes. Could we go with a more neutral color: black, grey, blue...?
 
**#Can we include descriptions of the domains depicted in the domain cartoons, or at least provide a link out to the relevant InterPro or PFAM pages?
 
**Gene Page ([mailto:cgrove@caltech.edu Christian A Grove])
 
***[http://dev.wormbase.org:8023/species/c_elegans/gene/WBGene00000846#09e--10 cup-5] (Sample Gene Page)
 
***Postponed:
 
****Human Diseases widget:
 
*****The description for the disease is cutoff: "...along the lysosomal pathway, affecting membra" ; is this intentional? Could we maybe add a "..." to the end?
 
****Phenotype widget
 
****Cross-platform issues (Chrome & Safari vs Firefox)
 
*****The table display for "Primary Sequence ID" in the Sequences widget is a little wonky on Firefox. As you can see from the screenshots (attached) as I resize the table in Firefox, that column adjusts strangely, skewing the text, whereas in Safari and Chrome, the table stays nice and neat.
 
******[[Media:Firefox_Table_1.png]]
 
******[[Media:Firefox_Table_2.png]]
 
******[[Media:Firefox_Table_3.png]]
 
******[[Media:Safari_Chrome_Table.png]]
 
***There are a number of unnecessary hyperlinks in the phenotype description details. For example, all remarks, the word "Genotype" after "Phenotype assay:", and the phrase "Uncharacterised_loss_of_function" after "Loss of function:" are all hyperlinked to what appears to be a general WormBase text search result. These should just be plain text.
 
***One general comment is that I think I would prefer that links out to other pages open a new tab by default, but I may be in the minority with that opinion (or that may have been your intention, but Firefox is not cooperating)
 
*'''Progress:'''
 
**Investigating hyperlinks in phenotype description details
 
*'''Related/Example Links:'''
 
*'''Completed:'''
 
**Gene Page
 
***External links widget:
 
****The OMIM link out appears to be malformed. The URL that the link directs me too is:
 
*****http://omim.org/enrty/OMIM:605248 (verbatim, entry is mispelled)
 
****whereas this is a working link:
 
*****http://omim.org/entry/605248
 
****(as I write this I realize that someone from OMIM just e-mailed to point this out)
 
****Queried ACeDB Database class for databases with URL, URL_Constructor, and Description and added those to the external_urls template
 
*****Need to verify that all links are working and add other links(i.e. databases that didn't match the query)
 
*****Added/updated descriptions (some from ACeDB, some from website's homepage)
 
***Homology widget:
 
****When I click on the download option for homologous proteins, a window pops up to show Sequence, Isoelectric Point, Molecular Weight, etc. I know this is more a Protein issue, but we should put units for Molecular Weight (e.g. "77.7 kiloDaltons" or "kD" as opposed to just "77.7")
 
***Genetics widget:
 
****In both the "Alleles" and the "Polymorphisms & Natural Variants" tables, the right-most column is "Protein effect" and "Location", respectively. Under "Protein effect" is displayed "nonsense" and "missense" (which are OK), but also "intron", which does not make sense. Under "Location" is displayed "intron" and "utr_3" (OK), but also "missense" and "silent" (not OK). I would suggest having a "Protein effect" and a "Location" column in each table and putting only relevant entries in each:
 
*****Protein effect: nonsense, missense, silent, etc.
 
*****Location: intron, exon, utr_3, utr_5, etc.
 
***Interactions widget:
 
****The different color nodes and thickness of lines are not explained in the legend. Can we add an explanation somewhere?
 
****In the "Citations" column is listed all of the WormBase Interaction objects affiliated with this interaction. I would expect to see paper references in a "Citations" column. Could we perhaps put the interactions under a "Interactions" column all the way to the left, and keep the "Citations" column at the right with WBPaper references?
 
****The Cytoscape network view looks great, but the window for viewing it is too short (vertically) in Firefox, and too tall in Safari and Chrome. Could we find an intermediate height or provide options to resize?
 
 
====Non-compliant Data Checks====
 
*'''Details:'''
 
**Using fatal_non_compliance check and debug option (enable in wormbase_local.conf) to determine if API methods return invalid data
 
***E.g. returning ACe objects(usually this should be a string) or empty arrays/hashes(should return undef)
 
**Usually these can be found when fields do not have data to return but give empty structures instead of just undef
 
*'''Progress:'''
 
**Correcting these as they're encountered
 
*'''Completed:'''
 
**Methods showing errors on the following pages were corrected:
 
***Variation, Person, Laboratory, Gene Class, Life Stages, Phenotype, Protein, Strain
 
====Interface Corrections====
 
*'''Details:'''
 
**Look into rearranging the elements in widgets so that they do not appear strange/distorted when switched from one- to two-column views or using smaller displays
 
**Will require modifying the templates
 
====Homology Widget====
 
*Needs to be modified; Contains too much information that shouldn't necessarily be grouped this way
 
*May need to move orthologs to separate widget
 
====Cytoscape plugin====
 
*Incorporate button/option to open pop-up for larger view (e.g. in case user has small screen or is using 2-column layout)
 
*Also, add plugin to interaction page (may need to wait until interactions are merged)
 
 
===Other Tasks===
 
Additional tasks that require further investigation
 
*Loading/processing large amounts of data
 
**Details:
 
***Likely approach will be to change the database used from ACeDB to NoSQL/CouchDB
 
***[http://beta.wormbase.org/resources/laboratory/VC#1--10 Sample Issue]
 
**Progress:
 
***[https://github.com/aduong/ace2couch Ace2Couch Scripts] for migrating data
 
***[https://github.com/aduong/AceCouch AceCouch Perl API]
 
 
 
==Getting Started==
 
==Getting Started==
 
See [[WormBase Staff Resources]].
 
See [[WormBase Staff Resources]].
  
===Members/Contacts===
+
===WormBase WebDev Team Members/Contacts===
*Abigail Cabunoc - [mailto:abigail.cabunoc@oicr.on.ca abigail.cabunoc@oicr.on.ca]
+
*Lincoln Stein - [mailto:lincoln.stein@gmail.com lincoln.stein@gmail.com]
 
*Todd Harris - [mailto:todd@wormbase.org todd@wormbase.org]
 
*Todd Harris - [mailto:todd@wormbase.org todd@wormbase.org]
*Lincoln Stein - [mailto:lincoln.stein@gmail.com lincoln.stein@gmail.com]
+
*Sibyl Gao - [mailto:sibyl@wormbase.org sibyl@wormbase.org]
*Quang Trinh
+
*Scott Cain - [mailto:scott@scottcain.net scott@scottcain.net]
  
 
===Meetings===
 
===Meetings===
*WormBase OICR Developers Teleconference
 
**Mondays, 3:00PM
 
**Phone-in 1-800-747-5150 id: 6738514
 
 
*WormBase OICR Developers Teleconference with Lincoln
 
*WormBase OICR Developers Teleconference with Lincoln
**Wednesdays, 4:00PM EST
+
**First Friday of the Month, 11:00AM
**Lincoln's office
+
**Phone-in 1-800-747-5150
**[http://wiki.wormbase.org/index.php/Conference_call_agendas_and_minutes Agenda and Minutes]
+
***Access Code: 6738514
*WormBase International Groups Teleconference
+
**OR: Lincoln's office
**Every other Thursday, 11:30AM
+
*WormBase International Group (HelpDesk) Teleconference
**Lincoln's Office
+
**First Thursday of the Month, 11:30AM
**Alternatively: 1-866-528-2256
+
**1-866-528-2256
 
***Access Code: 714646
 
***Access Code: 714646
*Group Meeting
+
**[[Help_desk_schedule|HelpDesk Schedule and Agendas]]
**Fridays, 3:00PM
+
**[[Conference calls|Past Agendas and Minutes]]
 +
*Stein Lab, Group Meeting
 +
**Fridays, 2:00PM
 
**HL31 Conference Room/TBA
 
**HL31 Conference Room/TBA
 +
* Web Development Issues Meeting
 +
** Tuesday 10:00am
 +
** [[WebDev_Issues_Meetings]]
  
 
==Tools/Resources==
 
==Tools/Resources==
Line 383: Line 40:
  
 
===Git===
 
===Git===
 +
====Intro to Git====
 +
Tutorial: [http://try.github.io try.github.io]
 +
 
Git is a version control system used for collaboration and backup in the development process. One starting point is the [http://progit.org/book/ch1-3.html progit tutorial]
 
Git is a version control system used for collaboration and backup in the development process. One starting point is the [http://progit.org/book/ch1-3.html progit tutorial]
 
*WormBase repository located at https://github.com/organizations/WormBase
 
*WormBase repository located at https://github.com/organizations/WormBase
Line 394: Line 54:
 
  git '''add''' a.txt
 
  git '''add''' a.txt
 
  git '''commit''' -m “Added ''change1'' and ''change2'' to a.txt”
 
  git '''commit''' -m “Added ''change1'' and ''change2'' to a.txt”
 +
git '''pull'''
 
  git '''push'''
 
  git '''push'''
 
</code>
 
</code>
If we have did not have the most recent version, then we will run into an issue when trying to push. In this case we can:
+
Notes:
<code>
+
* pull is similar to using 'git fetch' + 'git merge'. However, performing them separately allows you to review the changes coming in before merging.
  git '''fetch'''
+
* the -m flag allows the user to input the message inline. Exlcuding it will result in an editor being opened for message entry
  git '''merge'''
+
 
  git '''push'''
+
====Setting-Up Repository====
</code>
+
*If you do not already have an account on GitHub, sign up
Note: pull is similar to using fetch + merge
+
*Send a request to [mailto:todd@wormbase.org Todd] to grant you access to both the dev.wormbase.org machine and the WormBase repositories
 +
*Connect via ssh to dev.wormbase.org
 +
*Follow the GitHub instructions for [http://help.github.com/linux-set-up-git/ Set Up Git] to configure SSH keys
 +
**Skip step 1, git should already be installed
 +
*To check-out the repository:
 +
  $ cd /usr/local/wormbase/website
 +
  $ git clone git@github.com:WormBase/website.git
 +
  $ mv website [your username]
 +
*See [[Administration:Installing_WormBase]] for more details
  
 
===Browser (Debugging)===
 
===Browser (Debugging)===
Line 411: Line 80:
 
*Firefox->Web Developer->Web Console/Error Console
 
*Firefox->Web Developer->Web Console/Error Console
  
===Other===
+
===ACeDB===
Some other tools that you should be aware of but may not be required to know/interact with include:
 
*JavaScript
 
 
*ACeDB
 
*ACeDB
 
**Usage:
 
**Usage:
 
***Navigate to /usr/local/wormbase/acedb/bin and run: "./tace ../wormbase"
 
***Navigate to /usr/local/wormbase/acedb/bin and run: "./tace ../wormbase"
 
***For more help, try the [http://www.acedb.org/Cornell/tace.html tace tutorial]
 
***For more help, try the [http://www.acedb.org/Cornell/tace.html tace tutorial]
 +
**Troubleshooting:
 +
***To restart the server on dev: '''Must be super user'''
 +
/etc/init.d/xinetd stop
 +
ps aux | grep sgiface
 +
!!If sgifaceserver is still running use 'kill #####' (where ##### is the process id)
 +
/etc/init.d/xinetd start
 +
 +
===CouchDB===
 +
**Troubleshooting:
 +
***To restart the server on dev:
 +
  sudo /etc/init.d/couchdb restart
 +
 +
===MySQL===
 +
**Troubleshooting:
 +
***To restart the server on dev:
 +
  sudo /etc/init.d/mysql restart
 +
 +
===Template Toolkit===
 +
*[http://template-toolkit.org/docs/index.html Template Toolkit Documentation]
 +
*Common useful macros for templates can be found in:
 +
**/root/templates/config/main
 +
***e.g. tags2link
 +
**/root/templates/shared/page_elements.tt2
 +
***e.g. cell_content, build_data_table
 +
***cell_content is mainly used for build_data_table to format the information to be displayed
 +
****makes use of other macros for displaying special data such as links(MACRO tags2link) and evidence boxes(MACRO evidence)
 +
****runs into errors when an ACe object is passed. Corrections should be made in the API to prevent ACe objects being returned to interface
 +
****Can be used to display many types of data (not restricted for use in build_data_table)
 +
****determines type of the content through a series of checks
 +
*****If the passed reference is an array, it performs recursion on each element
 +
*****If hash, outputs the key followed by the formatted content
 +
*****To add new case, check for a uniquely identifying property of the data and then perform the desired action (e.g. run a corresponding MACRO on the data)
 +
# e.g. Evidence data is always stored in a hash with a 'text' key and an 'evidence' key
 +
# Therefore cell_content checks whether the evidence key is defined in the data to be outputted
 +
# If it is, the evidence() macro is run with the data.
 +
[% MACRO cell_content(cell) BLOCK;
 +
      ...
 +
      ELSIF cell.evidence.defined;
 +
        evidence(cell.evidence,table_name _ index, cell.text, 'details');
 +
      ...
 +
END; %]
 +
 +
===Other===
 +
Some other tools that you should be aware of but may not be required to know/use include:
 +
*JavaScript
 
*MySQL
 
*MySQL
 
*Xapian
 
*Xapian
Line 427: Line 139:
 
**Installation directory (for updating): [WormBase dir]/root/js/jquery/plugins/
 
**Installation directory (for updating): [WormBase dir]/root/js/jquery/plugins/
  
==General Info==
+
==General Concepts==
*[https://wiki.oicr.on.ca/display/BCF/In-House+Tutorials+and+Workshops Unix and Cluster training by Quang] (OICR login required)
+
===Starting the server===
 +
*In a terminal, connect to wb-dev (via ssh) and navigate to the root of your dev site (where you checked-out the git repository e.g. /usr/local/wormbase/website/[user]/)
 +
*Run the startup script: "./script/wormbase_server.pl -p ####" where #### is a port number
 +
**Typical options are -d for debugging and -r to auto-restart when changes to the code are detected
 +
**Output can be redirected into a log file ">>logs/server_log.log 2>&1" (redirects both output and errors into server_log.log)
 +
**Therefore a full command could be "./script/wormbase_server.pl -d -r -p 8023 >>logs/server_log.log 2>&1"
 +
**Logs can be monitored in real-time by running "tail -f logs/server_log.log" or "tail -f logs/wb-dev-catalyst.log" in a separate terminal
 +
*Check [https://wiki.oicr.on.ca/display/BCF/In-House+Tutorials+and+Workshops OICR Tutorials and Workshops] for useful Unix commands (OICR login required)
 +
 
 +
===Debugging Tools===
 +
* Chrome provides a very useful set of tools for debugging that can be accessed by either 'Inspect Element' in the context(right-click) menu, or from Menu->Tools->Developer Tools (Shift+Ctrl+i)
 +
* Firefox has a "firebug" plugin that provides similar functionality
 +
*The tools can be used to monitor different aspects of the page
 +
**Network information such as GET/POST requests and their corresponding info (e.g. url and parameters)
 +
**Underlying HTML and scripts for the page
  
==General Concepts==
 
 
===Widget Data Loading===
 
===Widget Data Loading===
 
[[File:DataLoadingWidget1.png|200px|thumb|Widget Loading]]
 
[[File:DataLoadingWidget1.png|200px|thumb|Widget Loading]]
Line 439: Line 164:
 
#API methods request data from appropriate databases, process/format/package the data, and then return it(/lib/WormBase/API/Object/[Class].pm)
 
#API methods request data from appropriate databases, process/format/package the data, and then return it(/lib/WormBase/API/Object/[Class].pm)
 
#Sends data to the template to be used in rendering the widget (/root/templates/classes/[Class]/[Widget].tt2)
 
#Sends data to the template to be used in rendering the widget (/root/templates/classes/[Class]/[Widget].tt2)
 +
===Tool Data Loading===
 +
[[File:ToolDataLoading.png|350px|thumb|Tool Loading]]
 +
Typical flow of information:
 +
#Request information such as the url and parameters are passed from the browser to the server
 +
#Tools controller in server processes requests to /tools/... and calls corresponding API method
 +
#*Determines tool and action from url (e.g. /tools/tree/run... => tool=tree, action=run)
 +
#** Note: schema and gmap tools have similar function to other tools, tree and epic respectively
 +
#** Any calls to these are translated to calls to the corresponding tool (e.g. schema->tree or gmap->epic)
 +
#*Processes special parameters such as inline (for embedding in widget)
 +
#*Calls the API method corresponding to the action (e.g. "run" in tree.pm)
 +
#API method processes information passed from Controller and returns processed/packaged info to Tools Controller
 +
#*Loads data passed from Controller(e.g. "my $name = $params->{'name'};" loads the name)
 +
#*Loads appropriate data from databases (e.g. ACe object corresponding to the given 'name' & 'class' params)
 +
#*Performs subsequent processing on data (e.g. blast tool runs blast with the given params)
 +
#*Formats/packages data (e.g. commonly used method is _pack_obj which stores appropriate info for creating links for the interface)
 +
#** Note: it is important to not return raw data (primarily ACe objects) to the interface. Two possible fixes are _pack_obj($object) or "$object" (stringify)
 +
#** Note: it is also important to return "undef" instead of null or empty arrays/hashes
 +
#Tools Controller passes API results to View (Template)
 +
#View (Template) arranges/outputs the data for display
 +
#*Uses various methods/macros
 +
#**tag2link to convert a hash generated using _pack_obj into a link
 +
#**build_data_table to generate a formatted data table
 +
#** Note: common/useful macros can be found in /root/templates/config/main and /root/templates/shared/page_elements.tt2
 +
 +
==Guides==
 +
* [[Adding Data to a Widget (Example)]]
 +
* [[Creating a widget]]
 +
* [[Creating a tool]]
  
 
==Other Info==
 
==Other Info==
Useful Macros when editing templates can be found in:
 
*/root/templates/config/main
 
**e.g. tags2link
 
*/root/templates/shared/page_elements.tt2
 
**e.g. build_data_table
 
 
Useful files for debugging include:
 
Useful files for debugging include:
*/logs/wb-dev-catalyst.log
+
*/logs/wormbase-catalyst-access.log
 
Other:
 
Other:
 
*If the port you have been using appears to be busy but the server is not running,
 
*If the port you have been using appears to be busy but the server is not running,
  ps -aux|grep -XXXX
+
  ps -aux|grep -####
 
  kill -9 ID
 
  kill -9 ID
where XXXX is the port number(e.g. 8023) and ID is the process id that is using the port(e.g. 10361)
+
where #### is the port number(e.g. 8023) and ID is the process id that is using the port(e.g. 10361)
'''NOTE: Do not kill other users' processes if you are using a shared dev machine (e.g. wb-dev)'''
 
 
*To dump data in API methods include Data::Dumper
 
*To dump data in API methods include Data::Dumper
 
  use Data::Dumper
 
  use Data::Dumper
Line 459: Line 206:
 
  warn(Dumper(\@data));
 
  warn(Dumper(\@data));
 
*Sometimes firewall blocks certain ports, so you may be unable to connect to your dev server on the wb-dev machine through the browser
 
*Sometimes firewall blocks certain ports, so you may be unable to connect to your dev server on the wb-dev machine through the browser
  ssh -L 8080:localhost:XXXX wb-dev.oicr.on.ca
+
  ssh -L 8080:localhost:#### wb-dev.oicr.on.ca
  ./wormbase_server.pl -p XXXX -d -r
+
(where '####' is the port that you have been assigned by Todd).
Then go to localhost:8080 in browser<br />
+
You will be prompted to login.  Then cd to /usr/local/wormbase/website/your_name/
 +
and type this command as usual:
 +
  ./wormbase_server.pl -p #### -d -r
 +
where #### is your desired port number. Then point your browser at localhost:8080 <br />
 +
 
 +
==Current Tasks==
 +
 
 +
Additional tasks that require further investigation
 +
*Loading/processing large amounts of data
 +
**Details:
 +
***Likely approach will be to change the database used from ACeDB to NoSQL/CouchDB
 +
***[http://beta.wormbase.org/resources/laboratory/VC#1--10 Sample Issue]
 +
**Progress:
 +
***[https://github.com/aduong/ace2couch Ace2Couch Scripts] for migrating data
 +
***[https://github.com/aduong/AceCouch AceCouch Perl API]
 +
 
 +
[[Category: Getting Started (Web Dev)]]

Latest revision as of 14:31, 17 May 2016

Getting Started

See WormBase Staff Resources.

WormBase WebDev Team Members/Contacts

Meetings

  • WormBase OICR Developers Teleconference with Lincoln
    • First Friday of the Month, 11:00AM
    • Phone-in 1-800-747-5150
      • Access Code: 6738514
    • OR: Lincoln's office
  • WormBase International Group (HelpDesk) Teleconference
  • Stein Lab, Group Meeting
    • Fridays, 2:00PM
    • HL31 Conference Room/TBA
  • Web Development Issues Meeting

Tools/Resources

The following are some of the tools with which many major aspects of WormBase are developed. You should familiarize yourself with them through documentation and examples.

Perl

There is significant documentation on getting started with Perl. One starting point is PerlMonks For information regarding the use of ACeDB in conjuction with Perl (e.g. retrieving data), check AcePerl Documentation. The Ace::Object section contains most of the information related to interacting with ACeDB objects.

Catalyst

Catalyst is the web development framework used to develop WormBase. To get started, read and try examples from

Git

Intro to Git

Tutorial: try.github.io

Git is a version control system used for collaboration and backup in the development process. One starting point is the progit tutorial

Given that we have modified two files a.txt and b.txt but do not wish to keep the changes made to b.txt

git status
git checkout b.txt
git add a.txt
git commit -m “Added change1 and change2 to a.txt”
git pull
git push

Notes:

  • pull is similar to using 'git fetch' + 'git merge'. However, performing them separately allows you to review the changes coming in before merging.
  • the -m flag allows the user to input the message inline. Exlcuding it will result in an editor being opened for message entry

Setting-Up Repository

  • If you do not already have an account on GitHub, sign up
  • Send a request to Todd to grant you access to both the dev.wormbase.org machine and the WormBase repositories
  • Connect via ssh to dev.wormbase.org
  • Follow the GitHub instructions for Set Up Git to configure SSH keys
    • Skip step 1, git should already be installed
  • To check-out the repository:
$ cd /usr/local/wormbase/website
$ git clone git@github.com:WormBase/website.git
$ mv website [your username]

Browser (Debugging)

Many browsers provide useful tools to developers that can be used for debugging
Chrome

  • Tools->Developer Tools/JavaScript Console

Firefox

  • Firefox->Web Developer->Web Console/Error Console

ACeDB

  • ACeDB
    • Usage:
      • Navigate to /usr/local/wormbase/acedb/bin and run: "./tace ../wormbase"
      • For more help, try the tace tutorial
    • Troubleshooting:
      • To restart the server on dev: Must be super user
/etc/init.d/xinetd stop
ps aux | grep sgiface
!!If sgifaceserver is still running use 'kill #####' (where ##### is the process id)
/etc/init.d/xinetd start

CouchDB

    • Troubleshooting:
      • To restart the server on dev:
 sudo /etc/init.d/couchdb restart

MySQL

    • Troubleshooting:
      • To restart the server on dev:
 sudo /etc/init.d/mysql restart

Template Toolkit

  • Template Toolkit Documentation
  • Common useful macros for templates can be found in:
    • /root/templates/config/main
      • e.g. tags2link
    • /root/templates/shared/page_elements.tt2
      • e.g. cell_content, build_data_table
      • cell_content is mainly used for build_data_table to format the information to be displayed
        • makes use of other macros for displaying special data such as links(MACRO tags2link) and evidence boxes(MACRO evidence)
        • runs into errors when an ACe object is passed. Corrections should be made in the API to prevent ACe objects being returned to interface
        • Can be used to display many types of data (not restricted for use in build_data_table)
        • determines type of the content through a series of checks
          • If the passed reference is an array, it performs recursion on each element
          • If hash, outputs the key followed by the formatted content
          • To add new case, check for a uniquely identifying property of the data and then perform the desired action (e.g. run a corresponding MACRO on the data)
# e.g. Evidence data is always stored in a hash with a 'text' key and an 'evidence' key
# Therefore cell_content checks whether the evidence key is defined in the data to be outputted
# If it is, the evidence() macro is run with the data.
[% MACRO cell_content(cell) BLOCK;
     ...
     ELSIF cell.evidence.defined;
       evidence(cell.evidence,table_name _ index, cell.text, 'details');
     ...
END; %]

Other

Some other tools that you should be aware of but may not be required to know/use include:

  • JavaScript
  • MySQL
  • Xapian
  • GFF
  • Cytoscape Web
    • Cytoscape Tutorial
    • Plugin generally used for pathway analysis
    • This plugin is used in the Interactions widget on the Gene page ([WormBase dir]/root/templates/classes/gene/interactions.tt2)
    • Installation directory (for updating): [WormBase dir]/root/js/jquery/plugins/

General Concepts

Starting the server

  • In a terminal, connect to wb-dev (via ssh) and navigate to the root of your dev site (where you checked-out the git repository e.g. /usr/local/wormbase/website/[user]/)
  • Run the startup script: "./script/wormbase_server.pl -p ####" where #### is a port number
    • Typical options are -d for debugging and -r to auto-restart when changes to the code are detected
    • Output can be redirected into a log file ">>logs/server_log.log 2>&1" (redirects both output and errors into server_log.log)
    • Therefore a full command could be "./script/wormbase_server.pl -d -r -p 8023 >>logs/server_log.log 2>&1"
    • Logs can be monitored in real-time by running "tail -f logs/server_log.log" or "tail -f logs/wb-dev-catalyst.log" in a separate terminal
  • Check OICR Tutorials and Workshops for useful Unix commands (OICR login required)

Debugging Tools

  • Chrome provides a very useful set of tools for debugging that can be accessed by either 'Inspect Element' in the context(right-click) menu, or from Menu->Tools->Developer Tools (Shift+Ctrl+i)
  • Firefox has a "firebug" plugin that provides similar functionality
  • The tools can be used to monitor different aspects of the page
    • Network information such as GET/POST requests and their corresponding info (e.g. url and parameters)
    • Underlying HTML and scripts for the page

Widget Data Loading

Widget Loading

REST Controller:

  1. Catches internal url (/rest/widget/...)
  2. Determines the class and widget from the url
  3. From class and widget, determines which fields are required from the configuration file (wormbase.conf)
  4. API methods request data from appropriate databases, process/format/package the data, and then return it(/lib/WormBase/API/Object/[Class].pm)
  5. Sends data to the template to be used in rendering the widget (/root/templates/classes/[Class]/[Widget].tt2)

Tool Data Loading

Tool Loading

Typical flow of information:

  1. Request information such as the url and parameters are passed from the browser to the server
  2. Tools controller in server processes requests to /tools/... and calls corresponding API method
    • Determines tool and action from url (e.g. /tools/tree/run... => tool=tree, action=run)
      • Note: schema and gmap tools have similar function to other tools, tree and epic respectively
      • Any calls to these are translated to calls to the corresponding tool (e.g. schema->tree or gmap->epic)
    • Processes special parameters such as inline (for embedding in widget)
    • Calls the API method corresponding to the action (e.g. "run" in tree.pm)
  3. API method processes information passed from Controller and returns processed/packaged info to Tools Controller
    • Loads data passed from Controller(e.g. "my $name = $params->{'name'};" loads the name)
    • Loads appropriate data from databases (e.g. ACe object corresponding to the given 'name' & 'class' params)
    • Performs subsequent processing on data (e.g. blast tool runs blast with the given params)
    • Formats/packages data (e.g. commonly used method is _pack_obj which stores appropriate info for creating links for the interface)
      • Note: it is important to not return raw data (primarily ACe objects) to the interface. Two possible fixes are _pack_obj($object) or "$object" (stringify)
      • Note: it is also important to return "undef" instead of null or empty arrays/hashes
  4. Tools Controller passes API results to View (Template)
  5. View (Template) arranges/outputs the data for display
    • Uses various methods/macros
      • tag2link to convert a hash generated using _pack_obj into a link
      • build_data_table to generate a formatted data table
      • Note: common/useful macros can be found in /root/templates/config/main and /root/templates/shared/page_elements.tt2

Guides

Other Info

Useful files for debugging include:

  • /logs/wormbase-catalyst-access.log

Other:

  • If the port you have been using appears to be busy but the server is not running,
ps -aux|grep -####
kill -9 ID

where #### is the port number(e.g. 8023) and ID is the process id that is using the port(e.g. 10361)

  • To dump data in API methods include Data::Dumper
use Data::Dumper
...
warn(Dumper(\@data));
  • Sometimes firewall blocks certain ports, so you may be unable to connect to your dev server on the wb-dev machine through the browser
ssh -L 8080:localhost:#### wb-dev.oicr.on.ca
(where '####' is the port that you have been assigned by Todd).
You will be prompted to login.  Then cd to /usr/local/wormbase/website/your_name/ 
and type this command as usual:
./wormbase_server.pl -p #### -d -r
where #### is your desired port number. Then point your browser at localhost:8080 

Current Tasks

Additional tasks that require further investigation