Difference between revisions of "Entity classes and problems"

From WormBaseWiki
Jump to navigationJump to search
 
(21 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
[http://www.wormbase.org/wiki/index.php/Caltech_documentation Caltech_documentation] <br>
+
back to [http://wiki.wormbase.org/index.php/Caltech_documentation Caltech_documentation] <br>
 +
back to [http://wiki.wormbase.org/index.php/Pipeline_for_markup GSA markup SOP]
  
−
==GSA markup==
+
==Entities marked up==
−
* [http://www.wormbase.org/wiki/index.php/Pipeline_for_markup Pipeline_for_markup]
+
All WB entities can be viewed [http://dev.textpresso.org/gsa/worm/known_entities/ here]
−
* Entity classes and problems
 
−
** [http://www.wormbase.org/wiki/index.php/Papers_that_have_been_markedup Marked up papers]]
 
−
** [http://www.wormbase.org/wiki/index.php/Markup_excerpts Marked up excerpts]
 
  
−
==Classes of entities that are being marked up as of Jan 2010==
+
===Entity Sources===  
−
to see the actual entities go to http://dev.textpresso.org/gsa/known_entities/
+
Entity lists are available [http://textpresso-dev.caltech.edu/gsa/worm/known_entities/ here]
  
−
''text in italics signifies possible changes to the entity lists'' <br>
+
01downloadModEntities.pl downloads entities<br>
−
* Stable classes = entity lists do not change much
+
From acedb on spica
−
** Clone (cosmids) ''(injection plasmids?)''
+
*Anatomy_name
−
** Rearrangement
+
*Clone
−
** Anatomy_name  (problem terms “set”, “cell”, “ray”, “CAN”, “D”, needs rules) -> link [http://www.wormbase.org/wiki/index.php/Markup_excerpts here] for further examples
+
*Rearrangement
 +
*Strain
 +
*Variation
 +
From postgres on tazendra
 +
*Genes
 +
*Transgenes
  
−
* Constant growth classes = all new entities will eventually be added to WB
+
02formSortedLexicon.pl forms the lexicon from the known_entities.
−
** Variations ''(other species?)''
 
−
** Person (authors only)
 
−
** Transgene “Is”  ''(Ex and genomic expressions, e.g. sur-5::GFP?)''
 
  
−
* Monitored growth classes = entity names need curator approval
+
===URL constructor===
−
** Gene (manually map published,  unapproved, gene name to reported cds page, the link will only be available through the paper)
+
Based on WormBase protocol [http://www.wormbase.org/wiki/index.php/Linking_To_WormBase here], the URL constructor used in the linking script is<br>
−
** Strain ''(all strain names regardless of CGC, other species?)''
+
http://www.wormbase.org/db/get?name=X;class=Y
  
−
===Classes we are working on linking===
+
Currently we are forming URLs based on public names versus object class ID's.  These links are redirected to the correct page, which is the URL based on object ID.  By using the public name for the linking instead of the ID, we can avoid problems that may result in links dying due to objects getting merged or killed. In some cases, such as Phenotype, the URL is generated by a mapped WBPhenotypeID.
−
*Person - based on WBPersonID.  This is a problematic class since publication names are not consistent and a single person could have the same name, initials as someone else, or have multiple aliases. However, we will limit the linking to the authors on the paper only and work closely with our person curator to make sure these links are made properly.  
 
  
−
*Non-elegans genes - in general as long as the gene name is unique from elegans gene names, and exists in the database, this should not be a problem.  One error we ran into is that some names are similar and distinguishing between the elegans gene and the non elegans gene required a correctly formatted species prefix e.g. "Cbr-".  Formatting errors are not easily dealt with, at the moment, by automated scripts.
+
===Entity classes===
 +
'''Object''' -> ''source'' ; web page link <br>
 +
<example object page><br>
  
−
*Anatomy terms - this class was problematic due to the linking of the terms to common reagents or paper features, e.g. EMS is a cell but also a common mutagen. For now, as we will be monitoring papers manually, we can chart how often this happens and fix these by hand. We may need to implement context specificity for some of these terms.
+
'''Gene''' -> ''postgres on tazendra'' ; Gene Summary page URL uses public gene name, which is redirected to WBGeneID<br>
 +
<http://www.wormbase.org/db/get?name=mpk-1;class=Gene><br>
 +
<http://www.wormbase.org/db/get?name=F43C1.1;class=Gene><br>
  
−
*Transgene expressions - currently only the gene elements in the transgene genotype is marked up and is linked to the gene page for example, in the expression eor-1::GFP, only eor-1 is marked up.  This is fundamentally wrong as the entire expression should be linked to the transgene page.  There are a couple of issues to deal with for this class.
 
−
**First, not all transgenes are captured by our database.  Specifically, Ex arrays and plamids/cosmids used for injections are not captured.  This was an idealogical decision based on the fact that expression patterns for these elements are fleeting, unlike integrated arrays, so are not stable data elements to curate.
 
−
**Second, in cases where people name the elements using standardized nomenclature, creating a silent link is easy, but more often than not, people do not name these elements and thus a name has to be created for them.  Without the name, a link cannot be made.
 
  
−
----
+
'''Protein''' -> ''postgres on tazendra'' ; Gene Summary page<br>
 +
<http://www.wormbase.org/db/get?name=egl-4;class=Gene><br>
 +
<br>
  
−
==When entities do not exist in WB already==
+
'''Allele/SNP''' -> ''acedb on spica'' ; Variation Report page<br>
−
Currently, if an entity does not exist in WB, a silent link will be created based on the term in the paper.
+
<http://www.wormbase.org/db/get?name=ga117;class=Variation> <br>
−
*If the object is an allele, strain, or rearrangement, there should not be a problem as these are lab specific objects that would not conflict with the general public, as long as lab designations were used.
+
<http://www.wormbase.org/db/get?name=hw42941;class=Variation><br>
−
*In the case of a gene, there is the potential problem that the authors are using a gene name that has not been approved, and may conflict with other labs or objects in WB.  We will need to monitor these objects carefully and create a robust feedback system with our nomenclature curators. 
+
<br>
−
**At present, authors have been asked to declare new genes or  gene names so that we are alerted to these objects.
 
−
**In practice, most of the time, authors have already contacted our nomenclature curators in private and these gene names have been added to the database awaiting publication of the paper.
 
−
**However, there are occasions when authors have not contacted our nomenclature curators.  In practice, our pipeline has been to alert our curators when we come across such a gene and let them handle the approval and creation of the gene in our database.  The GSA linking pipeline creates a problem with this pipeline in that we will be creating a link no matter if our curators approve the gene name or not.  There has been one test of this early on in the project, which was not a problem.  We have now come across the case of a brigssae gene being named, but not existing in WormBase yet.  The immediate solution is to link the gene to the sequence page as this was included in the publication, but this is a manual task and is not amenable to our long-term goal of automating as much of the process as possible. 
 
  
−
----
+
'''Transgene''' -> ''postgres on tazendra'' ; Transgene Summary page <br>
−
==Dealing with ambiguities --link by query?==
+
<http://www.wormbase.org/db/get?name=oxIs12;class=Transgene><br>
−
I'd like to explore setting up a silent link based on a query. By using a query url, we may be able to avoid creating a page based on a name that will not be approved for a while, in addition, in terms of Person, if only one person has the name, the query will return the single page, whereas if there are multiple people, the query will return a list of possible people for the reader to look through. This avoids the creation of a wrong link. A query link may also be able to tap into other tags for a class, case in point would be to tap into genotype for a transgene, so that full genotype expressions can be linked to the transgene page without having to require the creation of a page based on the expression itself.
+
<br>
  
−
----
+
'''Rearrangement''' -> ''acedb on spica'' <br>
−
==Author responsibilities==
+
<br>
−
We have relied on authors to declare WB naive objects to us so that we can make sure they are properly linked.  However, authors do not always get back to us in time, and in cases when they do, they do not always tell us what we need to know, or tell us in a manner that we can automate the addition of their responses to our pipeline.  Manual oversight on author input will need to be provided on an ongoing basis. <br>
 
  
−
----
+
'''Strain''' -> ''acedb on spica''; Strain Report page<br>
−
==Problems==
+
<http://www.wormbase.org/db/get?name=MH37;class=Strain ><br>
−
Proper formatting in the xml that is given us is critical to smooth running of Arun's scripts.  That is, if any part of "cbr-unc-119" is not italicized properly, the scripts may not recognize the entity as a gene, or worse, may only recognize part of the entity. In this case, the link could potentially go to the elegans unc-119 page rather than the briggsae unc-119 page.  In addition, parentheses need to be formatted correctly as well,  that is, in the expression cpy-4(sy5022), cyp-4 and sy5022 need to be independently italicized for them to be recognized as entities to be linked to independent pages.  If the whole expression is italicized, it won't be recognized as a phrase to be linked, so no link will be created. 
+
<br>
  
−
----
+
'''Clone''' -> ''acedb on spica''; Sequence Summary page<br>
−
It is our hope that as object linking to this MOD becomes a familiar event to readers, authors will see the value of partaking in the exercise to declare approved new objects to us before the linking scripts are run.  Until then we will continue to rely on curator moderation, rather than author input, and full automation.
+
<http://www.wormbase.org/db/get?name=yk1106g06.3;class=Sequence><br>
 +
<http://www.wormbase.org/db/get?name=OSTR153G5_1;class=Sequence ><br>
 +
<br>
 +
 
 +
'''Phenotype''' ->
 +
 
 +
 
 +
==Classes to add==
 +
*Molecules/Drugs
 +
*Human Diseases
 +
*Non-elegans genes - in general as long as the gene name is unique from elegans gene names, and exists in the database, this should not be a problem. One error we ran into is that some names are similar and distinguishing between the elegans gene and the non elegans gene required a correctly formatted species prefix e.g. "Cbr-". Formatting errors are not easily dealt with by automated scripts.
 +
 
 +
==Suppressed classes==
 +
*''' Anatomy/cell''' -> ''acedb on spica'' with corrections by Raymond ; Summary of anatomy ontology term<br>
 +
<http://www.wormbase.org/db/get?name=HSN;class=Anatomy_term><br>
 +
<http://www.wormbase.org/db/get?name=Z1.ppp;class=Anatomy_term><br>
 +
<br>
 +
 
 +
This is a problematic class since
 +
**Many cells are named as single letters, ex. "E", "F", "B", etc., which results in many erroneous links 
 +
**There are many ambiguities: some cells are also names for common reagents or paper features, e.g., EMS is a cell but also a common mutagen. These names are put on an exclusion list, and if the cell exists in the paper, they are manually linked to the correct page.
 +
**Sex specific anatomy terms such as "Z4.appaap nucleus hermaphrodite" and "Z4.appaap nucleus male" will probably not be an entity that is linked as in general that term is not used in normal writing.  The best that we can do is to link such terms to the non-sex-specific anatomy page "Z4.appaap nucleus" where the user can choose the sex-specific term.
 +
 
 +
 
 +
*'''Author''' stopped linking 04/10/12<br>
 +
This is a problematic class since
 +
**publication names are not consistent
 +
**a single person could have the same name or initials as someone else
 +
**a single person could have multiple aliases. <br>
 +
 
 +
The linking was limited to the authors on the paper only and verified with the WB person curator to make sure the links were made properly.<br>
 +
 
 +
Links are formed using the URL pattern: <br>
 +
http://www.wormbase.org/db/misc/person?name=$url_encoded_name;paper=$wbpaper_id, <br>
 +
where $url_encoded_name is the full name of an author with the following modifications: <br>
 +
i) periods after middle names removed and <br>
 +
ii) spaces converted to %20, which is HTML equivalent of a space. <br>
 +
 
 +
Examples:<br>
 +
Name: Eve W. L. Chow <br>
 +
Link: http://www.wormbase.org/db/misc/person?name=Eve%20W%20L%20Chow;paper=WBPaper00038245
 +
 
 +
Name: I. Russel Lee <br>
 +
Link: http://www.wormbase.org/db/misc/person?name=I%20Russel%20Lee;paper=WBPaper00038245
 +
 
 +
(There may be also be other special characters with accents, umlauts, etc., The best thing to do would be to check WormBase and put the correct link, if the author name already exists. Else, the English equivalent of the special character could be visually determined and a new author link can be formed and notified to Cecilia.)
 +
 
 +
==Linking issues==
 +
===Loss of biological relevance===
 +
*'''Genes within transgene expressions linked to gene pages rather than to transgene pages''' - gene elements in the transgene genotype is marked up and is linked to the gene page, for example, in the expression eor-1::GFP, only eor-1 is marked up.  This is fundamentally wrong as the entire expression should be linked to the transgene page.  There are a couple of issues to deal with for this class.
 +
**Not all transgenes are captured by our database.  Specifically, Ex arrays and plasmids/cosmids used for injections are not captured since expression patterns for these elements are fleeting, unlike integrated arrays, so are not stable objects to curate.
 +
**In cases where people name the elements using standard nomenclature, creating a silent link is easy, but more often than not, people do not name these elements and thus a name has to be created for them.  Without the name, a link cannot be made.
 +
 
 +
===Entities don't exist in WB===
 +
*'''Gene names''' need to be approved by WB nomenclature curators. Authors should declare any new gene names before the linking scripts are run, but authors do not always follow through. One action is to link the name to the official WBGene sequence ID. This will require that the authors have explicitly stated what the gene sequence is. Linking the name to the gene should not affect what happens in acedb, the link lives within the online paper alone. If an author is not explicit, the best option is to contact the author, if there is no response within a reasonable time, no link will be created. This situation is not a frequent event.
 +
*'''Variations''' can be verified through the variation nameserver and a silent link made to the object. Alert the variation curator that the object is being created through the GSA JFP for that paper. Details for the variation can be entered through the form or through the Phenotype OA.
 +
*'''Other entities'''- check with the corresponding curator as to the feasibility to create a silent link.
 +
 
 +
===Formatting problems break recognition===
 +
Entity recognition relies on exact string matches and XML formatting cues.  If entities are not formatted correctly in the XML then the entity may not be recognized and linked. QCFast does not allow links to be made that do not follow XML formatting restrictions, to link these entities, it is necessary to download the html, manually add the link, then upload the fixed html.
 +
 
 +
* Case (a): <nowiki><i>Cbr-dpy-1</i>  instead of  Cbr-<i>dpy-1</i></nowiki> <br>
 +
::Otherwise the <nowiki><i></nowiki> tag becomes part of the string and string matching fails on Cbr-dpy-1, even if we have it in the wormbase entity list.
 +
 
 +
* Case (b): <nowiki><i>unc-119</i> instead of <i>unc</i>-<i>119</i></nowiki>
 +
 
 +
* Case (c): Parentheses shouldn't be italicized since they are used as an indicator of string starts and stops for entity recognition. <br>
 +
::<nowiki> <i>cbr-unc-119</i>(<i>st20000</i>) instead of <i>cbr-unc-119(st20000)</i></nowiki>
 +
 
 +
===Combined entity expressions===
 +
* double mutants, double alleles
 +
* dpy-11/12
 +
These cases also need to be dealt with by manually making changes to the XML.
 +
 
 +
 
 +
==Dealing with ambiguities ==
 +
===General ambiguities===
 +
**We've come across entities that get mistaken for entities of a different class, this was noticed for a couple of strain names being identical to clone names.
 +
**Jargon terminology:  EMS (embryonic cell) vs EMS (mutagen)
 +
In these cases the term is added to an exclusion from linking list.  It would be ideal to try to use some sort of context specific rules to try to make these terms less ambiguous.  For now, we can manually identify and link these terms when necessary.
 +
 
 +
===Author names===
 +
Person names are difficult to recognize and map to a WBPersonID in an automated fashion, because of the many variations in how an author lists their name, in addition, there are cases when more than one person shares the name, initials, etc.. We will deal with this by designing a URL that will first query for a person name in the WBPerson class, that matches the extracted author name, if one is not found then the URL will point to an author page.  Since all authors will get a page eventually, that link is guaranteed to be alive at some point. Q. how often do authors get linked with WBPersonID's?
 +
 
 +
===Gene names===
 +
We anticipate many cases where the name of a gene (or gene name) will be used across different species, making the identification of that gene problematic, not only in a single-species paper (e.g., MEX-1 exists as a name for a mammalian gene as well as for an elegans protein).  We will start to deal with this by identifying all similar gene names, including synonyms, between the different organisms (SGD, flybase, mouse)
 +
 
 +
==See also==
 +
More [http://wiki.wormbase.org/index.php/Markup_excerpts Problematic cases]
 +
Marked up papers [http://textpresso-dev.caltech.edu/gsa/worm/html/ here]
 +
 
 +
 
 +
[[Category:Curation]]
 +
[[Category:GSA_markup]]

Latest revision as of 17:21, 25 July 2013

back to Caltech_documentation
back to GSA markup SOP

Entities marked up

All WB entities can be viewed here

Entity Sources

Entity lists are available here

01downloadModEntities.pl downloads entities
From acedb on spica

  • Anatomy_name
  • Clone
  • Rearrangement
  • Strain
  • Variation

From postgres on tazendra

  • Genes
  • Transgenes

02formSortedLexicon.pl forms the lexicon from the known_entities.

URL constructor

Based on WormBase protocol here, the URL constructor used in the linking script is

http://www.wormbase.org/db/get?name=X;class=Y

Currently we are forming URLs based on public names versus object class ID's. These links are redirected to the correct page, which is the URL based on object ID. By using the public name for the linking instead of the ID, we can avoid problems that may result in links dying due to objects getting merged or killed. In some cases, such as Phenotype, the URL is generated by a mapped WBPhenotypeID.

Entity classes

Object -> source ; web page link
<example object page>

Gene -> postgres on tazendra ; Gene Summary page URL uses public gene name, which is redirected to WBGeneID
<http://www.wormbase.org/db/get?name=mpk-1;class=Gene>
<http://www.wormbase.org/db/get?name=F43C1.1;class=Gene>


Protein -> postgres on tazendra ; Gene Summary page
<http://www.wormbase.org/db/get?name=egl-4;class=Gene>

Allele/SNP -> acedb on spica ; Variation Report page
<http://www.wormbase.org/db/get?name=ga117;class=Variation>
<http://www.wormbase.org/db/get?name=hw42941;class=Variation>

Transgene -> postgres on tazendra ; Transgene Summary page
<http://www.wormbase.org/db/get?name=oxIs12;class=Transgene>

Rearrangement -> acedb on spica

Strain -> acedb on spica; Strain Report page
<http://www.wormbase.org/db/get?name=MH37;class=Strain >

Clone -> acedb on spica; Sequence Summary page
<http://www.wormbase.org/db/get?name=yk1106g06.3;class=Sequence>
<http://www.wormbase.org/db/get?name=OSTR153G5_1;class=Sequence >

Phenotype ->


Classes to add

  • Molecules/Drugs
  • Human Diseases
  • Non-elegans genes - in general as long as the gene name is unique from elegans gene names, and exists in the database, this should not be a problem. One error we ran into is that some names are similar and distinguishing between the elegans gene and the non elegans gene required a correctly formatted species prefix e.g. "Cbr-". Formatting errors are not easily dealt with by automated scripts.

Suppressed classes

  • Anatomy/cell -> acedb on spica with corrections by Raymond ; Summary of anatomy ontology term

<http://www.wormbase.org/db/get?name=HSN;class=Anatomy_term>
<http://www.wormbase.org/db/get?name=Z1.ppp;class=Anatomy_term>

This is a problematic class since

    • Many cells are named as single letters, ex. "E", "F", "B", etc., which results in many erroneous links
    • There are many ambiguities: some cells are also names for common reagents or paper features, e.g., EMS is a cell but also a common mutagen. These names are put on an exclusion list, and if the cell exists in the paper, they are manually linked to the correct page.
    • Sex specific anatomy terms such as "Z4.appaap nucleus hermaphrodite" and "Z4.appaap nucleus male" will probably not be an entity that is linked as in general that term is not used in normal writing. The best that we can do is to link such terms to the non-sex-specific anatomy page "Z4.appaap nucleus" where the user can choose the sex-specific term.


  • Author stopped linking 04/10/12

This is a problematic class since

    • publication names are not consistent
    • a single person could have the same name or initials as someone else
    • a single person could have multiple aliases.

The linking was limited to the authors on the paper only and verified with the WB person curator to make sure the links were made properly.

Links are formed using the URL pattern:
http://www.wormbase.org/db/misc/person?name=$url_encoded_name;paper=$wbpaper_id,
where $url_encoded_name is the full name of an author with the following modifications:
i) periods after middle names removed and
ii) spaces converted to %20, which is HTML equivalent of a space.

Examples:
Name: Eve W. L. Chow
Link: http://www.wormbase.org/db/misc/person?name=Eve%20W%20L%20Chow;paper=WBPaper00038245

Name: I. Russel Lee
Link: http://www.wormbase.org/db/misc/person?name=I%20Russel%20Lee;paper=WBPaper00038245

(There may be also be other special characters with accents, umlauts, etc., The best thing to do would be to check WormBase and put the correct link, if the author name already exists. Else, the English equivalent of the special character could be visually determined and a new author link can be formed and notified to Cecilia.)

Linking issues

Loss of biological relevance

  • Genes within transgene expressions linked to gene pages rather than to transgene pages - gene elements in the transgene genotype is marked up and is linked to the gene page, for example, in the expression eor-1::GFP, only eor-1 is marked up. This is fundamentally wrong as the entire expression should be linked to the transgene page. There are a couple of issues to deal with for this class.
    • Not all transgenes are captured by our database. Specifically, Ex arrays and plasmids/cosmids used for injections are not captured since expression patterns for these elements are fleeting, unlike integrated arrays, so are not stable objects to curate.
    • In cases where people name the elements using standard nomenclature, creating a silent link is easy, but more often than not, people do not name these elements and thus a name has to be created for them. Without the name, a link cannot be made.

Entities don't exist in WB

  • Gene names need to be approved by WB nomenclature curators. Authors should declare any new gene names before the linking scripts are run, but authors do not always follow through. One action is to link the name to the official WBGene sequence ID. This will require that the authors have explicitly stated what the gene sequence is. Linking the name to the gene should not affect what happens in acedb, the link lives within the online paper alone. If an author is not explicit, the best option is to contact the author, if there is no response within a reasonable time, no link will be created. This situation is not a frequent event.
  • Variations can be verified through the variation nameserver and a silent link made to the object. Alert the variation curator that the object is being created through the GSA JFP for that paper. Details for the variation can be entered through the form or through the Phenotype OA.
  • Other entities- check with the corresponding curator as to the feasibility to create a silent link.

Formatting problems break recognition

Entity recognition relies on exact string matches and XML formatting cues. If entities are not formatted correctly in the XML then the entity may not be recognized and linked. QCFast does not allow links to be made that do not follow XML formatting restrictions, to link these entities, it is necessary to download the html, manually add the link, then upload the fixed html.

  • Case (a): <i>Cbr-dpy-1</i> instead of Cbr-<i>dpy-1</i>
Otherwise the <i> tag becomes part of the string and string matching fails on Cbr-dpy-1, even if we have it in the wormbase entity list.
  • Case (b): <i>unc-119</i> instead of <i>unc</i>-<i>119</i>
  • Case (c): Parentheses shouldn't be italicized since they are used as an indicator of string starts and stops for entity recognition.
<i>cbr-unc-119</i>(<i>st20000</i>) instead of <i>cbr-unc-119(st20000)</i>

Combined entity expressions

  • double mutants, double alleles
  • dpy-11/12

These cases also need to be dealt with by manually making changes to the XML.


Dealing with ambiguities

General ambiguities

    • We've come across entities that get mistaken for entities of a different class, this was noticed for a couple of strain names being identical to clone names.
    • Jargon terminology: EMS (embryonic cell) vs EMS (mutagen)

In these cases the term is added to an exclusion from linking list. It would be ideal to try to use some sort of context specific rules to try to make these terms less ambiguous. For now, we can manually identify and link these terms when necessary.

Author names

Person names are difficult to recognize and map to a WBPersonID in an automated fashion, because of the many variations in how an author lists their name, in addition, there are cases when more than one person shares the name, initials, etc.. We will deal with this by designing a URL that will first query for a person name in the WBPerson class, that matches the extracted author name, if one is not found then the URL will point to an author page. Since all authors will get a page eventually, that link is guaranteed to be alive at some point. Q. how often do authors get linked with WBPersonID's?

Gene names

We anticipate many cases where the name of a gene (or gene name) will be used across different species, making the identification of that gene problematic, not only in a single-species paper (e.g., MEX-1 exists as a name for a mammalian gene as well as for an elegans protein). We will start to deal with this by identifying all similar gene names, including synonyms, between the different organisms (SGD, flybase, mouse)

See also

More Problematic cases Marked up papers here

Retrieved from "https://wiki.wormbase.org/index.php?title=Entity_classes_and_problems&oldid=19360"