Specifications for CCC Curation from Textpresso Search Page

Summary

These specifications are for allowing a curator to search any Textpresso implementation using the CCC categories, submit the resulting sentences to a curation form, make annotations, and download the annotations in either a simple three-column format or the GO's gene_association file (GAF) format.

Searches

Curators would be able to search Textpresso using their chosen criteria and export the sentences to the CCC curation form. The XML mark-up of each returned sentence will be used to populate the curation boxes on the form and color-code the search results.

Curators will need an option to send results to a curation form with a way to name the search so they can select it in the list of source files on the curation form.

Future options for filtering search results may include filtering papers that are in the corpus but not appropriate for curation (e.g., WB's 'functional annotation papers' or TAIR's black list of papers on other organisms - Tanya sent me a list of 1243 papers like this) or restricting searches to a particular level of SVM classification (e.g., high-confidence SVM papers only). For the former, would we need a consistent tag across all implementations? For the latter, what would be the best way to integrate the SVM classification results? Store them in postgres? Store them in Textpresso?

HMM: If you don't want to store it anywhere, there should be a textfield (to be filled with a URL, a filename on a local computer, or the actual list of papers) that indicates a list of papers (one ID per line, perhaps), that should be excluded. That way the filtering is highly customizable. There might be case where each curator want to exclude a different set of papers or change that set on a daily basis, and then it can be very tedious to keep track of everybody's needs.

KMV: Yes, I agree that the filtering should be as customizable as possible since curators will likely have different needs for their searches. With respect to the list of 1243 papers that Tanya sent me, I think these papers are akin to WormBase papers that are marked for 'functional annotation' only in that they (TAIR) currently don't want them included in any of their searches. For this, it seems that a tag or field that marks them appropriately in the database would be best, as opposed to always having a curator select or de-select a button on the search form.

HMM: OK, then we have both ways; Some filtering happens via using a information from a database, and there is an option to use a local list (file) customized by a curator.

KMV: Okay, that will be good.

For the SVM results, it seems that the information should also be stored, so curators could have the option to select what level of confidence that might want for their Textpresso searches, and even the option to only search the predicted negatives. Closer integration of Textpresso searches with SVM results will be very helpful. This gets done programatically for CCC right now and manually (by me) for the macromolecular interactions. For the long term, having the option to combine the SVMs with both Textpresso searches and with the HMMs would be ideal.

HMM: I'll put that on Textpresso's to-do list.

KMV: Great. I'm still thinking about how best to handle the already long, and continuously growing, list of SVM results. The confidence level and data type lists are fixed and relatively small, respectively, but we'll need a way to deal with the individual SVM results:

http://caprica.caltech.edu/celegans/svm_results/

In practice, how would curators select which results to search? For example, suppose I wanted to perform a Textpresso search on only those papers that were high confidence for other_expr in the last six months of 2010.

HMM: Still need to figure out where the filtering based on database information should happen. Is Juancarlos filtering them in out in the CCC form?

KMV: For the next TAIR search, perhaps we could just filter the sentences before they get included into the source files used for the curation form.

This issue is not such a big deal for the current elegans CCC curation, since the number of for 'functional annotation' only papers in postgres has plummeted. This issue is a bigger deal, though, when WB curators want to search the whole WB corpus for something.

From Searches to the Curation Form

This pipeline would make use of the XML format of returned sentences to construct a version of the current CCC curation form; a version for TAIR is shown here:

http://tazendra.caltech.edu/~azurebrd/cgi-bin/forms/tair/tair_ccc.cgi

1) Keep three action buttons on the top - Submit, Search for Sentence, Search for Paper

2) Display all information from within the <bibliography> tags.

This will display more information than is currently shown, but seemed easier than just picking a few tags from within the bibliography section. The additional information may also be helpful to curators.

3) Do not display information within any <field_references> tags.

Is this how scrambled sentences are typically identified?

HMM: No. Arun wrote a function that uses heuristics such as length of sentence, repetitious patterns within the sentence.

KMV: Okay, got it. I had forgotten how that worked. Thanks.

4) Potentially curatable sentences are found within the <field_results> tags. Working from left to right on the curation form, here is how the information from the XML would translate to the curation form.

First box, Gene/Protein Name

This box lists all entities within the species-specific protein or gene tag. Right now, the category name for this will be different for each implementation, for example:

protein_celegans

genes_arabidopsis

dicty_genes

For TAIR we displayed both the name of the gene as presented in the sentence, as well as the name in the sentence mapped to a specific TAIR gene ID. This is because some Arabidopsis gene names are used for more than one locus and the TAIR curators wanted a way to ensure they'd be making the annotation to the correct locus. It would be fine to implement this universally, I think. This requires a mapping file from each group and a regular pipeline for updating the gene or protein names file.

Second box, Component Term in Sentence

This box lists the component term as identified by the component category.

For the various implementations there are currently several component categories.

For elegans and Arabidopsis:

localization_cell_components_082208 CCC_TAIR

For dicty:

localization_cell_components_050808 localization_cell_components_082208

If more than one cellular component category is used in the mark-up, we will need to figure out how to determine which one the curator wants to display on the curation form. Alternatively, we could restrict each implementation to only one component category at a time.

HMM: This shouldn't be a problem unless I don't understand what you are saying.

KMV: Okay, I was referring to a situation where there might be a category localization_cell_component_082208 as well as CCC_TAIR and whether or not it'd be difficult to discern which one the curator wanted to display in the curation form. Could the information in the category selection boxes be used for this?

HMM: Selection box should work.

Currently, this box only displays the component terms in sentences that matched the category. It might be helpful, though, if in the future curators could enter new component terms displayed in the sentence but not in the category and have this entry somehow be added to the category for the next mark-up.

Third box, CC Term in GO

This box displays, if available, any GO terms that have already been curated from component terms in sentences, or allows curators to enter a new GO term, if needed.

The suggested GO terms come from the relationship index that has been built up over time from elegans curation. I think it'd be good to use one relationship index for all implementations to leverage community efforts.

Curation Actions and Sentence Display

The color-key displayed at the top of the sentence can stay.

Additional options for sentence display include being able to see surrounding sentences and being able to link out to the corresponding PDF with the appropriate sentences marked-up in the PDF.

The curator options are:

Curating

To make a new GO annotation, curators need to select, or create, an entry in each of the three boxes on the left-hand side of the form. Once a selection is made, the region will be highlighted in blue. For the last column, CC term in GO, curators can either select one of the suggested GO terms in the list (where is this relationship file located, i.e. its path --K), or enter a new one. Then select curate from the list of radio buttons above the sentence and, if you are ready to enter your annotations, click on Make connections:Submit at the top of the page.

If there is no list of GO terms in the third box, that means that a GO annotation using the term in the second box has not yet been made. In this case, the curator will need to enter the new GO term manually.

An autocomplete function for adding new GO terms would be good here. Also, we need a more efficient way to handle having to make duplicate annotations from one sentence. Right now I reload the sentence using the search sentence button, but it'd be good if curators could make more than one annotation from a sentence more easily.

The way the form currently works is that once a sentence has been annotated and the information submitted, the sentence is removed from the display. Curators may want to be able to see what they've annotated though, so we might need to come up with a different strategy. Also, since some of these source files can be quite large, curators may want to have a way to just pick up where they left off if we decide not to remove sentences after curation.

Marking Sentences Not Used for Curation

If an annotation cannot be made from a sentence, then curators may record the reason that an annotation was not made. Keeping track of these sentences will help build up a training set for improving search results. The different options for not curating are described below:

Already Curated/Already Done

If a curator does not wish to make another annotation for information previously curated, they can select the appropriate entity from each curation box and the Already Curated radio button.

The way this was supposed to work for elegans was that the information about previous curation would be entered by curators using this form, stored in a postgres table and then, when next presented, displayed at the bottom after the Already Done heading in red. I'd imagine that this feature is something groups may want to specify from the outset of their searches with options perhaps to include them in the curation boxes, show the information as Already Done but not include them in the curation boxes, or not show this information at all.

Scrambled Sentence

If, during the pdf-to-text conversion, a sentence has become scrambled, curators can mark it as such here. These are becoming much less frequent.

Should we label this Scrambled Sentence/Run-on Sentence to collect both, split these two or not worry about the run-on sentences? Having a run-on sentence does not necessarily preclude making an annotation, though, so these options might have to become check boxes instead of radio buttons.

HMM: In Textpresso, scrambled sentences are identified by an algorithm. This is not a machine learning algorithm, so Textpresso has no use for the sentences marked scrambled other than use the actual information in the interfaces.

KMV: Okay, then it sounds like specifically marking sentences as scrambled doesn't help Textpresso. I still think it'd be good for curators to distinguish scrambled sentences from non-scrambled false positives, though, as the non-scrambled false positives may be a good resource for category updating or finding terms to use to exclude sentences from the results.

HMM: It doesn't help in Textpresso's current implementation, however, if curators get annoyed with the current state-of-art, we might start implementing a new algorithm which most likely will be machine-learning based. For that we would need the scrambled sentences marked. So for future development, I'd like to continue to collect this information. (I actually don't know where they are stored right now, Juancarlos would need to help me on that.)

False Positive

If a returned sentence has nothing to do with subcellular localization, then it is marked as a false positive. For example:

In contrast, PP2AA3 rescues root tip organization weakly even when expression is driven by the RCN1 promoter, demonstrating a more stringent requirement for A subunit function in the root apical meristem.

These sentences could be saved in a flat file possibly for future category refinement.

Not GO Curatable

This is intended to mark sentences that may describe subcellular localization, but the information contained in them would not normally be curated for GO. For example, the localization described is for a mutant protein, or the localization is for the wild-type protein in a mutant background. An example sentence:

No alteration in expression levels of soluble GFP or GFP::RAB-3 was observed in the synapses, cell body or axon in uba-1 animals (Figure 1A: d1-d6, f1-f6, 1C, 1D, Figure S4F, S4G, S4H).

Like the false positives, these sentences could be used to refine searches, e.g. to develop lists of terms or HMMs, that could be used to filter potentially uncuratable sentences from the search results.

Dumping Annotations

Annotations will be stored in postgres tables with two options available for dumping them out into tab-delimited text files. These options would be presented as buttons on the curation form.

Here are examples of what would be contained in the files using the sample sentence below:

SentenceID 9 -- S 7 P 43065 S s3 E The first enzyme, gamma-glutamate cysteine ligase (GSH1), responsible for synthesis of gamma-glutamylcysteine (gamma-EC), is, in Arabidopsis, exclusively located in the plastids, whereas the second enzyme, glutathione synthetase (GSH2), is located in both plastids and cytosol.

1) Three-column, tab-delimited format

1) 3-column user submission file

Locus Name	GO ID	Paper ID
Column 1 of ftp mapping file	GO:0009536	PMID:nnnnnnnn or TAIR:43065

AT4G23100 GO:0009536 TAIR:43065

AT5G27380 GO:0009536 TAIR:43065

AT5G27380 GO:0005829 TAIR:43065

For column 1, map the gene symbol (second column) to the locus name (first column) in the gene_aliases file:

ftp://ftp.arabidopsis.org/home/tair/Genes/gene_aliases.20101027

For column 2, use the geneontology.obo file to map the GO term to the GO ID:

http://www.geneontology.org/ontology/obo_format_1_2/gene_ontology_ext.obo

For column 3, use the PMID (preferred) or the TAIR paper ID, pipe separated, prefaced PMID: and TAIR: respectively.

This information could be taken from the bibliography XML.

2) 18-column, tab-delimited GAF 2.0

Column	Content	Required	Cardinality	TAIR Entry
1	DB	Required	1	TAIR
2	DB Object ID	Required	1	Column 1 of gene_aliases file
3	DB Object Symbol	Required	1	Column 2 gene_aliases file
4	Qualifier	Optional	0 or greater	NULL
5	GO ID	Required	1	GO:0005654
6	DB Reference	Required	1 or greater	PMID:21074051 or, if no PMID, TAIR:42184
7	Evidence Code	Required	1	IDA
8	With or From	Optional	0 or greater	NULL
9	Aspect	Required	1	C
10	DB Object Name	Optional	0 or 1	Column 3 of gene_aliases file
11	DB Object Synonym	Optional	0 or greater	ASK TANYA
12	DB Object Type	Required	1	protein
13	taxon)	Required	1 or 2	taxon:3702
14	Date	Required	1	Date annotation is made
15	Assigned By	Required	1	TAIR
16	Annotation Extension	Optional	0 or greater	NULL
17	Gene Product Form ID	Optional	0 or greater	NULL

Right now, TAIR only needs the simple, three-column format. Some elements of the Gene Association File Format, such as Taxon ID, will need to be hard-coded for each database, so we will have to store that information somewhere.

Back to Gene Ontology