Difference between revisions of "New 2012 Curation Status"

From WormBaseWiki
Jump to navigationJump to search
Line 178: Line 178:
  
  
= Loading Data =
+
= Loading Data into the Form =
  
 
On the ''Curation Statistics Options Page'', the ''Specific Paper Page'', or the ''Pre-populated Specific Paper Page'', curators have the option to specify what flagging methods (SVM, AFP, and/or CFP), curation sources (Ontology Annotator or cur_curdata [which is the data generated from this form]), and/or data types (e.g. geneint, rnai) they would like to view.
 
On the ''Curation Statistics Options Page'', the ''Specific Paper Page'', or the ''Pre-populated Specific Paper Page'', curators have the option to specify what flagging methods (SVM, AFP, and/or CFP), curation sources (Ontology Annotator or cur_curdata [which is the data generated from this form]), and/or data types (e.g. geneint, rnai) they would like to view.
  
cur_curdata: this captures all data entered through this form, meaning paper ID, data type, curator ID, validation status (e.g. "curated and positive"), pre-canned comment, and/or free text comment (and timestamp). Note: this table only stores data (and associated paper-data-type pairs) that has been manually entered through this form.
+
There are separate hashes for storing the different types of data, all of which have a key of datatype, subkey paperID, sub-subkeys of other things depending on the hash (see individual subsections below).
 +
 
 +
There is an option to select specific datatype, in which case only the data for those datatypes is loaded.  Similarly if only some paperIDs have been selected, only those paperIDs are loaded.
 +
 
 +
== Loading curatable papers ==
 +
 
 +
Only papers that have a 'valid' pap_status value and a 'primary' pap_primary_data value are considered curatable. These are stored in the %curatablePapers hash.
 +
 
 +
<pre>
 +
sub populateCuratablePapers {
 +
  my $query = "SELECT * FROM pap_status WHERE pap_status = 'valid' AND joinkey IN (SELECT joinkey FROM pap_primary_data WHERE pap_primary_data = 'primary')";
 +
  $result = $dbh->prepare( $query );
 +
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
 +
  while (my @row = $result->fetchrow) { $curatablePapers{$row[0]} = $row[1]; }
 +
} # sub populateCuratablePapers
 +
</pre>
 +
 
 +
== Loading afp_ data ==
 +
 
 +
for each of the chose datatypes, if they are allowed in %datatypeAfpCfp, query the corresponding afp_ postgres table, and if it's a curatable paper store the value in the %afpData hash (data type, paper ID, AFP result).
 +
 
 +
Query afp_email and if it's a curatable paper store in %afpEmailed hash (for afp emailed statistics)
 +
Query afp_lasttouched
 +
 
 +
<pre>
 +
sub populateAfpData {
 +
  foreach my $datatype (sort keys %chosenDatatypes) {
 +
    next unless $datatypesAfpCfp{$datatype};
 +
    my $pgtable_datatype = $datatypesAfpCfp{$datatype};
 +
    $result = $dbh->prepare( "SELECT * FROM afp_$pgtable_datatype" );
 +
    $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
 +
    while (my @row = $result->fetchrow) {
 +
      next unless ($curatablePapers{$row[0]});
 +
      $afpData{$datatype}{$row[0]} = $row[1]; }
 +
  } # foreach my $datatype (sort keys %chosenDatatypes)
  
 +
  $result = $dbh->prepare( "SELECT * FROM afp_email" );
 +
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
 +
  while (my @row = $result->fetchrow) {
 +
    next unless ($curatablePapers{$row[0]});
 +
    $afpEmailed{$row[0]}++; }
 +
  $result = $dbh->prepare( "SELECT * FROM afp_lasttouched" );
 +
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
 +
  while (my @row = $result->fetchrow) {
 +
    next unless ($curatablePapers{$row[0]});
 +
    foreach my $datatype (sort keys %chosenDatatypes) {
 +
      $afpFlagged{$datatype}{$row[0]}++; } }
 +
  foreach my $datatype (sort keys %chosenDatatypes) {
 +
    foreach my $joinkey (sort keys %{ $afpFlagged{$datatype} }) {
 +
      if ($afpData{$datatype}{$joinkey}) { $afpPos{$datatype}{$joinkey}++; }
 +
        else { $afpNeg{$datatype}{$joinkey}++; } } }
 +
} # sub populateAfpData
 +
</pre>
  
== Loading Curator Data from Curation Status ==
+
== Loading cur_curdata ==
  
 +
cur_curdata: this captures all data entered through this form, meaning paper ID, data type, curator ID, validation status (e.g. "curated and positive"), pre-canned comment, and/or free text comment (and timestamp). Note: this table only stores data (and associated paper-data-type pairs) that has been manually entered through this form.
  
 
Code:
 
Code:
Line 208: Line 260:
 
When populating curator data from curation status, read the cur_curdata postgres table, skip datatypes that were not chosen, skip papers that were not chosen.
 
When populating curator data from curation status, read the cur_curdata postgres table, skip datatypes that were not chosen, skip papers that were not chosen.
 
Store data in the %curData hash, key is datatype, subkey is paperID, then valuekeys are curator, donposneg (curator result of curated, validatedPos, validatedNeg, notValidated), select comment, text comment, timestamp.
 
Store data in the %curData hash, key is datatype, subkey is paperID, then valuekeys are curator, donposneg (curator result of curated, validatedPos, validatedNeg, notValidated), select comment, text comment, timestamp.
 +
 +
cur_curdata can only have one result for a specific paper-data-type pair, if a new result is entered it will overwrite the previous result.

Revision as of 23:28, 5 December 2012

Curation Status & Statistics Form (2012)

The sandbox/testing form can be found here


The CGI code is located on Tazendra/Mangolassi here:

/home/postgres/public_html/cgi-bin/curation_status.cgi



Precanned Comments

The keys stored in postgres for the hashes of these values are only numbers, so the descriptions/titles can change or be updated and still apply retroactively.

Code:

sub populatePremadeComments {
  $premadeComments{"1"} = "SVM Positive, Curation Negative";
  $premadeComments{"2"} = "pre-made comment #2";
  $premadeComments{"3"} = "pre-made comment #3";}

So, as of now:

| Key | Comment |
| 1 | "SVM Positive, Curation Negative" |
| 2 | "pre-made comment #2" |
| 3 | "pre-made comment #3" |


Hence, if a completely new comment is desired, a new key will need to be made and there after associated with that new comment. Also, old keys should never be recycled and documentation describing what each key refers to should be maintained in this Wiki.


New Result

Each paper-data-type pair can be assigned a 'New Result' indicating its status as curated (or not) or validated (or not), and if validated, positive or negative for the particular paper-data-type pair. These results can be entered via the "Add Results Page" or directly in the "<Get Results Page>" via the "New Results" column. The code is below:

Code:

sub populateDonPosNegOptions {
  $donPosNegOptions{""}             = "";
  $donPosNegOptions{"curated"}      = "curated and positive";
  $donPosNegOptions{"positive"}     = "validated positive";
  $donPosNegOptions{"negative"}     = "validated negative";
  $donPosNegOptions{"notvalidated"} = "not validated";}

where "curated", "positive", "negative", and "notvalidated" are the keys (for the %donPosNegOptions hash table in the form code) that will be stored in postgres and the corresponding values (e.g. "curated and positive") are what will be displayed on the form.

Note that "" and "not validated" represent no data for that paper-data-type pair, but "not validated" is present as an option to overwrite accidental validations (it is impossible to go back to a blank "" field via the form).


Data Types

The form determines which data types exist via a 'populateDatatypes' subroutine in the form code. As of 12-5-2012, the form first collects all data types used in SVM from the 'cur_svmdata' postgres table (which, as of 12-5-2012, all also are identically named in the Author First Pass (AFP) and Curator First Pass (CFP) tables) and then supplements with other data types not in SVM but in AFP and CFP (as of 12-5-2012, all anatomy curation related data types) plus one additional data type ("geneticablation") not in SVM, AFP, or CFP.

Here is the code:

sub populateDatatypes {
  $result = $dbh->prepare( "SELECT DISTINCT(cur_datatype) FROM cur_svmdata " );
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  while (my @row = $result->fetchrow) { $datatypesAfpCfp{$row[0]} = $row[0]; }
  $datatypesAfpCfp{'blastomere'}    = 'cellfunc';
  $datatypesAfpCfp{'exprmosaic'}    = 'siteaction';
  $datatypesAfpCfp{'geneticmosaic'} = 'mosaic';
  $datatypesAfpCfp{'laserablation'} = 'ablationdata';
  foreach my $datatype (keys %datatypesAfpCfp) { $datatypes{$datatype}++; }
  $datatypes{'geneticablation'}++;
} # sub populateDatatypes


As for the data types currently (12-5-2012) NOT in SVM but IN AFP and CFP, the data type name is different between the Curation Status form and the AFP and CFP forms. So, the data types named "cellfunc", "siteaction", "mosaic", and "ablationdata" in the AFP and CFP tables are respectively named "blastomere", "exprmosaic", "geneticmosaic", "laserablation" in the Curation Status form.

The IMPORTANT thing here is: if, at some point, the data types are changed (added, renamed, etc.), and the code is not updated in kind, the form will likely break. Curators should tell Juancarlos/Chris/Daniela to update the code.

new datatypes should be accounted in this code :

  • - no svm, no afp/cfp : add to %datatypes hash like 'geneticablation'.
  • - no svm, yes afp/cfp : add to %datatypesAfpCfp + %datatypes hashes like 'blastomere'
  • - yes svm, yes afp/cfp : add to code to populate cur_svmdata, which will populate in the SELECT query
  • - yes svm, no afp/cfp : add to code to populate cur_svmdata, which will populate in the SELECT query, but also subsequently delete from %datatypesAfpCfp (to prevent a postgres query to a non-existing table which will crash the form)


Creating PDF links to papers

In the <"Get Results Page"> each paper ID is linked to its corresponding PDF document using the code below:

Code:

sub populatePdf {
  $result = $dbh->prepare( "SELECT * FROM pap_electronic_path WHERE pap_electronic_path IS NOT NULL");
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  my %temp;
  while (my @row = $result->fetchrow) {
    my ($data, $isPdf) = &makePdfLinkFromPath($row[1]);
    $temp{$row[0]}{$isPdf}{$data}++; }
  foreach my $joinkey (sort keys %temp) {
    my @pdfs;
    foreach my $isPdf (reverse sort keys %{ $temp{$joinkey} }) {
      foreach my $pdfLink (sort keys %{ $temp{$joinkey}{$isPdf} }) {
        push @pdfs, $pdfLink; } }
    my ($pdfs) = join"<br/>", @pdfs;
    $pdf{$joinkey} = $pdfs;
  } # foreach my $joinkey (sort keys %temp)
} # sub populatePdf

sub makePdfLinkFromPath {
  my ($path) = shift;
  my ($pdf) = $path =~ m/\/([^\/]*)$/;
  my $isPdf = 0; if ($pdf =~ m/\.pdf$/) { $isPdf++; }           # kimberly wants .pdf files on top, so need to flag to sort
  my $link = 'http://tazendra.caltech.edu/~acedb/daniel/' . $pdf;
  my $data = "<a href=\"$link\" target=\"new\">$pdf</a>"; return ($data, $isPdf); }


Note the table name ("pap_electronic_path"), the URL path ("http://tazendra.caltech.edu/~acedb/daniel/"), and (because of the code 'target=\"new\"') that the link will open a new window or tab. Also note that opening another link on the original page (e.g. <"Get Results Page">) will open that link in that same new window/tab, clearing out what you had opened previously.


Creating hyperlinks to PubMed paper pages

In the <"Get Results Page"> each PubMed ID is linked to its corresponding PubMed webpage using the code below:

Code:

sub populatePmid {
  $result = $dbh->prepare( "SELECT * FROM pap_identifier WHERE pap_identifier ~ 'pmid'" );
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  my %temp;
  while (my @row = $result->fetchrow) { if ($row[0]) {
    my ($data) = &makeNcbiLinkFromPmid($row[1]);
    $temp{$row[0]}{$data}++; } }
  foreach my $joinkey (sort keys %temp) {
    my ($pmids) = join"<br/>", keys %{ $temp{$joinkey} };
    $pmid{$joinkey} = $pmids;
  } # foreach my $joinkey (sort keys %temp)
} # sub populatePmid
sub makeNcbiLinkFromPmid {
  my $pmid = shift;
  my ($id) = $pmid =~ m/(\d+)/;
  my $link = 'http://www.ncbi.nlm.nih.gov/pubmed/' . $id;
  my $data = "<a href=\"$link\" target=\"new\">$pmid</a>"; return $data; }

Note the table name ("pap_identifier"), the table specifier ("WHERE pap_identifier ~ 'pmid'"), the URL path ("http://www.ncbi.nlm.nih.gov/pubmed/"), and (because of the code 'target=\"new\"') that the link will open a new window or tab. Also note that opening another link on the original page (e.g. <"Get Results Page">) will open that link in that same new window/tab, clearing out what you had opened previously.


Populating the Journal Names

Journal names for each paper are populated via the following code:

sub populateJournal {
  $result = $dbh->prepare( "SELECT * FROM pap_journal WHERE pap_journal IS NOT NULL" );
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  while (my @row = $result->fetchrow) { if ($row[0]) { $journal{$row[0]} = $row[1]; } }
} # sub populateJournal


Note the table "pap_journal".


Loading Data into the Form

On the Curation Statistics Options Page, the Specific Paper Page, or the Pre-populated Specific Paper Page, curators have the option to specify what flagging methods (SVM, AFP, and/or CFP), curation sources (Ontology Annotator or cur_curdata [which is the data generated from this form]), and/or data types (e.g. geneint, rnai) they would like to view.

There are separate hashes for storing the different types of data, all of which have a key of datatype, subkey paperID, sub-subkeys of other things depending on the hash (see individual subsections below).

There is an option to select specific datatype, in which case only the data for those datatypes is loaded. Similarly if only some paperIDs have been selected, only those paperIDs are loaded.

Loading curatable papers

Only papers that have a 'valid' pap_status value and a 'primary' pap_primary_data value are considered curatable. These are stored in the %curatablePapers hash.

sub populateCuratablePapers {
  my $query = "SELECT * FROM pap_status WHERE pap_status = 'valid' AND joinkey IN (SELECT joinkey FROM pap_primary_data WHERE pap_primary_data = 'primary')";
  $result = $dbh->prepare( $query );
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  while (my @row = $result->fetchrow) { $curatablePapers{$row[0]} = $row[1]; }
} # sub populateCuratablePapers

Loading afp_ data

for each of the chose datatypes, if they are allowed in %datatypeAfpCfp, query the corresponding afp_ postgres table, and if it's a curatable paper store the value in the %afpData hash (data type, paper ID, AFP result).

Query afp_email and if it's a curatable paper store in %afpEmailed hash (for afp emailed statistics) Query afp_lasttouched

sub populateAfpData {
  foreach my $datatype (sort keys %chosenDatatypes) {
    next unless $datatypesAfpCfp{$datatype};
    my $pgtable_datatype = $datatypesAfpCfp{$datatype};
    $result = $dbh->prepare( "SELECT * FROM afp_$pgtable_datatype" );
    $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
    while (my @row = $result->fetchrow) {
      next unless ($curatablePapers{$row[0]});
      $afpData{$datatype}{$row[0]} = $row[1]; }
  } # foreach my $datatype (sort keys %chosenDatatypes)

  $result = $dbh->prepare( "SELECT * FROM afp_email" );
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  while (my @row = $result->fetchrow) {
    next unless ($curatablePapers{$row[0]});
    $afpEmailed{$row[0]}++; }
  $result = $dbh->prepare( "SELECT * FROM afp_lasttouched" );
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  while (my @row = $result->fetchrow) {
    next unless ($curatablePapers{$row[0]});
    foreach my $datatype (sort keys %chosenDatatypes) {
      $afpFlagged{$datatype}{$row[0]}++; } }
  foreach my $datatype (sort keys %chosenDatatypes) {
    foreach my $joinkey (sort keys %{ $afpFlagged{$datatype} }) {
      if ($afpData{$datatype}{$joinkey}) { $afpPos{$datatype}{$joinkey}++; }
        else { $afpNeg{$datatype}{$joinkey}++; } } }
} # sub populateAfpData

Loading cur_curdata

cur_curdata: this captures all data entered through this form, meaning paper ID, data type, curator ID, validation status (e.g. "curated and positive"), pre-canned comment, and/or free text comment (and timestamp). Note: this table only stores data (and associated paper-data-type pairs) that has been manually entered through this form.

Code:

sub populateCurCurData {
  $result = $dbh->prepare( "SELECT * FROM cur_curdata ORDER BY cur_timestamp" );        # in case multiple values get in for a paper-datatype (shouldn't happen), keep the latest
  $result->execute() or die "Cannot prepare statement: $DBI::errstr\n";
  while (my @row = $result->fetchrow) {
    next unless ($chosenPapers{$row[0]} || $chosenPapers{all});
    next unless ($chosenDatatypes{$row[1]});
    $curData{$row[1]}{$row[0]}{curator}    = $row[2];
    $curData{$row[1]}{$row[0]}{donposneg}  = $row[3];
    $curData{$row[1]}{$row[0]}{selcomment} = $row[4];
    $curData{$row[1]}{$row[0]}{txtcomment} = $row[5];
    $curData{$row[1]}{$row[0]}{timestamp}  = $row[6]; }
} # sub populateCurCurData


When populating curator data from curation status, read the cur_curdata postgres table, skip datatypes that were not chosen, skip papers that were not chosen. Store data in the %curData hash, key is datatype, subkey is paperID, then valuekeys are curator, donposneg (curator result of curated, validatedPos, validatedNeg, notValidated), select comment, text comment, timestamp.

cur_curdata can only have one result for a specific paper-data-type pair, if a new result is entered it will overwrite the previous result.