NGASP Results

From WormBaseWiki
(Redirected from Results)
Jump to: navigation, search

NGASP Gene prediction competition

The nGASP project parallels recent computational prediction initiatives including CASP, GASP, and EGASP. Participation is open to all academic, private sector, and government researchers and results will be made immediately available through a public database. A summary of the results will be submitted for peer-reviewed publication.

For nGASP, a set of regions representing ~10% (10 Mb) of the C. elegans genome release WS160 was selected to evaluate the performance of the participating gene predictors. We have selected two sets of regions: the training set (10 Mb) is to be used in training your software (if needed) and includes curated gene models from WormBase. We will used the test set (also 10 Mb) for evaluating gene prediction software.


Initial results for NGASP

These are the nucleotide-level and exon-level results for NGASP. Transcript-level and gene-level results will be announced soon.

The nucleotide-level and exon-level results were calculated by using a benchmark consisting of all confirmed and non-confirmed isoforms in all genes in the test set regions.

In this analysis, when we say "exon-level" accuracy we are referring to the accuracy in predicting coding exons (CDSs). Accuracy in predicting UTRs is not considered in these results (but will be included in the final NGASP results, to be announced soon).

In the data below, "Sn" is an abbreviation for sensitivity and "Sp" is an abbreviation for specificity. The genefinders in each category are ranked according to the average of their sensitivity and specificity.

The numbers given below are percentages. In the names of the programs we give the NGASP category and version. For example, "MGENE_1_2" refers to the MGENE category 1, version 2 submitted gene set.

These results have been confirmed by two different evaluation codes written by different individuals, and therefore we are confident that they are correct. However, if you notice any anomalies with these results, please contact the NGASP organisers.

We thank all participants for taking part in NGASP, and will be announcing further results (eg. transcript-level and gene-level results) soon.

Nucleotide-level results for NGASP

Sn      = Sensitivity

Sp = Specificity average = (Sn + Sp)/2

Category 1 (ab initio) genefinders

MGENE_1_1 average=92.31 Sn=93.14 Sp=91.48
MGENE_1_3 average=92.15 Sn=92.71 Sp=91.59
MGENE_1_2 average=92.14 Sn=92.67 Sp=91.60
FGENESH_1_1 average=91.16 Sn=95.20 Sp=87.11
CRAIG_1_1 average=91.03 Sn=91.15 Sp=90.91
GLIMMERHMM_1_1 average=90.45 Sn=93.28 Sp=87.62
AUGUSTUS_1_1 average=90.34 Sn=91.66 Sp=89.02
AUGUSTUS_1_2 average=90.31 Sn=91.31 Sp=89.30
GENEMARKHMM_1_1 average=88.91 Sn=94.76 Sp=83.06
EXONHUNTER_1_1 average=88.71 Sn=91.38 Sp=86.03
SNAP_1_1 average=87.46 Sn=90.45 Sp=84.47
GENEID_1_1 average=87.41 Sn=86.57 Sp=88.24
EUGENE_1_1 average=87.07 Sn=84.65 Sp=89.48
AGENE_1_1 average=84.22 Sn=85.02 Sp=83.41

Category 2 (dual/multi-genome) genefinders

MGENE_2_2 average=92.33 Sn=93.74 Sp=90.91
MGENE_2_1 average=92.30 Sn=93.68 Sp=90.91
EUGENE_2_1 average=89.35 Sn=91.21 Sp=87.48
NSCAN_2_1 average=89.33 Sn=90.59 Sp=88.07
SGP2_2_1 average=86.93 Sn=83.86 Sp=89.99

Category 3 genefinders (use alignments of proteins, ESTs, or mRNAs

MGENE_3_3 average=92.54 Sn=93.19 Sp=91.88
MGENE_3_1 average=92.52 Sn=93.15 Sp=91.89
FGENESH++_3_1 average=92.33 Sn=94.95 Sp=89.70
AUGUSTUS_3_1 average=92.28 Sn=94.03 Sp=90.52
GRAMENE_3_2 average=91.49 Sn=88.21 Sp=94.77
MGENE_3_2 average=91.40 Sn=94.93 Sp=87.86
GRAMENE_3_1 average=90.82 Sn=86.21 Sp=95.42
EXONHUNTER_3_1 average=90.21 Sn=93.10 Sp=87.31
EUGENE_3_2 average=89.97 Sn=94.84 Sp=85.09
EUGENE_3_1 average=89.69 Sn=94.08 Sp=85.30
MAKER_3_1 average=85.71 Sn=82.92 Sp=88.50
MAKER_3_2 average=84.67 Sn=78.28 Sp=91.05
EXONHUNTER_3_2 average=81.49 Sn=71.02 Sp=91.96

Category 4 genefinders (combiners)

JIGSAW_4_2 average=93.80 Sn=95.93 Sp=91.66
JIGSAW_4_1 average=93.45 Sn=93.68 Sp=93.22
EVIGAN_4_1 average=93.43 Sn=97.27 Sp=89.59
GENEID_4_2 average=93.27 Sn=94.57 Sp=91.97
GENEID_4_1 average=93.23 Sn=94.96 Sp=91.50
FGENESH++C_4_1 average=92.60 Sn=95.51 Sp=89.68
GENOMIX_4_2 average=92 Sn=93.60 Sp=90.40
GLEAN_4_1 average=91.88 Sn=96.48 Sp=87.28
EUGENE_4_2 average=90.41 Sn=95.46 Sp=85.35
EUGENE_4_4 average=90.24 Sn=95.14 Sp=85.34
EUGENE_4_1 average=90.24 Sn=94.88 Sp=85.60
EUGENE_4_3 average=90.05 Sn=94.50 Sp=85.60
GENOMIX_4_1 average=89.84 Sn=91.13 Sp=88.55
GESECA_4_1 average=89.79 Sn=96.74 Sp=82.84
GRAMENE_4_1 average=85.61 Sn=90.33 Sp=80.89

Exon-level results for NGASP

Sn      = Sensitivity

Sp = Specificity average = (Sn + Sp)/2

Category 1 (ab initio) genefinders

MGENE_1_1 average=78.12 Sn=77.65 Sp=78.58
MGENE_1_2 average=77.87 Sn=77.04 Sp=78.70
MGENE_1_3 average=77.86 Sn=77.08 Sp=78.63
FGENESH_1_1 average=76.08 Sn=78.60 Sp=73.55
CRAIG_1_1 average=75.63 Sn=73.11 Sp=78.15
AUGUSTUS_1_2 average=74.55 Sn=74.76 Sp=74.33
AUGUSTUS_1_1 average=74.33 Sn=76.10 Sp=72.55
GLIMMERHMM_1_1 average=73.84 Sn=76.30 Sp=71.37
GENEMARKHMM_1_1 average=71.02 Sn=76.45 Sp=65.58
EUGENE_1_1 average=70.11 Sn=67.22 Sp=73.00
GENEID_1_1 average=67.34 Sn=66.05 Sp=68.63
EXONHUNTER_1_1 average=65.12 Sn=67.70 Sp=62.53
SNAP_1_1 average=64.61 Sn=67.92 Sp=61.30
AGENE_1_1 average=60.40 Sn=59.71 Sp=61.09

Category 2 (dual/multi-genome) genefinders

MGENE_2_1 average=78.58 Sn=78.80 Sp=78.35
MGENE_2_2 average=78.57 Sn=78.84 Sp=78.30
EUGENE_2_1 average=73.17 Sn=73.51 Sp=72.82
NSCAN_2_1 average=71.67 Sn=72.50 Sp=70.83
SGP2_2_1 average=67.22 Sn=64.17 Sp=70.27

Category 3 genefinders (using alignments of proteins, ESTs, or mRNAs)

FGENESH++_3_1 average=82.50 Sn=84.06 Sp=80.93
AUGUSTUS_3_1 average=81.40 Sn=82.59 Sp=80.20
MGENE_3_1 average=80.45 Sn=80.22 Sp=80.67
MGENE_3_3 average=80.44 Sn=80.27 Sp=80.61
MGENE_3_2 average=78.58 Sn=81.27 Sp=75.88
EUGENE_3_1 average=76.14 Sn=80.05 Sp=72.22
EUGENE_3_2 average=76.07 Sn=81.81 Sp=70.32
GRAMENE_3_1 average=72.93 Sn=74.09 Sp=71.76
EXONHUNTER_3_1 average=72.26 Sn=75.19 Sp=69.33
GRAMENE_3_2 average=71.77 Sn=75.76 Sp=67.77
EXONHUNTER_3_2 average=67.08 Sn=57.23 Sp=76.92
MAKER_3_1 average=65.63 Sn=64.99 Sp=66.27
MAKER_3_2 average=65.49 Sn=61.48 Sp=69.49

Category 4 genefinders (combiners)

JIGSAW_4_1 average=85.28 Sn=83.20 Sp=87.36
EVIGAN_4_1 average=84.43 Sn=86.54 Sp=82.31
FGENESH++C_4_1 average=84.39 Sn=86.07 Sp=82.70
GENEID_4_1 average=84.27 Sn=84.75 Sp=83.78
GENEID_4_2 average=84.19 Sn=83.35 Sp=85.03
JIGSAW_4_2 average=83.16 Sn=83.28 Sp=83.04
GENOMIX_4_2 average=83.04 Sn=82.55 Sp=83.53
EUGENE_4_1 average=79.80 Sn=84.53 Sp=75.07
EUGENE_4_2 average=79.42 Sn=86.20 Sp=72.63
GLEAN_4_1 average=78.79 Sn=82.21 Sp=75.37
EUGENE_4_3 average=78.21 Sn=82.24 Sp=74.18
EUGENE_4_4 average=77.81 Sn=83.85 Sp=71.76
GENOMIX_4_1 average=77.17 Sn=76.98 Sp=77.36
GESECA_4_1 average=73.44 Sn=80.07 Sp=66.81
GRAMENE_4_1 average=61.56 Sn=74.40 Sp=48.71

Overall summary

Please note that this summary is intended only to give a general flavour for which programs are best at the nucleotide and exon level. This summary only deals with the exon-level and base-level results. The transcript-level and gene-level results (which have yet to be announced) are also important components of gene prediction assessment and will reveal further detail about which programs perform best overall.

In category 1 (ab initio genefinders), MGENE version 1 does best on the exon-level and base-level.

In category 2 (dual/multi-genome gene-finders), MGENE version 1 does best on the exon-level and MGENE version 2 does best on the base-level (but the difference between MGENE versions 1 and 2 is 0.03% at the base-level and 0.01% at the exon-level).

In category 3 (gene-finders that use alignments of proteins, ESTs, or mRNAs), FGENESH++ does best at the exon-level and MGENE version 3 does best at the base-level. However, FGENESH++ is 2.06% better than MGENE version 3 at the exon-level, while MGENE version 3 is only 0.21% better than FGENESH++ at the base-level.

In category 4 (combiners) JIGSAW version 1 does best at the exon-level and JIGSAW version 2 best best at the base-level. However, JIGSAW version 1 does 2.12% better than JIGSAW version 2 at the exon-level, while JIGSAW version 2 does only 0.35% better than JIGSAW version 1 at the base-level. Please note however that not all of the category 4 genefinders used exactly the same training sets, so we will be analysing these results further to figure out the effect of training set.

Get the Predictions

The final set of predictions that will be used to form the canonical gene sets for each species are available on the Wormbase FTP site