OPAL: Open-community Profiling Assessment tooL produced on 2019-03-20 10:28 with OPAL version 1.0.0
1st_CAMI_Challenge_Dataset_3_CAMI_high submissions
  • Metrics
  • Rankings
  • Alpha diversity
  • Beta diversity
  • Gold standard
WorstBestMedian

Presence/absence of taxa

Tool Gold standard CLARK_v1.1.3 CommonKmers FOCUS MetaPhlAn2.0 db_v20 MetaPhyler_V1.25 mOTU_1.1.1 Quickr Taxy-Pro Taxy-Pro_sjanssen TIPP_1.1
Completeness (recall)Completeness (recall): ratio of taxa correctly predicted as present and all taxa present at the selected taxonomic rank. It ranges from 0 (worst) to 1 (best).
 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)
Purity (precision)Purity (precision): ratio of taxa correctly predicted as present and all predicted taxa at the selected taxonomic rank. It ranges from 0 (worst) to 1 (best).
 1 (
0standard error: 0
)		 0.633 (
0.0333standard error: 0.0333
)		 0.867 (
0.0816standard error: 0.0816
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 0.667 (
0standard error: 0
)		 0.733 (
0.0667standard error: 0.0667
)		 0.933 (
0.0667standard error: 0.0667
)		 1 (
0standard error: 0
)
F1 scoreF1 score: harmonic average of completeness and purity at the selected taxonomic rank. It ranges from 0 (worst) to 1 (best).
 1 (
0standard error: 0
)		 0.773 (
0.0267standard error: 0.0267
)		 0.92 (
0.049standard error: 0.049
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 0.8 (
0standard error: 0
)		 0.84 (
0.04standard error: 0.04
)		 0.96 (
0.04standard error: 0.04
)		 1 (
0standard error: 0
)
True positivesTrue positives: number of taxa correctly predicted as present at the selected taxonomic rank.
 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)
False positivesFalse positives: number of taxa incorrectly predicted as present at the selected taxonomic rank.
 0 (
0standard error: 0
)		 1.2 (
0.2standard error: 0.2
)		 0.4 (
0.245standard error: 0.245
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 1 (
0standard error: 0
)		 0.8 (
0.2standard error: 0.2
)		 0.2 (
0.2standard error: 0.2
)		 0 (
0standard error: 0
)
False negativesFalse negatives: number of taxa incorrectly predicted as absent at the selected taxonomic rank.
 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)		 0 (
0standard error: 0
)
Jaccard indexJaccard index: indicator of similarity between the sets of true and predicted taxa at the seletec taxonomic rank computed as the ratio of the number of taxa in the intersection of these sets to the number of taxa in their union. It ranges from 0 (complete dissimilarity) to 1 (complete overlap).
 1 (
0standard error: 0
)		 0.633 (
0.0333standard error: 0.0333
)		 0.867 (
0.0816standard error: 0.0816
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 1 (
0standard error: 0
)		 0.667 (
0standard error: 0
)		 0.733 (
0.0667standard error: 0.0667
)		 0.933 (
0.0667standard error: 0.0667
)		 1 (
0standard error: 0
)

Abundance estimates

Tool Gold standard CLARK_v1.1.3 CommonKmers FOCUS MetaPhlAn2.0 db_v20 MetaPhyler_V1.25 mOTU_1.1.1 Quickr Taxy-Pro Taxy-Pro_sjanssen TIPP_1.1
Weighted UniFrac errorWeighted UniFrac error: tree-based measure of similarity between the true and predicted abundances at all taxonomic ranks ranging from 0 (high similarity) to 16 (low similarity).
 0 (
0standard error: 0
)		 5.45 (
0.0565standard error: 0.0565
)		 7.65 (
0.0569standard error: 0.0569
)		 7.14 (
0.0485standard error: 0.0485
)		 9.37 (
0.0752standard error: 0.0752
)		 6.39 (
0.0214standard error: 0.0214
)		 8.67 (
0.0431standard error: 0.0431
)		 7.79 (
0.0526standard error: 0.0526
)		 5.31 (
0.0151standard error: 0.0151
)		 7.47 (
0.016standard error: 0.016
)		 6.96 (
0.0372standard error: 0.0372
)
Unweighted UniFrac errorUnweighted UniFrac error: similar to the weighted UniFrac error, but measuring how well a profiler correctly identified the presence and absence of taxa instead of their relative abundances. The maximal value is the product of the number of taxonomic ranks minus 1 and the number of true taxa at all taxonomic ranks.
 0 (
0standard error: 0
)		 8.5e+03 (
39.3standard error: 39.3
)		 5.58e+03 (
100standard error: 100
)		 5.39e+03 (
38.9standard error: 38.9
)		 6.27e+03 (
39.4standard error: 39.4
)		 1.18e+04 (
68.4standard error: 68.4
)		 6.31e+03 (
81.4standard error: 81.4
)		 9.93e+03 (
123standard error: 123
)		 5.59e+03 (
25.8standard error: 25.8
)		 1.13e+04 (
160standard error: 160
)		 7.16e+03 (
57.1standard error: 57.1
)
L1 norm errorL1 norm error: sum of the absolute differences between the true and predicted abundances at the selected taxonomic rank ranging from 0 (perfect reconstruction of the relative abundances) to 2 (totally incorrect reconstruction).
 0 (
0standard error: 0
)		 0.0159 (
0.00195standard error: 0.00195
)		 0.0241 (
0.00431standard error: 0.00431
)		 0.0279 (
0.00339standard error: 0.00339
)		 0.14 (
0.0203standard error: 0.0203
)		 0.000988 (
7.47e-05standard error: 7.47e-05
)		 0.258 (
0.0373standard error: 0.0373
)		 0.0721 (
0.00264standard error: 0.00264
)		 0.00888 (
0.00118standard error: 0.00118
)		 0.00879 (
0.00121standard error: 0.00121
)		 0.00772 (
0.00105standard error: 0.00105
)
Bray-Curtis distanceBray-Curtis distance: normalized form of the L1 norm ranging from 0 to 1.
 0 (
0standard error: 0
)		 0.00796 (
0.000973standard error: 0.000973
)		 0.0121 (
0.00216standard error: 0.00216
)		 0.014 (
0.0017standard error: 0.0017
)		 0.0701 (
0.0102standard error: 0.0102
)		 0.000494 (
3.73e-05standard error: 3.73e-05
)		 0.129 (
0.0186standard error: 0.0186
)		 0.036 (
0.00132standard error: 0.00132
)		 0.00444 (
0.000591standard error: 0.000591
)		 0.0044 (
0.000604standard error: 0.000604
)		 0.00386 (
0.000526standard error: 0.000526
)

Alpha diversity

Tool Gold standard CLARK_v1.1.3 CommonKmers FOCUS MetaPhlAn2.0 db_v20 MetaPhyler_V1.25 mOTU_1.1.1 Quickr Taxy-Pro Taxy-Pro_sjanssen TIPP_1.1
Taxon countsTaxon counts: number of predicted taxa at the selected taxonomic rank.
 2 (
0standard error: 0
)		 3.2 (
0.2standard error: 0.2
)		 2.4 (
0.245standard error: 0.245
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 2 (
0standard error: 0
)		 3 (
0standard error: 0
)		 2.8 (
0.2standard error: 0.2
)		 2.2 (
0.2standard error: 0.2
)		 2 (
0standard error: 0
)
Shannon diversityShannon diversity: ranges from 0 to the natural logarithm of the number of predicted taxa, where the latter represents maximal possible diversity with all taxa being evenly represented.
 0.0635 (
0.00643standard error: 0.00643
)		 0.0966 (
0.00948standard error: 0.00948
)		 0.114 (
0.0149standard error: 0.0149
)		 0.119 (
0.0017standard error: 0.0017
)		 0.28 (
0.0291standard error: 0.0291
)		 0.063 (
0.00648standard error: 0.00648
)		 0.399 (
0.0379standard error: 0.0379
)		 0.224 (
0.00482standard error: 0.00482
)		 0.043 (
0.00422standard error: 0.00422
)		 0.0432 (
0.00414standard error: 0.00414
)		 0.0799 (
0.00773standard error: 0.00773
)
Shannon equitabilityShannon equitability: normalized form of Shannon diversity index ranging from 0 to 1 with 1 indicating complete evenness.
 0.0916 (
0.00928standard error: 0.00928
)		 0.0851 (
0.0104standard error: 0.0104
)		 0.134 (
0.0128standard error: 0.0128
)		 0.172 (
0.00245standard error: 0.00245
)		 0.403 (
0.0419standard error: 0.0419
)		 0.0909 (
0.00934standard error: 0.00934
)		 0.576 (
0.0546standard error: 0.0546
)		 0.203 (
0.00439standard error: 0.00439
)		 0.0444 (
0.00771standard error: 0.00771
)		 0.0566 (
0.00517standard error: 0.00517
)		 0.115 (
0.0112standard error: 0.0112
)
  • Relative performance
  • Absolute performance
Hint 1: click on the columns of scores for sorting.
#
Sum of scores
Completeness (recall)
Purity (precision)
L1 norm error
Weighted UniFrac error
0
TIPP_1.1
231
CLARK_v1.1.3
0
CommonKmers
8
MetaPhyler_V1.25
5
Taxy-Pro
1
1
MetaPhyler_V1.25
239
Quickr
44
MetaPhlAn2.0 db_v20
24
FOCUS
45
CLARK_v1.1.3
4
2
FOCUS
309
TIPP_1.1
53
mOTU_1.1.1
43
TIPP_1.1
68
MetaPhyler_V1.25
10
3
CLARK_v1.1.3
334
MetaPhyler_V1.25
83
FOCUS
85
Taxy-Pro_sjanssen
72
TIPP_1.1
15
4
Taxy-Pro_sjanssen
350
Taxy-Pro
83
TIPP_1.1
95
Taxy-Pro
97
FOCUS
20
5
Taxy-Pro
366
Taxy-Pro_sjanssen
112
MetaPhyler_V1.25
141
CLARK_v1.1.3
105
Taxy-Pro_sjanssen
25
6
CommonKmers
399
FOCUS
159
Taxy-Pro_sjanssen
141
CommonKmers
136
CommonKmers
31
7
Quickr
441
mOTU_1.1.1
181
Quickr
178
Quickr
185
Quickr
34
8
mOTU_1.1.1
455
MetaPhlAn2.0 db_v20
186
Taxy-Pro
185
mOTU_1.1.1
191
mOTU_1.1.1
40
9
MetaPhlAn2.0 db_v20
476
CommonKmers
224
CLARK_v1.1.3
225
MetaPhlAn2.0 db_v20
221
MetaPhlAn2.0 db_v20
45
Hint 2: slide the bars to change the weight of the metrics.
1
1
1
1
1
1
1
1
  • Shannon
  • Proportions
  • Rarefaction curves
Hint: click on a plot for its legend and drag it around the screen as necessary. Click on the same plot again to hide the legend.
  • OPAL always assumes that the samples are from the same environment.
  • Dotted lines are accumulation curves.