OPAL: Open-community Profiling Assessment tooL produced on 2019-03-20 12:33 with OPAL version 1.0.0
1st_CAMI_Challenge_Dataset_1_CAMI_low submissions
- Metrics
- Rankings
- Alpha diversity
- Beta diversity
- Gold standard
WorstBestMedian
Presence/absence of taxa
| Tool | Gold standard | CLARK_v1.1.3 | CommonKmers | CommonKmers Sensitive Unnormalized | CommonKmers sjanssen | DUDes | FOCUS_cfk7b | FOCUS_cfk7bd | FOCUS_cfk7d | FOCUS_cfk8b | FOCUS_cfk8bd | FOCUS_cfk8d | FOCUS sjanssen | MetaPhlAn2.0 db_v20 | MetaPhyler_V1.25 | mOTU_1.1.1 | Quickr | Taxy-Pro | Taxy-Pro sjanssen | TIPP_1.1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Completeness (recall)Completeness (recall): ratio of taxa correctly predicted as present and all taxa present at the selected taxonomic rank. It ranges from 0 (worst) to 1 (best). |
1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Purity (precision)Purity (precision): ratio of taxa correctly predicted as present and all predicted taxa at the selected taxonomic rank. It ranges from 0 (worst) to 1 (best). |
1 | 0.333 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 1 | 0.333 | 0.5 | 0.5 | 0.5 |
F1 scoreF1 score: harmonic average of completeness and purity at the selected taxonomic rank. It ranges from 0 (worst) to 1 (best). |
1 | 0.5 | 0.667 | 0.667 | 0.667 | 1 | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 1 | 0.667 | 1 | 0.5 | 0.667 | 0.667 | 0.667 |
True positivesTrue positives: number of taxa correctly predicted as present at the selected taxonomic rank. |
1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
False positivesFalse positives: number of taxa incorrectly predicted as present at the selected taxonomic rank. |
0 | 2 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 2 | 1 | 1 | 1 |
False negativesFalse negatives: number of taxa incorrectly predicted as absent at the selected taxonomic rank. |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Jaccard indexJaccard index: indicator of similarity between the sets of true and predicted taxa at the seletec taxonomic rank computed as the ratio of the number of taxa in the intersection of these sets to the number of taxa in their union. It ranges from 0 (complete dissimilarity) to 1 (complete overlap). |
1 | 0.333 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 1 | 0.5 | 1 | 0.333 | 0.5 | 0.5 | 0.5 |
Abundance estimates
| Tool | Gold standard | CLARK_v1.1.3 | CommonKmers | CommonKmers Sensitive Unnormalized | CommonKmers sjanssen | DUDes | FOCUS_cfk7b | FOCUS_cfk7bd | FOCUS_cfk7d | FOCUS_cfk8b | FOCUS_cfk8bd | FOCUS_cfk8d | FOCUS sjanssen | MetaPhlAn2.0 db_v20 | MetaPhyler_V1.25 | mOTU_1.1.1 | Quickr | Taxy-Pro | Taxy-Pro sjanssen | TIPP_1.1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Weighted UniFrac errorWeighted UniFrac error: tree-based measure of similarity between the true and predicted abundances at all taxonomic ranks ranging from 0 (high similarity) to 16 (low similarity). |
0 | 8.75 | 11.3 | 11 | 11 | 11.7 | 10.3 | 10.1 | 10 | 10.3 | 10 | 9.97 | 10 | 11.5 | 8.7 | 12.2 | 8.61 | 8.35 | 10.4 | 9.58 |
Unweighted UniFrac errorUnweighted UniFrac error: similar to the weighted UniFrac error, but measuring how well a profiler correctly identified the presence and absence of taxa instead of their relative abundances. The maximal value is the product of the number of taxonomic ranks minus 1 and the number of true taxa at all taxonomic ranks. |
0 | 9.66e+03 | 696 | 861 | 861 | 2.67e+03 | 1.48e+03 | 1.62e+03 | 1.34e+03 | 1.76e+03 | 1.81e+03 | 1.59e+03 | 1.81e+03 | 489 | 1.19e+04 | 2.32e+03 | 8.84e+03 | 3.86e+03 | 9.84e+03 | 6.04e+03 |
L1 norm errorL1 norm error: sum of the absolute differences between the true and predicted abundances at the selected taxonomic rank ranging from 0 (perfect reconstruction of the relative abundances) to 2 (totally incorrect reconstruction). |
0 | 0.00232 | 0.00248 | 0.00244 | 0.00244 | 0 | 0.0663 | 0.0688 | 0.0601 | 0.0668 | 0.0695 | 0.0573 | 0.0695 | 0 | 0.00149 | 0 | 0.093 | 0.0054 | 0.00469 | 0.00619 |
Bray-Curtis distanceBray-Curtis distance: normalized form of the L1 norm ranging from 0 to 1. |
0 | 0.00116 | 0.00124 | 0.00122 | 0.00122 | 0 | 0.0332 | 0.0344 | 0.0301 | 0.0334 | 0.0347 | 0.0287 | 0.0347 | 0 | 0.000746 | 0 | 0.0465 | 0.0027 | 0.00235 | 0.00309 |
Alpha diversity
| Tool | Gold standard | CLARK_v1.1.3 | CommonKmers | CommonKmers Sensitive Unnormalized | CommonKmers sjanssen | DUDes | FOCUS_cfk7b | FOCUS_cfk7bd | FOCUS_cfk7d | FOCUS_cfk8b | FOCUS_cfk8bd | FOCUS_cfk8d | FOCUS sjanssen | MetaPhlAn2.0 db_v20 | MetaPhyler_V1.25 | mOTU_1.1.1 | Quickr | Taxy-Pro | Taxy-Pro sjanssen | TIPP_1.1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Taxon countsTaxon counts: number of predicted taxa at the selected taxonomic rank. |
1 | 3 | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 1 | 2 | 1 | 3 | 2 | 2 | 2 |
Shannon diversityShannon diversity: ranges from 0 to the natural logarithm of the number of predicted taxa, where the latter represents maximal possible diversity with all taxa being evenly represented. |
0 | 0.00944 | 0.00955 | 0.00941 | 0.00941 | 0 | 0.146 | 0.15 | 0.135 | 0.146 | 0.151 | 0.13 | 0.151 | 0 | 0.00611 | 0 | 0.219 | 0.0187 | 0.0165 | 0.021 |
Shannon equitabilityShannon equitability: normalized form of Shannon diversity index ranging from 0 to 1 with 1 indicating complete evenness. |
0 | 0.00859 | 0.0138 | 0.0136 | 0.0136 | 0 | 0.21 | 0.216 | 0.195 | 0.211 | 0.218 | 0.188 | 0.218 | 0 | 0.00882 | 0 | 0.199 | 0.0269 | 0.0239 | 0.0302 |
- Relative performance
- Absolute performance
Hint 1: click on the columns of scores for sorting.
#
Sum of scores
Completeness (recall)
Purity (precision)
L1 norm error
Weighted UniFrac error
0
Quickr
97
CLARK_v1.1.3
2
MetaPhlAn2.0 db_v20
2
Quickr
10
Taxy-Pro
0
1
FOCUS_cfk8d
99
Quickr
14
CommonKmers Sensitive Unnormalized
12
MetaPhyler_V1.25
12
Quickr
1
2
FOCUS_cfk8bd
115
Taxy-Pro sjanssen
18
CommonKmers
13
FOCUS_cfk8d
18
MetaPhyler_V1.25
2
3
FOCUS_cfk7d
122
Taxy-Pro
23
DUDes
14
FOCUS sjanssen
20
CLARK_v1.1.3
3
4
MetaPhyler_V1.25
124
TIPP_1.1
33
CommonKmers sjanssen
17
FOCUS_cfk8bd
21
TIPP_1.1
4
5
DUDes
134
DUDes
34
mOTU_1.1.1
18
FOCUS_cfk7d
27
FOCUS_cfk8d
5
6
FOCUS sjanssen
140
MetaPhyler_V1.25
34
FOCUS_cfk7d
35
FOCUS_cfk7bd
34
FOCUS_cfk7d
6
7
TIPP_1.1
141
FOCUS_cfk8bd
35
FOCUS_cfk8d
38
TIPP_1.1
35
FOCUS_cfk8bd
7
8
FOCUS_cfk7bd
146
FOCUS_cfk8d
38
FOCUS_cfk7b
41
Taxy-Pro sjanssen
43
FOCUS sjanssen
8
9
Taxy-Pro sjanssen
148
FOCUS_cfk8b
46
FOCUS_cfk8b
42
Taxy-Pro
54
FOCUS_cfk7bd
9
10
FOCUS_cfk8b
156
FOCUS_cfk7bd
53
FOCUS_cfk7bd
50
CommonKmers sjanssen
57
FOCUS_cfk8b
10
11
CommonKmers Sensitive Unnormalized
157
FOCUS_cfk7d
54
FOCUS_cfk8bd
52
FOCUS_cfk8b
58
FOCUS_cfk7b
11
12
CommonKmers sjanssen
157
FOCUS sjanssen
56
FOCUS sjanssen
56
CommonKmers Sensitive Unnormalized
58
Taxy-Pro sjanssen
12
13
CLARK_v1.1.3
158
FOCUS_cfk7b
58
TIPP_1.1
69
FOCUS_cfk7b
58
CommonKmers sjanssen
13
14
Taxy-Pro
160
mOTU_1.1.1
67
Quickr
72
CLARK_v1.1.3
63
CommonKmers Sensitive Unnormalized
14
15
MetaPhlAn2.0 db_v20
160
CommonKmers sjanssen
70
Taxy-Pro sjanssen
75
MetaPhlAn2.0 db_v20
65
CommonKmers
15
16
CommonKmers
165
CommonKmers
70
MetaPhyler_V1.25
76
CommonKmers
67
MetaPhlAn2.0 db_v20
16
17
FOCUS_cfk7b
168
CommonKmers Sensitive Unnormalized
73
Taxy-Pro
83
DUDes
69
DUDes
17
18
mOTU_1.1.1
189
MetaPhlAn2.0 db_v20
77
CLARK_v1.1.3
90
mOTU_1.1.1
86
mOTU_1.1.1
18
Hint 2: slide the bars to change the weight of the metrics.
1
1
1
1
1
1
1
1
- Shannon
- Proportions
- Rarefaction curves
Hint: click on a plot for its legend and drag it around the screen as necessary. Click on the same plot again to hide the legend.
- OPAL always assumes that the samples are from the same environment.
- Dotted lines are accumulation curves.
