First LPT Challenge: Two-Pulse results

This page presents published results for the TP (Two-Pulse) case of the 1st LPT Challenge. The link for result submission, as well as details about the test case and rules for result formatting are given there.

Tables can be ordered according to each of the quantities reported. Clicking on an algorithm name in a table will open a page gathering its result, together with information on the submitter.

Definition of performance metrics

Evaluation pertains to both particle localization, at the two pulse times t0 and t1, and estimation of the displacement vectors. For either particles or vectors, we count :

  • a True Positive (TP) if a ground truth (GT) particle is found in the neighborhood of a detection (maximum componentwise distance less than 1 voxel). For vectors, both detections should be TPs and associated to the same GT particle.
  • a False Negative (FN, a.k.a. missed particle) if there is no detection in the neighborhood of a GT particle. A FN vector is a GT vector that is not associated to a TP vector.
  • a False Positive (FP, a.k.a. ghost particle/vector) particle/vector when it is not a TP particle/vector.

Precision and Recall are derived quantities reflecting the detection performance, defined as:

 

Root-mean-square (rms) error on particle position or displacement are defined by only considering the TP within the result. 

In the tables, they are given in voxel, with a voxel size here equal to 60 µm; figures also show these errors in mm. Conversion from displacement to velocities can be done straightforwardly by applying the inter-frame time Dt = 600 µs; for reference, the free-stream velocity is here equal to 0.667 m/s.

Below the tables, we provide plots comparing some of these quantities among participants, as well as displacement error slices for a selected test point.

ppp: 0.005
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
3d-fluid-flow ETH Zrich 0.271 0.967 212 6187 16660 0.346 0.129 0.109 0.302 0.273 0.976 6243 16604 156 0.279 0.126 0.092 0.232 0.249 0.888 5680 17167 719 0.643
I-DFTPTV ONERA 1 0.999 4 6395 0 0.0495 0.021 0.0168 0.0415 1 0.999 6395 0 4 0.0506 0.0217 0.0157 0.0429 1 0.999 6395 0 4 0.0675
LaPIV INRAE 0.999 0.985 98 6301 7 0.0527 0.0191 0.0176 0.0459 0.999 0.984 6299 9 100 0.0574 0.0225 0.0199 0.0489 0.999 0.984 6299 9 100 0.0777
TP-STB DLR 1 1 1 6398 0 0.0431 0.0154 0.014 0.0378 1 1 6398 0 1 0.0433 0.0155 0.0141 0.0379 1 1 6398 0 1 0.0607
TP-STB@DaVis LaVision GmbH 1 0.999 5 6394 1 0.0508 0.0169 0.0158 0.0453 1 0.999 6394 1 5 0.0502 0.0166 0.0155 0.0447 1 0.999 6394 1 5 0.0662
VICCTOR ONERA-CEA 1 0.999 6 6393 1 0.0434 0.0197 0.0135 0.0363 1 0.999 6393 1 6 0.0434 0.0198 0.0136 0.0362 1 0.999 6393 1 6 0.0582
ppp: 0.025
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
3d-fluid-flow ETH Zrich 0.341 0.956 1431 30730 59478 0.299 0.119 0.106 0.252 0.342 0.959 30847 59361 1314 0.256 0.115 0.084 0.213 0.333 0.933 30011 60197 2150 0.515
I-DFTPTV ONERA 1 0.995 173 31988 5 0.0681 0.0279 0.0226 0.0578 1 0.995 31988 5 173 0.068 0.0279 0.0224 0.0578 1 0.994 31979 14 182 0.0864
LaPIV INRAE 0.983 0.988 399 31762 540 0.0712 0.0267 0.0257 0.0608 0.983 0.987 31751 551 410 0.0694 0.026 0.0243 0.0596 0.982 0.987 31735 567 426 0.0943
TP-STB DLR 1 1 13 32148 0 0.0491 0.0173 0.016 0.0431 1 1 32148 0 13 0.0494 0.0176 0.0161 0.0433 1 0.999 32144 4 17 0.0678
TP-STB@DaVis LaVision GmbH 1 0.998 61 32100 1 0.0603 0.0209 0.0191 0.0532 1 0.998 32100 1 61 0.0602 0.021 0.0193 0.053 1 0.998 32096 5 65 0.077
VICCTOR ONERA-CEA 1 0.998 77 32084 7 0.0473 0.021 0.0146 0.0397 1 0.998 32084 7 77 0.0473 0.021 0.0152 0.0395 1 0.997 32078 13 83 0.0628
ppp: 0.05
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
3d-fluid-flow ETH Zrich 0.346 0.987 823 62854 118678 0.278 0.11 0.096 0.237 0.347 0.988 62922 118610 755 0.239 0.111 0.077 0.197 0.341 0.971 61814 119718 1863 0.453
I-DFTPTV ONERA 0.999 0.98 1282 62395 63 0.12 0.044 0.0387 0.105 0.999 0.98 62395 63 1282 0.119 0.0433 0.0395 0.104 0.998 0.979 62351 107 1326 0.147
LaPIV INRAE 0.981 0.977 1474 62203 1194 0.0818 0.0305 0.029 0.0702 0.981 0.977 62189 1208 1488 0.0792 0.0298 0.0269 0.0682 0.98 0.976 62141 1256 1536 0.108
TP-STB DLR 1 1 31 63646 3 0.056 0.0201 0.0185 0.0489 1 0.999 63644 5 33 0.0558 0.0206 0.0188 0.0484 1 0.999 63627 22 50 0.0768
TP-STB@DaVis LaVision GmbH 1 0.997 195 63482 6 0.0726 0.0249 0.0237 0.0639 1 0.997 63480 8 197 0.0724 0.0247 0.0237 0.0638 1 0.997 63461 27 216 0.0928
VICCTOR ONERA-CEA 0.999 0.997 169 63508 51 0.0525 0.0223 0.017 0.0444 0.999 0.997 63508 51 169 0.0519 0.0222 0.0172 0.0437 0.999 0.997 63493 66 184 0.0683
ppp: 0.08
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
3d-fluid-flow ETH Zrich 0.434 0.979 2126 99816 129994 0.289 0.123 0.105 0.239 0.428 0.966 98428 131382 3514 0.254 0.113 0.0834 0.211 0.419 0.944 96205 133605 5737 0.453
I-DFTPTV ONERA 0.992 0.887 11497 90445 758 0.223 0.0791 0.0738 0.195 0.992 0.887 90434 769 11508 0.224 0.0799 0.0743 0.195 0.987 0.883 89976 1227 11966 0.27
LaPIV INRAE 0.944 0.935 6619 95323 5647 0.12 0.0432 0.0403 0.104 0.945 0.936 95432 5538 6510 0.109 0.0398 0.0357 0.095 0.942 0.933 95116 5854 6826 0.148
TP-STB DLR 1 0.999 98 101844 16 0.0638 0.023 0.0213 0.0556 1 0.999 101842 18 100 0.0642 0.0235 0.0217 0.0556 0.999 0.998 101787 73 155 0.0871
TP-STB@DaVis LaVision GmbH 1 0.997 314 101628 23 0.0892 0.0308 0.0278 0.079 1 0.997 101631 20 311 0.0907 0.0311 0.0279 0.0805 0.999 0.996 101581 70 361 0.116
VICCTOR ONERA-CEA 0.998 0.997 336 101606 197 0.0568 0.0237 0.0179 0.0484 0.998 0.997 101602 201 340 0.0567 0.024 0.0178 0.0482 0.998 0.996 101564 239 378 0.0744
ppp: 0.12
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
3d-fluid-flow ETH Zrich 0.501 0.949 7726 144498 143885 0.317 0.138 0.12 0.26 0.496 0.939 142966 145417 9258 0.286 0.125 0.0944 0.239 0.482 0.914 139135 149248 13089 0.477
TP-STB DLR 1 0.999 172 152052 31 0.0726 0.0264 0.0245 0.0631 1 0.999 152048 35 176 0.0728 0.0268 0.0246 0.0631 0.999 0.998 151911 172 313 0.0988
TP-STB@DaVis LaVision GmbH 1 0.981 2927 149297 32 0.111 0.0386 0.0349 0.0982 1 0.981 149292 37 2932 0.111 0.0383 0.0348 0.098 0.999 0.98 149205 124 3019 0.143
VICCTOR ONERA-CEA 0.998 0.997 517 151707 283 0.0622 0.025 0.0194 0.0536 0.998 0.997 151702 288 522 0.0614 0.0247 0.0192 0.0528 0.998 0.996 151655 335 569 0.0815
ppp: 0.16
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
3d-fluid-flow ETH Zrich 0.411 0.907 18980 184631 264203 0.344 0.151 0.129 0.282 0.409 0.901 183472 265362 20139 0.316 0.139 0.105 0.264 0.398 0.878 178709 270125 24902 0.487
TP-STB DLR 1 0.998 307 203304 61 0.0805 0.029 0.0263 0.0704 1 0.998 203298 67 313 0.0807 0.0287 0.0263 0.0707 0.999 0.998 203144 221 467 0.11
TP-STB@DaVis LaVision GmbH 0.999 0.979 4199 199412 110 0.139 0.0486 0.0426 0.123 1 0.979 199424 98 4187 0.139 0.048 0.0426 0.123 0.998 0.978 199182 340 4429 0.182
VICCTOR ONERA-CEA 0.998 0.997 589 203022 425 0.0661 0.0261 0.0201 0.0573 0.998 0.997 203015 432 596 0.0661 0.026 0.02 0.0574 0.998 0.997 202942 505 669 0.0884

1st LPT Challenge TP results