First LPT Challenge: Time-Resolved results

This page presents published results for the TR (Time-Resolved) case of the 1st LPT Challenge. The link for result submission, as well as details about the test case and  rules for result are given there.

Tables can be ordered according to each of the quantities reported. Clicking on an algorithm name in a table will open a page gathering its result, together with information on the submitter.

Definition of performance metrics

Evaluation pertains to both particle localization, at the two pulse times t0 and t1, and estimation of the displacement vectors. For either particles or vectors, we count :

  • a True Positive (TP) if a ground truth (GT) particle is found in the neighborhood of a detection (maximum componentwise distance less than 1 voxel). For vectors, both detections should be TPs and associated to the same GT particle.
  • a False Negative (FN, a.k.a. missed particle) if there is no detection in the neighborhood of a GT particle. A FN vector is a GT vector that is not associated to a TP vector.
  • a False Positive (FP, a.k.a. ghost particle/vector) particle/vector when it is not a TP particle/vector.

Precision and Recall are derived quantities reflecting the detection performance, defined as:

 

Root-mean-square (rms) errors, whether on particle position, velocity or acceleration, are defined by considering only the TP within the result. Corresponding units are as follows:

  • Position errors (raw and fitted) are given in voxel, with a voxel size here equal to 60 µm
  • Velocity errors are dimensionless, with corresponding reference velocity Vinf = 0.667 m/s
  • Acceleration errors are dimensionless, with corresponding reference acceleration Ainf = Vinf2/D = 44.4 m/s2, where D = 0.01 m is the cylinder diameter (see there for more information on the physical setup).

Below, we provide tables showing these quantities for all test points. Then, we show figures comparing these quantities among participants, as well as groups of error slices for velocity and acceleration at a given test point.

ppp: 0.005
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.999 1 6429 8 3 0.0547 0.0418 0.0154 0.0146 0.036 0.00773 0.00277 0.00243 0.0068 0.378 0.222 0.204 0.229
LCTI INRAE 0.999 1 6431 5 1 0.0578 0.0433 0.0154 0.0155 0.0373 0.00837 0.00293 0.00249 0.00744 0.378 0.221 0.204 0.229
STB DLR 1 1 6432 0 0 0.0444 0.0224 0.00861 0.00804 0.0191 0.00226 0.00104 0.00107 0.0017 0.0444 0.0255 0.0248 0.0266
STB@DaVis LaVision GmbH 1 1 6432 0 0 0.0523 0.0566 0.0257 0.0226 0.0451 0.0115 0.00578 0.00648 0.00748 0.218 0.107 0.132 0.137
ppp: 0.025
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.997 0.996 31534 83 116 0.0687 0.0542 0.0201 0.0189 0.0467 0.0244 0.0121 0.0106 0.0183 0.356 0.199 0.197 0.221
LCTI INRAE 0.998 0.998 31576 59 74 0.079 0.0647 0.0239 0.0229 0.0556 0.0163 0.00562 0.0046 0.0146 0.357 0.199 0.197 0.221
STB DLR 1 1 31648 0 2 0.0479 0.025 0.00954 0.00881 0.0214 0.00217 0.001 0.000938 0.00168 0.0397 0.021 0.0221 0.0254
STB@DaVis LaVision GmbH 1 0.999 31626 2 24 0.06 0.0593 0.0233 0.0228 0.0496 0.0117 0.00592 0.00613 0.00795 0.21 0.113 0.119 0.13
ppp: 0.05
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.996 0.994 63577 234 395 0.0806 0.0659 0.0243 0.0225 0.0569 0.0412 0.0207 0.0184 0.0305 0.346 0.204 0.188 0.208
LCTI INRAE 0.995 0.99 63355 307 617 0.099 0.0849 0.0307 0.0292 0.0735 0.0405 0.0302 0.0114 0.0245 0.347 0.203 0.189 0.208
STB DLR 1 1 63958 0 14 0.0523 0.0283 0.0105 0.0101 0.0242 0.00244 0.00106 0.00107 0.00192 0.0437 0.0225 0.0239 0.0289
STB@DaVis LaVision GmbH 1 0.998 63873 15 99 0.0721 0.0691 0.0268 0.026 0.0582 0.0119 0.00578 0.00622 0.00839 0.196 0.105 0.113 0.12
ppp: 0.08
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.992 0.987 101424 774 1375 0.096 0.0791 0.0293 0.0258 0.0688 0.0737 0.0453 0.0329 0.048 0.346 0.202 0.181 0.214
LCTI INRAE 0.991 0.97 99664 933 3135 0.13 0.112 0.0413 0.0379 0.097 0.0613 0.0493 0.0161 0.0327 0.345 0.202 0.18 0.214
STB DLR 1 1 102774 2 25 0.0559 0.0307 0.0115 0.0111 0.0262 0.00268 0.00124 0.00122 0.00204 0.0473 0.0247 0.0233 0.033
STB@DaVis LaVision GmbH 1 0.998 102567 18 232 0.0925 0.0867 0.0321 0.0305 0.0745 0.0131 0.00592 0.00601 0.01 0.194 0.101 0.105 0.128
ppp: 0.12
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.967 0.982 150941 5172 2805 0.122 0.101 0.0373 0.0333 0.0881 0.0254 0.0121 0.00756 0.021 0.365 0.205 0.203 0.223
STB DLR 1 0.999 153662 18 84 0.0609 0.036 0.0134 0.0132 0.0307 0.00318 0.00136 0.00153 0.00244 0.0559 0.025 0.0319 0.0385
STB@DaVis LaVision GmbH 1 0.992 152572 15 1174 0.107 0.0997 0.0366 0.0346 0.086 0.0151 0.00676 0.00722 0.0114 0.237 0.12 0.135 0.154
ppp: 0.16
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.818 0.774 158272 35313 46343 0.275 0.259 0.0937 0.0852 0.226 0.0736 0.0328 0.0201 0.0628 0.35 0.203 0.191 0.212
STB DLR 1 0.999 204472 14 143 0.0662 0.038 0.0145 0.0138 0.0323 0.00305 0.00141 0.00135 0.00234 0.0457 0.024 0.0244 0.0303
STB@DaVis LaVision GmbH 1 0.991 202781 38 1834 0.129 0.119 0.043 0.0393 0.103 0.0163 0.00706 0.00715 0.0129 0.226 0.115 0.124 0.15
ppp: 0.2
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
KLPT INRAE 0.126 0.105 26830 186418 228495 0.578 0.57 0.25 0.237 0.454 0.235 0.124 0.0764 0.185 0.291 0.161 0.161 0.181
STB DLR 1 0.999 255082 23 243 0.0767 0.0446 0.0159 0.0229 0.0348 0.00343 0.00152 0.00182 0.00248 0.0553 0.0284 0.033 0.0341
STB@DaVis LaVision GmbH 0.999 0.988 252239 131 3086 0.16 0.147 0.0522 0.0467 0.13 0.0194 0.0078 0.00747 0.0161 0.245 0.117 0.123 0.176

1st LPT Challenge TR results