Second LPT Challenge Case 1: Results

This page presents published results for Case 1 of the 2nd LPT Challenge. The link for result submission, as well as details about the test case and  rules for result are given there.

Tables can be ordered according to each of the quantities reported. Clicking on an algorithm name in a table will open a page gathering its result, together with information on the submitter.

Definition of performance metrics

Evaluation pertains to both particle localization, at the two pulse times t0 and t1, and estimation of the displacement vectors. For either particles or vectors, we count :

  • a True Positive (TP) if a ground truth (GT) particle is found in the neighborhood of a detection (maximum componentwise distance less than 1 voxel). For vectors, both detections should be TPs and associated to the same GT particle.
  • a False Negative (FN, a.k.a. missed particle) if there is no detection in the neighborhood of a GT particle. A FN vector is a GT vector that is not associated to a TP vector.
  • a False Positive (FP, a.k.a. ghost particle/vector) particle/vector when it is not a TP particle/vector.

Precision and Recall are derived quantities reflecting the detection performance, defined as:

 

Root-mean-square (rms) errors, whether on particle position, velocity or acceleration, are defined by considering only the TP within the result. Corresponding units are as follows:

  • Position errors (raw and fitted) are given in voxel, with a voxel size here equal to 110 µm
  • Velocity errors are dimensionless, with corresponding reference velocity Vinf = 0.2 m/s
  • Acceleration errors are dimensionless, with corresponding reference acceleration Ainf = Vinf2/D = 40 m/s2, where D = 0.1 m is the channel half-width (see there for more information on the physical setup).

Below, we provide tables showing these quantities for all test points. Then, we show figures comparing these quantities among participants, as well as groups of error slices for velocity and acceleration at a given test point.

dt: 1
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
PaIRS University of Naples Federico II 0.999 0.991 94017 105 868 0.1680 0.1340 0.0595 0.0622 0.1030 0.0117 0.0061 0.01 0.0075 1.2700 0.6840 0.7230 0.7930
PaIRS_2 Università degli Studi di Napoli “Federico II” 1.000 0.998 94684 43 201 0.0899 0.0727 0.0309 0.0326 0.0572 0.0072 0.0032 0.00 0.0056 1.0700 0.4850 0.4610 0.8310
STB DLR 0.999 0.995 94409 68 476 0.1450 0.1130 0.0492 0.0530 0.0863 0.0083 0.0040 0.00 0.0061 0.9050 0.4670 0.4490 0.6310
STB@DaVis LaVision GmbH 0.998 0.991 93985 202 900 0.2140 0.1620 0.0733 0.0787 0.1220 0.0182 0.0096 0.01 0.0112 1.3500 0.7320 0.7780 0.8320
dt: 2
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
PaIRS University of Naples Federico II 0.999 0.989 93924 113 1036 0.1700 0.1420 0.0660 0.0675 0.1070 0.0155 0.0081 0.01 0.0095 1.4300 0.7610 0.8410 0.8630
PaIRS_2 Università degli Studi di Napoli “Federico II” 0.999 0.998 94735 51 225 0.0899 0.0807 0.0343 0.0360 0.0636 0.0075 0.0034 0.00 0.0059 1.0800 0.4810 0.4600 0.8550
STB DLR 0.999 0.994 94437 82 523 0.1460 0.1230 0.0550 0.0575 0.0933 0.0097 0.0046 0.00 0.0071 0.8900 0.4690 0.4470 0.6100
STB@DaVis LaVision GmbH 0.994 0.978 92841 538 2119 0.2210 0.2130 0.1040 0.1150 0.1460 0.0237 0.0126 0.01 0.0142 1.4800 0.7940 0.8850 0.8870
dt: 3
Algorithm Institution Precision Recall #TP #FP #FN Raw pos. rms error Pos. rms error X pos. rms error Y pos. rms error Z pos. rms error Vel. rms error X vel. rms error Y vel. rms error Z vel. rms error Acc. rms error X acc. rms error Y acc. rms error Z acc. rms error
PaIRS University of Naples Federico II 0.998 0.985 93569 179 1423 0.1750 0.1650 0.0790 0.0828 0.1180 0.0192 0.0104 0.01 0.0115 1.5500 0.8530 0.9060 0.9250
PaIRS_2 Università degli Studi di Napoli “Federico II” 0.999 0.998 94765 57 227 0.0934 0.0895 0.0377 0.0392 0.0711 0.0078 0.0040 0.00 0.0056 0.9510 0.4470 0.4200 0.7270
STB DLR 0.999 0.993 94372 109 620 0.1470 0.1330 0.0626 0.0609 0.1000 0.0116 0.0063 0.01 0.0079 1.0100 0.5540 0.4990 0.6810
STB@DaVis LaVision GmbH 0.989 0.929 88258 955 6734 0.2460 0.2840 0.1430 0.1580 0.1880 0.0275 0.0151 0.02 0.0164 1.5700 0.8610 0.9310 0.9290

2nd LPT Challenge Case 1 results