Second LPT Challenge Case 3: Two-Pulse results

This page presents published results for the TP (Two-Pulse) case of the 2nd LPT Challenge Case 3. The link for result submission, as well as details about the test case and rules for result formatting are given there.

Tables can be ordered according to each of the quantities reported. Clicking on an algorithm name in a table will open a page gathering its result, together with information on the submitter.

Definition of performance metrics

Evaluation pertains to both particle localization, at the two pulse times t0 and t1, and estimation of the displacement vectors. For either particles or vectors, we count :

  • a True Positive (TP) if a ground truth (GT) particle is found in the neighborhood of a detection (maximum componentwise distance less than 1 voxel). For vectors, both detections should be TPs and associated to the same GT particle.
  • a False Negative (FN, a.k.a. missed particle) if there is no detection in the neighborhood of a GT particle. A FN vector is a GT vector that is not associated to a TP vector.
  • a False Positive (FP, a.k.a. ghost particle/vector) particle/vector when it is not a TP particle/vector.

Precision and Recall are derived quantities reflecting the detection performance, defined as:

 

Root-mean-square (rms) error on particle position or displacement are defined by only considering the TP within the result. 

In the tables, they are given in voxel, with a voxel size here equal to 528 µm; figures also show these errors in mm. Conversion from displacement to velocities can be done straightforwardly by considering the acquisition frequency, set to 30 Hz; for reference, the free-stream velocity is here equal to 0.6633 m/s.

Below the tables, we provide plots comparing some of these quantities among participants, as well as displacement error slices for a selected test point.

ppp: 0.005
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
ITE-PTV ONERA-CEA 0.852 0.94 1184 18622 3235 0.851 0.412 0.224 0.711 0.852 0.94 18623 3234 1183 0.85 0.411 0.225 0.709 0.851 0.939 18600 3257 1206 0.374
VICCTOR ONERA-CEA 0.866 0.907 1850 17956 2770 0.949 0.432 0.289 0.794 0.866 0.906 17941 2785 1865 0.955 0.435 0.287 0.8 0.863 0.903 17888 2838 1918 0.727
ppp: 0.015
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
ITE-PTV ONERA-CEA 0.802 0.951 2453 47271 11706 0.88 0.419 0.234 0.737 0.802 0.951 47282 11695 2442 0.877 0.418 0.233 0.735 0.799 0.948 47135 11842 2589 0.444
VICCTOR ONERA-CEA 0.849 0.915 4225 45499 8120 1.01 0.45 0.306 0.846 0.848 0.915 45492 8127 4232 0.994 0.447 0.303 0.834 0.843 0.909 45191 8428 4533 0.813
ppp: 0.03
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
ITE-PTV ONERA-CEA 0.883 0.889 13294 106281 14043 0.895 0.418 0.241 0.753 0.883 0.889 106256 14068 13319 0.895 0.419 0.242 0.753 0.878 0.884 105682 14642 13893 0.524
VICCTOR ONERA-CEA 0.874 0.798 24172 95403 13790 1.1 0.486 0.386 0.907 0.876 0.8 95649 13544 23926 1.06 0.466 0.357 0.884 0.862 0.787 94165 15028 25410 1.03
ppp: 0.045
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
ITE-PTV ONERA-CEA 0.876 0.844 28011 151280 21388 0.909 0.421 0.25 0.765 0.876 0.843 151219 21449 28072 0.91 0.421 0.249 0.767 0.869 0.837 149997 22671 29294 0.592
VICCTOR ONERA-CEA 0.863 0.718 50644 128647 20365 1.06 0.456 0.345 0.891 0.866 0.72 129022 19990 50269 1.02 0.442 0.318 0.865 0.852 0.708 126925 22087 52366 0.972
ppp: 0.06
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
ITE-PTV ONERA-CEA 0.847 0.779 52955 186155 33635 0.933 0.421 0.265 0.789 0.847 0.778 186058 33732 53052 0.932 0.42 0.266 0.789 0.836 0.769 183816 35974 55294 0.664
ppp: 0.075
Algorithm Institution Prec. t0 Recall t0 #FN t0 #TP t0 #FP t0 Pos. rms error t0 X pos. rms error t0 Y pos. rms error t0 Z pos. rms error t0 Prec. t1 Recall t1 #TP t1 #FP t1 #FN t1 Pos. rms error t1 X pos. rms error t1 Y pos. rms error t1 Z pos. rms error t1 Vector Prec. Vector Recall #TP vectors #FP vectors #FN vectors Disp. rms error
ITE-PTV ONERA-CEA 0.752 0.682 94991 203443 67016 0.96 0.423 0.297 0.808 0.752 0.682 203489 66970 94945 0.949 0.419 0.289 0.801 0.738 0.669 199505 70954 98929 0.76

2nd LPT Challenge case 3 TP results