New TPC alignment impact on reconstruction speed

The new TPC alignment, as provided in PR#702, needs two items:
  1. A new code library, built currently as SL24y
  2. A new BFC chain option, "CorrZ"
Previously, comparisons of nightly tests using SL24y vs. DEV demonstrated a notable difference for any dataset, before or after iTPC installation, that was determined to be the result of increasing TPC hit errors (as used in tracking only) for de-convoluted hit (flag==2) by a factor of x4, increasing the number of found tracks by a few percent.

As part of a study for modifying vertex-finding for FXT data, I first processed some data in DEV, then I was asked to process it with the new alignment. This I only partly did at first, using the new code library but old BFC chain. After realizing this mistake, I processed again using the new code library with the new BFC chain option. This allowed me to compare the reconstruction speed difference due to the two items separately. My findings, using over 40k events from the production_4p59GeV_fixedTarget_2019 dataset are evident in the following plots:
  • Left: using the old BFC chain, time per event [seconds] in SL24y vs. DEV (black points are individual events [excluding the first], and red points are a profile)
  • Middle: using SL24y, time per event [seconds] with the new alignment BFC chain option vs. the old chain
  • Right: profiles of the previous two plots (library comparison in blue, alignment comparison in red) with linear fits, giving slopes of 1.027 and 1.500 respectively



Smaller sets of events were tried more than once to ensure reproducibility, and all direct comparisons were run on the same node at the same time to ensure comparability.

Looking at the end of the log files where time spent in individual makers in the BFC chain is presented, it is clear that the increased time for the new alignment is predominantly from the tracking ("Sti"), nearly doubling from 32777 CPU-seconds (47.5% of the full chain) in SL24y with the old chain to 63015 CPU-seconds (62.6% of the full chain) in SL24y with the new chain. At this time, it is not obvious to me why the new alignment affects tracking speed so significantly.

I believe this impact on reconstruction speed was a contributing factor to why the new TPC alignment test production conducting in a TFG library this past summer saw jobs take longer than the 3-day queue limit at SDCC. The 3-day queue limit never affected prior official BES-II data productions, but a closer look will be needed for any production with this new TPC alignment.



An important question is whether other iTPC era datasets than FXT may be impacted. I processed 200 Run 22 pp500 events to study this and obtained the following results, which show behavior consistent with the above FXT findings: the new library is a few percent slower, and the new alignment is roughly x1.5 slower. Also, the slow-down is again predominantly a near-doubling of the time spent in tracking ("Sti").

Importantly, this slow-down will have a big impact on my earlier Run 22 pp500 production estimates - more streams.




-Gene