Following up on the problematic JetCorr PWG findings...

Nightly tests showed a drop in the number of tracks with 15+ hits on 2016-07-02, as shown in these plots of that quantity vs. date of the nightly tests in both Run 14 AuAu200 and Run 13 pp510. The red vertical bars show the freezing of libraries SL16d, SL17d, and SL18f, and the drop of note is not long after the freezing of SL16d, well before the freezing of SL17d:


(small side note: the Run 13 pp510 nightly test data goes back a little further in time because, of course, the Run 14 test sample didn't become available [wasn't acquired] until a later date)

I have no specific confirmation of whether DEV was successfully re-built on nights prior to the night before 2016-07-02, so the commits may have come on any of the few days before that. First, let me note that there were no commits into StDb, StarDb, StarVMC, nor pams on any of these dates; all commits were in StRoot and the ones into the MAIN CVS branch are listed here (there were commits into an StiCA branch which I am omitting). Looking at the commits day-by-day in reverse date order, I see:

UPDATE: Commits with a yellow background are identified as potential problem candidates by Dmitri's investigation.

2016-07-01
* StRoot/StGenericVertexMaker/StiPPVertex/Vertex3D.cxx
* StRoot/StSstDaqMaker/StSstDaqMaker.cxx

The change to Vertex3D.cxx was to patch the fact that the function StiKalmanTrack::getPoint() was no longer available. That missing function probably meant DEV did not build the night before that as that function was no longer available. So these changes are probably not our culprit (they look innocuous anyhow).

2016-06-30
* StRoot/StBTofCalibMaker/StBTofCalibMaker.cxx (unrelated to track reconstruction)
* StRoot/StBTofCalibMaker/StBTofCalibMaker.h (unrelated to track reconstruction)
* StRoot/Sti/StiDefaultToolkit.cxx
* StRoot/Sti/StiDefaultToolkit.h
* StRoot/Sti/StiHitTest.cxx
* StRoot/Sti/StiKalmanTrack.cxx
* StRoot/Sti/StiKalmanTrack.h
* StRoot/Sti/StiKalmanTrackFinder.cxx
* StRoot/Sti/StiLinkDef.h
* StRoot/Sti/StiLocalTrackMerger.cxx (removed)
* StRoot/Sti/StiLocalTrackMerger.h (removed)
* StRoot/Sti/StiToolkit.h
* StRoot/Sti/StiTrack.h
* StRoot/StiCA/StiCATpcTrackerInterface.cxx
* StRoot/StiMaker/StiMaker.cxx
* StRoot/StiMaker/StiMaker.h
* StRoot/StiMaker/StiStEventFiller.cxx

So the change that broke Vertex3D.cxx was committed on 06-30 as the StiKalmanTrack::getPoint() method was removed. The commit to StiMaker.cxx included (among other things) a fix for a compilation issue: the disappearance of StiCADefaultToolkit.h made to CVS on 06-29. So the list of commits from 06-29 are also fair game for what caused the change in tracking because the AutoBuild of DEV would've failed two nights in a row.

2016-06-29
* StRoot/Sti/StiDefaultToolkit.cxx
* StRoot/Sti/StiDefaultToolkit.h
* StRoot/Sti/StiDetector.cxx
* StRoot/Sti/StiDetector.h
* StRoot/Sti/StiKalmanTrack.cxx
* StRoot/Sti/StiKalmanTrack.h
* StRoot/Sti/StiKalmanTrackFinder.cxx
* StRoot/Sti/StiKalmanTrackFinder.h
* StRoot/Sti/StiKalmanTrackNode.cxx
* StRoot/Sti/StiLocalTrackSeedFinder.h
* StRoot/Sti/StiMapUtilities.cxx
* StRoot/Sti/StiMapUtilities.h
* StRoot/Sti/StiToolkit.h
* StRoot/Sti/StiTrack.h
* StRoot/Sti/StiTrackFinder.h
* StRoot/Sti/StiTrackNode.cxx
* StRoot/Sti/StiTrackNode.h
* StRoot/Sti/StiVMCToolKit.cxx
* StRoot/StiCA/StiCATpcSeedFinder.cxx
* StRoot/StiCA/StiCATpcSeedFinder.h
* StRoot/StiCA/StiCATpcTrackerInterface.cxx
* StRoot/StiCA/StiCATpcTrackerInterface.h
* StRoot/StiCA/StiCADefaultToolkit.cxx (removed)
* StRoot/StiCA/StiCADefaultToolkit.h (removed)
* StRoot/StiCA/StiCAKalmanTrack.cxx (removed)
* StRoot/StiCA/StiCAKalmanTrack.h (removed)
* StRoot/StiCA/StiCAKalmanTrackFinder.cxx (removed)
* StRoot/StiCA/StiCAKalmanTrackFinder.h (removed)

I'm not sure where to begin with all of those commits, and they may be difficult to disentangle from one another (i.e. difficult to test separately). If I understand Dmitri's investigation properly, then it indicates that in fact those commits are co-dependent and one would need to actually pick out changes to individual routines within the commits to isolate the culprit.

-Gene