2015 nightly tests shows variation over time and variation with respect to 32bit vs 64bit.  Question is why?

1. The 2014, 2015 and 2016 nightly tests were setup as two step productions: a simulation job followed by a reconstruction job.
2. For both sim and reco, misalignments were applied.
3. I note that y2015a geometry was used... this is incorrect at the simulation stage, as it does not apply the misalignments applied during reco (but should not introduce any features seen above)
4. Looking at the simulation stage, we immediately recognize a problem.  The primary event generated (pythia8 in this case) is not the same between 32 and 64bit.  A few things seem to be going on here:
   
   1. RNG was getting out of sync between 32 and 64bit
   2. Physics lists are a cause: 
      
      1. During initialization geant populates lookup tables for interaction cross sections
      2. I presume there are some distributions which are sampled using the rejection technique / or other branchings based on RNG output, which are giving rise to different numbers of random numbers required during initialization
      3. Would result in different list of random numbers during physics
   3. EM / hadronic cascades are another cause:
      
      1. During track propagation, again sampling techniques may run into this roundoff error + branching issue we have seen previously, throwing the rest of the event out of sync betweem 32 and 64bit.

Fix was to reduce the list of physics processes for the nightly tests.  When this is done, the first event in consistent in the number of hits and (to first order) the hit response.

Asked Amol to update the 2015 nightly test, and we will see the result next week.