Run 22 pp500 production estimates - more streams

Here are some very rough projections for processing Run 22 data:
  • st_physics stream, ~650k files
    • P23if production in February-March completed ~2500 jobs per day
    • => Estimated total = 650k/2.5k = 260 days
  • st_fwd stream, ~1100k files
    • P24ia production without tracking in April-May completed ~4000 jobs per day
      • tape-drive limited, but not using all drives for just this...
      • ...concurrently processing BES-II data
    • P23if production with tracking in January completed ~1600 jobs per day...
      • ...but there were some memory issues slowing down jobs
    • True dedicated rate is perhaps around ~3000 jobs per day
    • => Estimated total = 1100k/3k = 370 days
  • st_W stream, ~50k files
    • => Estimated total = 50k/2.5k = 20 days
Estimate of full total = ~650 days (~20-24 months) for all of Run 22

NB: Run 22 data volume acquired was roughly ~50% above projected in prior Computing Plan and BUR.

Other factors:
  • SDCC Farm OS transition to Alma 9 will happen mid-production, which may force our hand on moving (finally) to 64-bit (the above assumed currently-used 32-bit), and that should be ~20% faster
    • We must be very careful to avoid and/or understand any impacts of OS change!
    • Last September, I made an estimate of CPU time per job for st_physics in 64-bit, and it was ~1.35 days per job (typically), which is consistent with bringing the ~2500 jobs per day number up to ~3000 per day on 4000 production CPU cores
  • Could possibly allocate more nodes for production (taken from analysis & embedding ) to reduce this by perhaps 25% at best??? Very unlikely to reduce by 50%
  • BES-II re-productions will continue to use some CPU
  • Run 23 AuAu200 has to be accounted in priorities
Best case scenario is ~400 days (~13-15 months) for all of Run 22, but we rarely achieve best case