The STAR experiment

From Lanny Ray:

Single run issue:

  1. Run 23049049 decreased number of hits in mid-section of Sector 8, RDO 7

  2. Run 23052056 iTPC sector 13, number of hits very low; ADCs low.

  3. Run 23053023 st_fwd and st_physics data streams, the dE/dx mean is much too high and the ionization energy band resolution is very poor.

  4. Run 23055041, Run with problematic Scalar. Gene investigated and determined that the absence of RICH scalers in the first part of this run prevented TPC distortion corrections from being calculated and caused those st_physics files analyzed by fastoffline QA to fail. The files used for st_fwd QA did have RICH scalers and produced normal QA histograms. These QA analyzed files were taken at about 1000 s into the run.  Gene recommends that the first part of this run, without RICH scaler information, be marked BAD, but the rest are good.

Some general issues:

   Tracking: 

      Starting last week the QA shift crew (Leszek) noticed that the distribution of the number of primary tracks was starting to decrease. Gene constructed a plot showing the trends. This plot confirms Leszek’s observation. The degradation in primary track yield appears in MB, HT and st_fwd XX trigger groups in the QA (see QA issue 18109).

   Vertex Z:

      Primary vertex z-distribution: We checked the fast-offline QA for the dip in PV distributions at z=0 and found that it goes back as far as the QA is still available, about 2 weeks to 20 days. It appears in MB, HT and XX. The MB case was hard to recognize because the usual st_physics stream has very few triggers and PVs in the plots. 

      We note that the global track DCA distributions are broad (from -200 to +200 cm) and peaked near z = 0 as expected, even when the PV distributions dip at z=0.

      Interestingly yesterday’s runs 23055064, 065 with very low ZDC/EPD rates, where sTGC settings were being tested, show normal looking PV distributions.

From different PWG groups:

   1. 23050020-> is laser run, after loss beam. Not any thing physics. 

   2. Continue to see strange multi-peak Vz distributions; located at +-50cm. Also see this from the hlt monitor trigger. Need to ask experts about the real issues.

   3. We see TPC Vz - VPD Vz = 10cm from HF with 45cm vertex cut; TPC Vz - VPD Vz = 5cm from BulkCorr without vertex cut.
 

For Run-22 offline QA:
From Lanny Ray:
  Single run issues:
    1. Run 23037073 all padrows in outer sector 8 appear hot, but the number of hits is reduced. Runs before and after are OK.
    2. Run 23039057 the number of hits in sector 23, RDO/anode 2 are non-uniform and generally low, unlike either an RDO or anode voltage issue and more like a problem with multiple FEEs. Flemming and Tonko discussed this briefly but as it only affected one run there’s not much they can do about it.
    3. In run 23037050 st_fwd data stream, the dE/dx mean is much too high and the ionization energy band resolution is very poor. The st_physics data dE/dx for this same run are normal. -> only st_fwd stream, the st_physics stream is normal.
    4. In run 23035011 there is an enhancement in the number of BTOW clusters from f = -2 to -1 rad.

 For multi runs:
    5. Starting in run 23037042 and continuing to the present, there is an extra spike in the BTOW cluster energy distribution at about 4 GeV.
    6. In run 23038042 the dEdx, TOF and MTD-track matching QA plots are empty. In the past this indicated loss of VPD start time or database issue. 
    7. In the last few days many fastoffline production jobs for st_fwd have seg-faulted causing many runs to have empty QA plots, and therefore no offline QA examination of these data could be done. Some of the crashed jobs are for runs 23039018, 031, 036, 040 and 23040003. Problem persist yesterday. There are likely others. -> Gene checked into the problem, fixed it and today fast offline processing for st_fwd is working well and is beginning to catch up with the QA from yesterday (day 041), but failed jobs from days 039 and 040 remain. Gene stop running the crashing code, it is not fixed yet, but will done soon. Not likely to do the QA for 039 040 again, 

From pwg: 
 1. Some of the problems only see in JP trigger, not in other triggers. Most problematic runs are short runs.
 2. BulkCorr group found that Vz has multiple peaks, very different from what we see weeks ago (See slide3 from Ashish’s report).
 3. Continue to see the offset in |TPC Vz - VPD Vz| ~ 5cm

For BES-II data QA:
 1. For 14.6GeV QA, found runs with 2 RDO that is problematic, that will likely to reject 12M events.
 2. For 3.2GeV QA, it is starting but have not get report ready yet. Will schedule a meeting to discuss the QA details. 
 3. Li-ke and Ashik are doing consistency check, will report next week.

For Run-22 offline QA:

From Lanny Ray:

  1. Run 23042061 TPC Sec. 5 RDO 6 was only partially filled with hits. QA issue # 18095. -> happen several times during Run22. Some minor issues with RDO, started a new run will fix the problem. 

  2. In run 23043075 st_fwd data stream, the dE/dx mean is much too high and the ionization energy band resolution is very poor. The st_physics data dE/dx for this same run are normal.

  3. In runs 230045045 the dEdx, TOF and MTD-track matching QA plots are empty. In the past this indicated loss of VPD start time.

  4. In runs 23043051 and 044033 there is an enhancement in the number of BTOW clusters from f = -2 to -1 rad. -> 043051 is a very short run.

  5. Run 043051 only has 1 file processed so far, while 044033 lists ZDC/EPD AND rates as “-1/-1”. -> A very short run, no samples in the database, will get “-1/-1”. Again, if it is a very short run, SL should mark it as bad.

From Cold QCD:

  1. 10% BEMC multiplicity gradually decreases from 23042001 to 23045006. Also from Run 23045006 To 23045007, there is a jump. It looks like there are several “BTOW configuration failed for crate Id 1. • Had to powercycle iTPC RDO S4:3.” error continues to happen in some of the subsequent runs around that time.

  2. Run 23042056 To 23042057, there is a jump 20% increase of EEMC tower Et. Did not see anything in the run log that might explain what happened.

From BulkCorr & HF:

  1. Continue to see strange multi-peak Vz distributions; do not know what the problem is. We start to see this since last week from st_physics branch.

  2. Also continue to see TPC Vz - VPD Vz = 5cm

For BES-II 14.6GeV QA:

  1. Li-Ke and Ahsik updated the run-by-run QA results with duplicated 14.6 GeV data.

  2. 100 consistent bad runs, Ashik find 5 extra bad runs,Li-Ke find 10.

  3. Total 1470 runs, 369(25.1% of total runs) injection runs excluded, 115 (7.8%) bad runs flagged

  4. Total 651M events, 42M(6.5%) injection runs, 44M(6.7%) bad runs

For Run-22 offline QA:
From Lanny Ray: 
1) On day 018, (last Tuesday) the anode voltages in grid area 5 in sectors 8 – 11 were turned off. This persisted from runs 23018065 – 23019026 as reported with QA issue 18064. Flemming was notified. 
    -> Flemming: There were massive TPC anode crash at that time, voltage disappear, DO have to manually set it up. So lots of confusion at that time, it was set to a wrong value 100voltage?, not zero. 
2) TPC tracking: Run integrated signed DCA were shifted >0 on day 015, then shifted <0 towards the end of day 016, remaining so through early Tuesday (day 018) 
   -> ZDC counts were low at day 015, ZDC scalar is not correctly for a few days. There will be efforts to recalibrate this.
3) TPC tracking: Signed DCA vs run-time variations often exceed 0.1 cm.
4) BTOW: A section of towers were reported out for runs 23016081, 082 by D.O. Daniel Kincses (see left plot below). However, such large scale outages are difficult for the QA shift crew to detect when the histograms are dominated by unmasked, hot towers. -> These hot towers were masked starting on Thursday (day 020). -> It looks like experts mask these hot channels out. This is very helpful for offine QA, will continue to check this.

From pwg: 
1. Run23015034/35/36 has 0 good events, marked as questionable in Runlog: iupsal: TOF HV off, no TOF hits
   -> All the events are good if you are not using TOF data. These runs should show up on the bad-run lists for analysis using TOF. 
   -> Also, run 23015031 only had TOF hits for the first ~10% of the run.
2. VPD vz - TPC vz is now centering around 0
3. day 014 has power dip, so take cosmic whole day.

For Run-22 offline QA:
From Lanny Ray: 
  In run 23025052 RDO 11-2 was out, but not completely. The problem went away in run 025053 but returned in 025060, and remained fully masked out after that.
  In run 23027017 RDO 1-8 midpadrows displayed reduced hit number:
  In run 23024003 the dE/dx resolution was poor:
  In run 23022065 there are no TPC hits in the 4 files processed.
  In runs 23023022, 23027046 the BTOF, dEdx and MTD-TPC track matching plots are all empty. Usually this indicates the absence of a good start time for the collision. 
  Starting on Wed. (day 026) the run integrated signed DCA distributions were shifted negative < -0.8 cm.
     -> it maybe calibration issues, but calibration changes happen last week ~Jan. 21, not likely to have big impact on this week.

From pwg: 
 1. For run 23022065, Tof is out of this run, so 0 events with tof requirement.  
 2. JP2 and JP1 trigger ID changes (JP1: 890209 to 890229. JP2: 890210 to 890230), related to the exclusion of the forward detectors from these. 
     -> Gene: For MuDst, see trigger ID for JP2 and JP1: root -l -b st_blah.MuDst.root, then: MuDst.Scan("MuEvent.mTriggerIdCollection.mL1TriggerId.mId[]")
 3. Continue to see Vz - VPD Vz = 5 cm in fast offline QA (same as last week, a mistake in last week’s email).
 4. There is shift observed in <DCA> and pion yields, at the same run 23026056 seen by Bulkcorr group. Shift log says: "TPC sector 17 channel 3 tripped, "clear trip" didn't work. We cleared it manually following the operations from the instruction, worked".
 5. Tong check the QA plots for "890210 trigger for runs 22351048 - 23019026" as asked by Raghav. Not all of the files are available, so it is not very meaningful to report/mark the problematic runs. 
 6. TPC calibration happened on Dec.22, Jan 21. So there maybe something changed after these date. 

For BES-II data QA:
  1. Dan is goring through the final 14.6GeV QA list. 
  2. Data production is going well, will have QA for the other dataset soon.
  3. Li-ke is writing a document about the QA.

For Run-22 offline QA:
From Lanny Ray:
  1. Continue to see normal distribution for most of the runs. 
  2. All iTPC sectors were masked out in runs 23029012 and 013. 
  3. In runs 23032041 – 065 TPC sector 8 anode sections 5 and 6 appeared to be at very low voltage. Flemming asked Jeff Landgraf to include TPC plots for 2D ADC and clusters for the online QA shift crew to help them catch such issues earlier. -> Do not have good pedestal for these runs.
  4. sign-DCA have fluctuations -> Gene: Not much improvement can be done to sign-DCA fluctuations at this stage, as it is caused by pile-up tracks.

From pwg: 
  1. About Idx 15: 23026057 To About Idx 35: 23027046, lower values for nSigmaTofPi in JP triggers. -> need to compare with the offline QA, see if problems can be found.
  2. 23029047, low statistics, very short run -> should marked as bad
  3. Continue to see TPC Vz - VPD Vz = 5 cm

For BES-II data QA:
  1. 14.6GeV QA status update:  
    1). Still non-negligible difference between the results from Ashik and Li-Ke.
    2). 14.6GeV data are duplicated on RCF now, so the QA code should be able to use the same data files. -> Suggest Ashik and Li-Ke to run the code over again, so we can get more consistency. 
  2. QA For FXT runs:
    1). some of the dataset is ready.
    2). Run-by-run is on going, need to discuss the software issues for new users.