October 6th, 2024

Recap of the Past 24 Hours:

• MCR provided 56x56 Au+Au collisions around 5 AM (with crossing angle but no rebucketing yet). Collisions were stable with low background. We focused on setting up global timing, made significant progress, and took one physics run. The beam was dumped at 9:25 AM. sPHENIX had a 20-30 minute access, followed by APEX and RHIC rebucketing.
• ZDC AND rate was 3 KHz. We started with tune_AuAu_2024. Global timing looked good, with TAC numbers for BBC (blue background was high, 2x collision rate). VPDs and TOF were turned on, starting with zdc-tac, then switching to zdc-mb.
• Ran tune_AuAu_2024 with TPC, TOF, BTOW, and L4 (no L4 events).
• Ran setup_AuAu_2024 with TPC, TOF, BTOW, and L4 (L4 events recorded, but no plots). Reached out to Jeff to resolve L4 issues, as plots were needed for global timing. Continued running without L4 until Jeff fixed the issue.
• The evp /a disk was 60% full; Jeff cleared files. l4Cali and l4Disp are still down. Diyu was contacted and will work on the calibration server within 2-3 hours.
• After receiving the L4 plot, global timing looked good, with only a 0.5 ns offset.
• Vz was off by ~20 cm. MCR wasn't seeing the real-time vertex, but this was fixed and improved.
• BBC setup plan: TAC is a bit off, but BBC looks good. Shift crew was instructed to take 5 runs per BBC HV setting, adjust config, collect 50k events, and log runs as BBC HV scan in ESL.
• Oleg initiated a cal_scan, timing within 1 ns. He will review the data later.
• Akio loaded BBC HV data.
• EPD experts took initial scans for TAC adjustment, with a follow-up planned.
• Run 25280025 was the first Au+Au 200 production setup; fastoffline was requested and is in progress (no forward tracker, eTOF, or GMT yet).
• Forward trackers (FST/sTGC) will be turned on after rebucketing (post-10 AM).

Issues Encountered:

• No major issues. BCW issue was solved after Tim accessed and fixed the create.
• l4Cali and l4Disp are still down, Diyu is working on it.

Plan for the Day (RHIC & STAR):

• Access now; RHIC will handle rebucketing.
• Forward trackers will be turned on mid-next fill (when background is low).
• EPD work is ongoing; experts will provide updates on EPD and calorimeters.

Looking Ahead:

• First Au+Au collisions at STAR (56x65) occurred this morning; one physics run was completed, and the next fill is expected in a few hours (delays likely).
• This fill wasn’t useful for sPHENIX; they’re taking access. They’ll request 6x6 for background studies when stable. RHIC is planning to install a 56 MHz RF on Monday for a narrow vertex. Stable beams are needed by Wednesday.
• Commissioning plan: compile a list of detector experts (EPD: Mike needs a call when the next fill starts, FST & sTGC experts need a call mid-fill, FST: time bin, sTGC performance will be checked and followed up, VPD will use run23 files).
• Update the detector readiness checklist (shift crew to continue with setup_AuAu_2024: TRG+DAQ+iTPC+TPX+TOF+BTOW+ETOW+L4+FCS).
• For 6x6, we’ll use minbias for regular production; fastoffline will be requested.
• Shift leader for the next owl found; the day shift will run with one DO.

Saturday, October 5th, 2024

Urgent Issues:

1. BCW VME crate repair: Tim was informed, but repairs are on hold due to RHIC's beam development
2. Shift leader missing: We have no shift leader for the next owl shift.

Recap of the Past 24 Hours:

• Collected cosmic data throughout the day as no beam development work was done (despite efforts).
• MCR called for APEX at 1:30 am but reported issues at 3 am.
• Encountered L0 trigger not starting and BCW components not working, which were removed.
• Yellow Abort Kicker issue continues.
• Blue & Yellow injection and ramp setup continues, updates expected in a few hours.

Encountered Issues:

• BCW configuration error and FPGA failure:

  • BCW and BCW_DSM2 nodes are missing from the component tree.
  • The BCW VME crate requires repair (need to discuss access and coordination).
  • Tim will replace the board and power cycle it locally, which should take 30 minutes (we have spare boards).
  • Follow-up with Tim; we can run min-bias without the fix and should take opportunistic access if available.

• L0/L1 not responding:

  • Power cycling crate 62 failed initially.
  • Prashanth went in manually, and David helped resolve it.

• 2000+ timeouts:

  • Stopped the run.
  • Shift leader consulted Akio, Jeff, and Hank.
  • Jeff tried multiple power cycles of L0/L1, and it eventually came back.

• EPD hot tiles:

  • Restarted the run, and the issue was gone.
  • Maria will investigate further.

• EVB23 issue:

  • EVB23 is still out of the run, with EVB22 and EVB24 expected to return soon.
  • Awaiting further updates from Jeff.

Plan for the Day (RHIC & STAR):

• Continue Yellow Abort Kicker work.
• Blue & Yellow injection and ramp scheduled for the evening.
• Ramp development overnight.
• Continue taking cosmic runs when no beam is available; switch to APEX otherwise.

Looking Ahead:

• First Au+Au collisions at STAR expected Sunday evening (possible delay). First fill could be 56x56.
• October 4-8: sPHENIX requests 111x111 initially, then 6x6 for background study when stable.
  • RHIC plans to install 56 MHz RF on Monday for narrow vertex; stable beam required before Wednesday.
• STAR needs/plans for bunches/luminosity:
  • TPC group requests the same crossing angle for 6x6 (one or two fills based on sPHENIX needs).
• Detector commissioning plan:
  • Compile a list of experts to call.
  • VPD will use Run23 files.
  • EPD requires calibration runs (contact experts).
  • FST (time bin), sTGC, and others need timing adjustments (trigger and global timing with JH/Akio), followed by cal-scan.
  • Follow up with experts.
• Update the detector readiness checklist:
  • Use tune_AuAu_2024 (zdc_mb), setup_AuAu_2024, and production_AuAu_2024.
• 6x6 production: We can use minbias and regular production settings.
• Shift sign-up issues: Dan is resolving these with various STAR institutions; we may run with one DO. Follow-up with Daniel/Pavel recommended.

Friday, October 4, 2024

Recap of the Past 24 Hours:

• Cosmic data: Collected cosmic data throughout the day due to no beam development work.
• Minor issues: Solved with help from experts.
• Yellow Abort Kicker: Still awaiting new tools to address the issue.
• RHIC power supply: Power supply work was completed; RF conditioning is ongoing.

Encountered Issues:

1. Run control GUI crash:

   • The GUI crashed and disappeared at the start of the run.
   • Re-established connection using XLaunch, which helped restart the GUI.

2. ETOW configuration failure:

   • Crate 1 and 2 experienced a failure.
   • After multiple unsuccessful reloads, ETOW was removed.
   • Expert rebooted the crates, but DAQ mon still showed errors (Crate 1). The shift crew followed suggestions from experts, but the issue remains unresolved.

3. TRG L0 issue:

   • Run failed to start due to a TRG L0 error.
   • Power cycling the VME trigger crate resolved the issue.

4. FCS dead:

   • The run was stopped multiple times (more than 60 seconds, three times consecutively) due to FCS dead errors.
   • DOs performed a power cycle on the VME trigger crate, which fixed the issue.

5. EVB23 issue:

   • EVB23 was dead in the component tree, preventing the run from starting.
   • Awaiting further action from Jeff (work in progress).

Plan for the Day (RHIC & STAR):

• Yellow Abort Kicker work and g9-blw-ps work.
• Ramp development overnight.
• Blue & Yellow injection setup overnight.
• Continue collecting cosmic runs when no beam is available.
• Perform the usual pedestal runs (FCS LED and others).
• Possibility of controlled access for 4–6 hours (open for interest).

Looking Ahead:

• First Au+Au collisions at STAR expected Saturday overnight (possible delay); first fill may be 56x56.
• October 4–8: sPHENIX requests 111x111 initially, followed by 6x6 for background study once stable.
  • RHIC plans to install 56 MHz RF on Monday for narrow vertex. Stable beam is needed before Wednesday.
• STAR needs/plans for bunches/luminosity:
  • TPC group requests the same crossing angle for 6x6 (one or two fills driven by sPHENIX needs).
• Detector commissioning plan:
  • Compile a list of experts to call.
  • VPD will use Run23 files.
  • EPD requires calibration runs (call experts).
  • FST (time bin), sTGC, and others (timing adjustments, trigger, global timing with JH/Akio), then cal-scan. Follow up with experts.
• Update detector readiness checklist:
  • Use tune_AuAu_2024 (zdc_mb), setup_AuAu_2024, and production_AuAu_2024.
• 6x6 run can use minbias and regular production settings.
• Shift sign-up issues: Being resolved; Dan is working with various STAR institutions. May need to run with one DO.

Thursday, October 3, 2024

Recap of the Past 24 Hours:

• We ran cosmic data all day as no beam development work was done.
• Ongoing Yellow Abort Kicker work.
• RHIC power supply work completed, but RF conditioning will require more time.

Encountered Issues:

• iTPC/TPC issues overnight: Power-cycling RDOs 1-4 did not resolve the problem. Currently, iTPC sector 1 RDOs 1, 2, 3 are masked, and iTPC sector 1 RDO 4 is not masked. (this is not correct according to Tonko) ZhengXi will address this.
• EVB23 in the component tree prevented the run from starting yesterday. The shift crew removed it. Jeff will look into it, and I will follow up.
• A hot tile on the EPD was found yesterday. Maria is investigating it, and I will follow up.

Plan for the Day (RHIC & STAR):

• Continue work on DX training.
• Yellow Abort Kicker work.
• RF conditioning of storage cavities.
• Blue injection setup scheduled for tonight.
• Yellow injection setup planned for tomorrow.
• Continue taking cosmic runs.
• Take usual pedestal runs (FCS LED and others).

Looking Ahead:

• First Au+Au collisions at STAR: Scheduled for Saturday overnight*, with the first fill possibly 56x56. (There may be a delay.)
• Oct 4-8: sPHENIX requests 111x111 initially, then 6x6 for background study once stable. RHIC plans 56 MHz RF installation on Monday for narrow vertex; stable beams needed before Wednesday.
• STAR needs/plans for bunches/luminosity? Remove crossing angle for 6x6 (to be determined).
• Commissioning plans for detectors: VPD & EPD need calibration runs, FST (timebin), sTGC, and others (all timing adjustments, trigger, and global timing). Follow up with experts.
• Plan for eTOF during this Au+Au run should mirror p+p configuration.
• Update the detector readiness checklist (setup_auau200_2024_minbias, tune_auau200_2024_minbias, auau200_2024_minbias). Jeff will manage this, and the shift leader will review.
• Urgently need a shift leader and detector operator for next week’s owl shift. Dan Cebra is unreachable; Frank will follow up.

Wednesday, October 2, 2024
 

Recap of the Past 24 Hours:

• We ran cosmic data all day as no beam development work was done.
• Akio implemented the initial file for the BBC high-voltage run for Au+Au 2024.
• David Tlusty updated the TOF control for TOF West sector 5, resolving the previous issue.
• ESMD was turned off as per Jacobs' request and will be excluded from future runs.
• MCR postponed the Yellow Abort Kicker repair to today.

Encountered Issues:

• EVB23 in the component tree prevented the run from starting. The shift crew removed it, and Jeff will look into it.
• One hot tile on the EPD was identified. Maria is investigating.

Plan for the Day (RHIC & STAR):

• No significant progress on DX training yet; it may start this evening or tomorrow morning.
• Yellow Abort Kicker work is still ongoing.
• RF conditioning: Most parts are running smoothly, and most components are conditioned.
• Blue injection ramp setup is scheduled for tonight, with Yellow injection tomorrow.
• We continue to run cosmics but will prioritize expert needs.

Looking Ahead:

• First Au+Au collisions at STAR are expected Friday overnight, with the first fill potentially 56x56.
• Oct 4-8: sPHENIX ramp-up/stochastic cooling, aiming for 111x111 (or possibly fewer), followed by 6x6 for background studies (planned for Tuesday) once silicon and TPC are stable. RHIC plans to install 56 MHz RF, and we'll explore the use of 6x6 (Gene will advise).
• STAR needs/plans for bunches/luminosity: Remove the crossing angle for 6x6 (to be determined).
• RHIC DX training and Yellow Abort Kicker work access will be completed.
• eTOF plan for Au+Au should remain the same as for p+p runs.
• Update the detector readiness checklist (setup_auau200_2024_minbias, tune_auau200_2024_minbias, auau200_2024_minbias). Jeff will follow up.

Tuesday, October 1, 2024
Recap of Past 24 Hours:

• Yesterday we had access till from 8:00 till ~16:00
• Took cosmics for rest of the day, went to apex around 4 am and then cosmics again ~6 am. 
• CAS brought the magnet down at the beginning of the access for some cleaning work. Ramped up at about noon.
• TOF: problem with W5 POS HV is fixed, so we are back to 100% ToF acceptance
• eToF: replaced TCD fanout box, works now (?)
• FST status checked, no further refilling needed
• After the access, STAR detector was put in the APEX/beam development mode. Again took cosmic when no beam related work was performed

• BSMD:  CAEN HV system failed; no spare parts (controller boar) are available. BSMD is off now and not included in the run configuration. Most probably BSMD will be unavailable till the end of the Run24.
• MXQ crate went off without apparent reason, fixed by power-cycling the crate

• Opportunity for restricted access from 8:00 – 18:00 (DX training for collider), some DX training for tomorrow (access opportunity tomorrow), please let us know if you want to use this
• We continue running cosmic but prioritize expert’s needs
• Separate work for Pulsed Power group for yellow abort kicker (need 6 hours of work)
• RF conditioning of storage cavities overnight
• Injection and ramp possible (APEX mode tonight !!)

• First Au+Au collisions at STAR expected Wednesday (Oct. 2nd) night or Thursday (Oct. 3rd)
• RHIC power supply not ready, DX training, access for yellow board kicker, crossing angles setup 
• What is the plan for eTOF during this AuAu run ?
• Update the detector readiness checklist
• Geary is asking about David’s schedule, IOC related work for TOF, Alex will contact him and cc Geary

 

Monday, September 30, 2024

Recap of Past 24 Hours:

• Overall: Physics and data taking, the last p+p store ended at 08:00,

Encountered Issues:

• BSMD HV did not turn ON. Shift team contacted Oleg, and then removed BSMD from the run configuration (starting at about midnight).A few common issues with TPC RDOs, Trigger/RunControl, and sTGC

Plan for the Day:

• access opportunity till ~16:00
• CAS brought the magnet down for some cleaning work. It needs coordination with Prashanth.
• Plans for the access:
  • Inspection if FST cooling refill is needed
  • TOF: W5 POS HV cable to be moved from 7.5 to 3.0
  • Time permitting: eTOF: check TCD connection to the rack
• After the access:
  • we plan to bring the magnet up
  • STAR detector stays in the APEX/beam commissioning mode

Looking Ahead

Tuesday, Oct. 1st: We expect a few hours of access opportunity
The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Sunday, September 29, 2024

Recap of Past 24 Hours:

• Overall: Physics and data taking

Encountered Issues:

• DAQ: EVB24 started causing problems at about 8 am today, removed from the run configuration after consulting Tonko. Jeff will work on solving this problem later today.
• ESMD: MAPMT FEE  4P1 turned red for a while, the problem fixed itself automatically
• A few common issues with TPC RDOs and Trigger/RunControl
• B1U polarization measurements not available for the current fill (#35153) due to broken target. The issue showed up at the end of fill #35152.

Plan for the Day:

• Physics 

Looking Ahead

Monday, Sept. 30: Maintenance day

• the last p+p store ends at 08:00,
• then ~8 - 9 hour of maintenance
• Plans for the access:
  • We plan to keep the magnet on
  • Inspection if FST cooling refill is needed

Tuesday, Oct. 1st: We expect a few hours of access opportunity
The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

 

Saturday, September 28, 2024

Recap of Past 24 Hours:

• Most of the time: Physics and data taking
• Problems with FCS ECAL. Akio and  Tim made access at 19:40, replaced one of MPOD modules for FCS ECAL power supply, reconfigured and turned back on.
• We are running with ToF without West Sector 5, so with 90% of ToF acceptance (and 90% of expected ToF multiplicity). Jeff modified IDs for triggers that include ToF Multiplicity to keep track of runs with this state.

Encountered Issues:

• Evening shift noticed 2 new cold tiles in the EPD West ADC plot. Performed "Reboot All" according to the EPD cold tile response manual, but that did not solve the issue. It should be resolved after running pedestal_rhicclock_clean. Run pedestal_rhicclock_clean was postponed to the end of the fill, and done in the morning.
  • Note to shift teams: please follow the EPD instruction carefully: After “Reboot All”, one needs to run pedestal_rhicclock_clean, even if we have beams.
• A few common issues with TPC RDOs, Trigger/RunControl

Plan for the Day:

• Physics 

Looking Ahead

Physics for the rest of the week
Monday, Sept. 30: Maintenance day

• the last p+p store ends at 08:00,
• then ~8 - 9 hour of maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Friday, September 27, 2024

Recap of Past 24 Hours:

• One hour access for sPHENIX at 10 am, then some problems with injection.
• back to Physics at 13:49, then data taking
• EEMC HV monitoring is in "paused" mode to reduce the number of GUI crashes

Encountered Issues:

• TOF: "over current" alarms in TOF HV West Sector 5, followed by the "Positive - HV West Sector 5" internal trip.
  • Shift team contacted Rongrong, no success with resolving the problem during the night. Geary was trying to fix this issue today in the morning, but no success so far. We can run with the current ToF state (without TOF West Sector 5),
    which gives 90% of ToF acceptance, so 90% of ToF Multiplicity. We need to change the IDs for triggers that include ToF Multiplicity to keep track of this situation. To fix the problem, we need a short access.
• TOF+MTD gas system lost communication with Slow Control.
  • Alexei solved the problem by restarting the program that provides the connection to the Slow Control database.
• Run Control disappeared, solved after contacting Jeff (RTS02 machine died)
  • Jeff will work on preparing a spare machine in the case RTS02 dies for good.
• FCS trigger was running very high, which caused FST 100% dead in DAQ (two cases). Resolving the 1st one required  “reboot all”, the 2nd - fixed after restarting the run.
• EEMC GUI communication problem, solved by the shift team.
  • Note to shift teams: please read carefully the email from Will Jacobs on Sept. 26 about EEMC monitoring
• A few common issues with TPC RDOs, Trigger/RunControl

Plan for the Day:

• Physics 

Looking Ahead

Physics for the rest of the week
Monday, Sept. 30: Maintenance day

• the last p+p store ends at 08:00,
• then ~8 - 9 hour of maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

 

Thursday, September 26, 2024

Recap of Past 24 Hours:

• APEX (10:00 – 16:00), then machine development (16:00 -  18:00)
• back to Physics at 19:16, then smooth data taking

Encountered Issues:

• A few common issues with TPC RDOs, Trigger/RunControl, and sTGC
• EEMC communication problem, resolved by the shift team
• Shift team noticed single hot channel in Sector-22 from the TPC Sec. 22 charge per pad plot, but no corresponding peak in the RDO-bytes plot. The were unable to identify the RDO, and not sure if it's a known issue.
  • It is safe to continue data taking in such a case, but pay attention to TPC dead time.
• FCS trigger was running very high, which caused FST 100% dead in DAQ. Resolved after restarting the run.
  • Note to shift leaders: Please pay attention to trigger rates. If the trigger rates are red, restart the run.

Plan for the Day:

• Physics 

Looking Ahead

Monday, Sept. 30: Maintenance day

• the last p+p store ends at 08:00,
• then ~8 - 9 hour of maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Wednesday, September 25, 2024

Recap of Past 24 Hours:

• Overall: Physics and data taking
• TPC Grid Leak: David reset the HV power supply,  TPC Grid Leak is back to normal conditions

Encountered Issues:

• A few common issues with TPC RDOs, Trigger/RunControl, and sTGC
• EEMC communication problem, solved after calling the expert

Note to shift teams: this EEMC communication problem, and how to resolve it, is discussed in the EEMC manual. Please check it before
calling the expert.

Plan for the Day:

• APEX from 10:00 to 16:00,
• then 2 hours of machine development (16:00-18:00) (test of a new
  polarimeter target)
• return to Physics at about 19:00   

Looking Ahead

Monday, Sept. 30: Maintenance day

• the last p+p store ends at 08:00,
• then ~8 - 9 hour on maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

 

Tuesday, September 24, 2024

Recap of Past 24 Hours:

• Overall: Physics and data taking
• 1 hour access for sPHENIX at 2 pm, shift team took Cosmics during the access
• We are running with very stable beam condition, so there is no need to take bgRemedyTest_2024 runs. They were removed from Detector Readiness Checklist

Encountered Issues:

• TPC Grid Leak: sectors 20 and 24 lost their voltages. Alexei tried to fix the problem, but without success. Also intermittent alarms for
  sector 23.
  • We will request ~0.5 hour access today, after the current fill, to fix the problem (change the board)
• FCS trigger rates were very high, which caused FST going 100% dead in DAQ. Restarting a run resolved the problem (cleared the bad FCS conditions)
• TPC gas PI-10 yellow alarm, shift team contacted Alexei, the alarm
  cleared by itself.
• A few common issues with TPC RDOs, Trigger/RunControl
• sTGC ROB #12 power cycled.

Note to shift leaders: please read carefully the "TPC reference plots and issue problem solving" manual. If there is a single hot channel,
there is no need to stop the run.

Plan for the Day: Physics

Looking Ahead

Wednesday, Sept. 25.:

• APEX from 10:00 to 16:00,
• then 2 hours of machine development (16:00-18:00) (test of a new
  polarimeter target)
• return to Physics at about 19:00   

Monday, Sept. 30: Maintenance day

• the last p+p store ends at 08:00,
• then ~8 - 9 hour on maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Monday, September 23, 2024

Recap of Past 24 Hours:

• Overall: Physics and smooth data taking
• Prashanth worked on the sTGC gas system. Shift team needs to monitor sTGC PT-1 pressure through database since its alarm is disabled, and call Prashanth if its pressure drops below 19 psi.

Encountered Issues:

• A few common issues with TPC RDOs
• Yellow alarms for sTGC PT-2 and PT-3 pressure.  Shift team consulted Prashanth. The pressure should be monitored. Call Prashanth if the pressure drops down to 14-15 psi.

Plan for the Day: Physics

Looking Ahead
- plan for tomorrow: Physics
- APEX on 9/25
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Sunday, September 22, 2024

Recap of Past 24 Hours:

• Overall: Physics and smooth data taking
• Alexei refilled GMT gas bottle

Encountered Issues:

• A few common issues with TPC RDOs and Trigger/RunControl
• sTGC gas interlock alarm went off (PT1 gauge showing a high pressure). Shift team contacted Prashanth and turned down the pressure regulator on the gas tank outside the TPC gas room.
• sTGC ROB #8: was power cycled and after that its current was lower than expected (0.5A vs 0.7A). Shift team consulted Prashanth. After starting new run, the issue resolved itself.
• FCS: DEP05:1 failed. The shift team called Tonko, restarting the run solved the issue.
  Note to shift leaders: Please read error messages carefully. In this case, one should try restarting the run one more time before calling the expert, as the error message describes.

Plan for the Day: Physics

Looking Ahead
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

 

Saturday, September 21, 2024

Recap of Past 24 Hours:

• Most of the time Physics and smooth data taking
• Beam dumped at 19:00 for access for sPHENIX
• Shift team took Cosmics during the access
• Back to Physics and data taking at 0:30

Encountered Issues:

• A few common issues with TPC RDOs and Trigger/RunControl
• BSMD: HV GUI lost connection, resolved after consulting the expert (Oleg)
• Important: If there is an issues with BSMD: do not wait for BSMD, remove BSMD from the run configuring and start the run. Then try to resolve the problem.

Plan for the Day: Physics

Looking Ahead
- Physics during the weekend
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

 

Friday, September 20, 2024

Most of the time Physics and smooth data taking

Encountered Issues:

• Shift team was confused what should be the state of the VME #55 GP-Pulser.  It must be off!
• Trigger: Critical error message for run #25264020 and #25264018: "l1 CRITICAL l1Task l1Task.C:#281 Got an invalid token: 0"
  • Shift team consulted Akio and continued data taking. Jeff will investigate this issue. If this error happens again, it is safe to ignore, but please make a note in the Shift log.
• Inconsistent temperature alarms on VME #51 and #55
• A few minor issues with TPC RDOs and configuring MXQ_QTD
• sTGC PT-2 and PT-3 gas alarms went off, shift team consulted Prashanth
• Shift team stopped the laser run when noticed peaks in the TPC ADC vs time plots.
  • During laser runs, one should expect spikes in the TPC ADC distribution, so no reason to stop the run. Please consult the "TPC reference plots and issue problem solving" manual.

Plan for the Day: Physics

Looking Ahead
- Physics during the weekend
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Thursday, September 19, 2024
 

Recap of Past 24 Hours:

• APEX till 23:30, then ~1 hour access for sPHENIX
• Unexpected beam loss during the first fill.
• Physics started at 3 am.
• eTOF: Do not include eTOF in a run until further notice from experts.

Encountered Issues:

• RunControl:
  • Could not stop run from the run control. GUI froze. Resolved after calling Jeff.
  • Configurations waiting for more than three minutes, runs were notstopping. Resolved by "reboot all" multiple times after consultingJeff.
• Hot channels in TPC
  • If there is a single hot channel, there is no need to stop the run. Flemming has sent to STAR operations email list his detailed suggestions for dealing with the hot channels in TPC:
  • Spike in RDO bytes
    • Identify the RDO
    • Is it a single hot channel?
      • Yes: continue the run and reboot at end
      • No: (for example: Noisy full RDO, Noisy FEE or parts)
        • Stop run and reboot, unless it’s a laser run stop when there are >3000 events recorded.
    • Example from last few  days
      • 261017 sector tpx 12-6  single hot pad
      • 263013 (last night) ~16 channels in TPX 18-6

Plan for the Day: Physics

Looking Ahead

• No maintenance on 9/25
• Maintenance moved to 9/30 (start of Au run)
• The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Wednesday, September 18, 2024

• Access for sPHENIX at 10:30 am yesterday, then some problems with cryo
• Shift team took cosmics during the accesses,
• Detector Readiness Checklist was updated (updated eTOF procedure, a note about GMT)
• Physics started at 5:16 pm, data taking till 7:30 am this morning
• Akio uploaded new FCS Ecal HV files
  

• Run control GUI froze completely. Fixed by Jeff by terminating the Run Control GUI remotely on his end, then the shift leader started the GUI.
• A few minor issues with TPC RDOs.
• Error configuring MXQ_QTD Node -> Shift team powercycled VME crate MXQ, which resolved the issue.

• APEX from 8:00 till 23:00.
• Dmitry will do maintenance on databases.
• Tonko will work on RDOs and TCD.
• Access for sPHENIX at 23:00 to work on cooling of MVTX (estimated duration: 30 minutes)
• Back to physics at about midnight

• No maintenance on 9/25
• Maintenance moved to 9/30 (start of Au run)
• Pp run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
  

Tuesday, July 30, 2024

• TPC sector 11 channel 8 anode tripped, clear the trip manually

• spike for the TPX sector 5 in this run, power-cycled

• sTGC ROB10 channel 1 fluctuating

1st power dip (~11:00):

• Lost the control for all the detectors. We got the global and sTGC interlock alarm, lost the power to the platform, lost water, network, MTD gas, air blowers.

• PMD Power was off in the interlock page

• powercycle the VME EQ4 crate 

• All back on ~ 14:33

• TPC LASER controls were reset, we see pico drivers alive now.

2st power dip (~15:10):

• MCW is running, but magnet tripped 

• reset the FST cooling

• Turn on the BBC and ZDC. VPD is responding, so turned them off

• BCE in red in the component tree, then fixed

• Will recovered EEMC 

• BTOW, BSMD, ETOW, ESMD, FCS have been tested and ready to go. (18:04)

• Magenet tripped (18:41)

• restored control of TOF/MTD/eTOF HV and LV.

• pedAsPhys run with TOF+MTD+TRG+DAQ, now only TOF tray 117 error remains, Rongrong masked out this tray.

• Rebooted crate #63 (MXQ), rebooted trg/daq/l2. Now this run finished properly without any errors.

• Magenet tripped again (21:41)

• unable to turn on the VPD

Current issue:

• “Bermuda” computer has a problem, Wayne had an access but couldn’t fix it. Copy the disk to a new one now, it is running ~30% at ~9:30. Wayne is also preparing a new desktop for this in the meantime. 

• MCW was lost due to blown fuses on the 80T chiller (for the MCW). Water is back online. Only MCW was lost, everything else is fine. (~6:20),

• Lost the communication of TPC air blower (didn’t trigger the global interlock). - David & Tim

• VME processor in Crate80 initiallizing correctly, but not communication. But right now is BTOW is back

• GLW lost communication, need to be checked during access/ or David can re-establish com. - recovered - Tim

• Can't start run due to mxq_qtd: qt32d-52 VP003 QT32D is dead - 63 crate - Hank will call control room

• mix, mix_dsm2 - 69 crate - Need a physical power-cycle - Tim

• Laser can be turn on but can’t be tuned

To shifters:

 

• Shiftleaders please pass all the informations to the next shift, walkthrough all the problems happened during the shift, and the remaining problems

• check the weather before processing the recovery, just in case there will be another thunder storm/power-dip happens soon

• clean the control room

Monday, July 29, 2024

Status & Issues:

• TPC: 

  • #25210022,  a spike in the TPX RDO_bytes plot for sector 4. Power-cycled. 

  • #25211009, ITPC RDO S04:1, power-cycled

  • #25211016, iTPC RDO iS17:2, TPX S13:4, power-cycled

  • TPC Anode Trip, Sector 11 channel 8, 5 times - apply 45V down, will also remind the SC expert

• Laser:

  • The laser can turn on but is not able to tune. Prashanth will try to fix it during the next access (Monday afternoon/Wednesday). 

  • Now the procedure for laser run is: 1) Warm up the laser in advance for 5 minutes and do not try to tune the laser. 2) After 5 minutes, start the laser run. Do not tune the laser during the laser run.

• Trigger:

  • #25210037 couldn’t start the run, rebooted TRG+DAQ 

  • Carl did a test for the new trigger configuration. Need to do a quick check at the end of this fill

• sTGC:

  • Red alarm from sTGC Air blower AC failure, the problem cannot be fixed during the run, need to have access. It triggered sTGC interlock after about 20 minutes.  DOs powered down the HV & LV. Shifters switch the bypass key from the left side to the right side following the instruction from David.

  • David had short access ~ 18:30, then the sTGC blower AC was restored. (~18:50)

  • sTGC ROB 10 channel 1 (sTGC::LV2::114::1::imon) keeps making yellow alarms repeatedly and quickly disappears. (~01:12). 

• TOF:

  • Prashanth & Jim restarted TOF/MTD archiver from the TOF machine in the gas room. Changed SF6 cylinder and Freon cylinder.

• FCS:

  • Error in “FEE count 44, expected 55; FEE count 33, expected 55 -- restart run. If the problem persists contact expert”. Then got a “configuration error”. DOs power-cycled the FEEs and reboot the FCS in the run control. But still have the same issue. Called Oleg.

  • a problem with FCS ECal North. One of the MPOD PS boards shows 'outputfailure Maxs' all V and currents are at 0. It is not clear if it is a failure of MPOD itself, or if it is caused by one of the ECal FEE. 

  • Gerard found that FCS power channel u4 configuration readback values were wrong, looked like all defaults. Likely, this channel got a radiation upset. Reconfiguring the MPOD with setup script 'setup_FCS_LV.sh' restored correct operation

  • FCS: DEP08:3 failed, restart the run fixed the problem

• Network:

  • MQ01 server: Disconnected the MQ01 server, unplugged all 4 disks from the MQ01 server, installed in the backup server (labeled in STARGW1), and then connected the backup server online with Wayne’s help. After rebooting the server, things seem to be working fine. DB monitoring is also back online.

  • TOF/MTD Gas monitoring: went to the gas room, and started the EpicsToF program. The PVs start to update online. Alarms cleared.

  • EPD: Tim forced a reboot of TUFF1 and 2. Now the EPD GUI reports "connected". Working fine now.

Schedule & Plans:

 

• cosmic 13:00-19:00 request by sPHENIX, access: AC. FCS S 10, VME 62, BBC East 1, the fan of TOF(east vpd); reboot scserv (Wayne), TPC Laser (Prashanth)

• Physics for the rest of the time

• Low luminosity tomorrow or Thursday (6x6)
  

Sunday, July 28, 2024

Status & Issues:

• TPC: 

  • #25209041, iTPC S13:1, DOs power-cycled it

  • #25209057, TPX  S02:6, DOs power-cycled it

  • #25209065, 100% TPX/ITPC deadtime for over 1 mintue

  • #25210015, iTPC S09:3, DOs power-cycled it, but still get the same error, masked it out

  • #25210020 - TPX S22:04, higher value in the TPX Totall bytes per RDO, power-cycled it after the run

• MTD:

  • #25209043, some hot strips in the MTD strips vs BL (CirticalShiftPlots->MTD-<StripsvsBL) plot

• Network;

  • 19:15, EPD, EPD: TUFF[2] dead - check TUFF if RUNNING!; 19:25, lost the connection; QA plots look okay

  • 00:00, TOF/MTD Gas; lost the connection; The computers in the gas room running ok, it is just the online database stop updating

  • DOs visit the gas room once an hour, check the gas values in-person, Alexei provided some inputs on which value we can look for

  • Lost the control of laser for camera 1 and 3

  • call from Wayne. He said the online monitoring network issue is caused by MQ01 computer. He let us to reboot the MQ01 and check the net work connection of dashboard1 computer in the DAQ room. The MQ01 is dead, will try to replace the power supply. 

• Others:

  • DAQ rate is a little bit high

  • TPC pulser crate #55 is in an unknown state! Please make sure it is OFF! - it is off

Schedule & Plans:

 

• A short access after this fill (request by sPHENIX), physics for the rest of the day

• Tomorrow afternoon - 6 hours cosmic request by sPHENIX
  

Saturday, July 27, 2024

Status & Issues:

• TPC: 

  • ITPC S11:2, masked out

  • TPX S19:3, power-cycled; Shift Crew should look for spikes in rdoNobytes, and if spikes look for appropriate sector adc plots—details in the TPC reference plots and issue problem-solving slides.

  • TPX S01:6 (#25208024), power-cycled

  • iTPC S21:2 (#25208045, #25208046), power-cycled

  • ITPC S16:4 (#25308048), power-cycled

  • (#25208050 - #25208053) ITPC S17:1, S04:1, S16:4, power-cycled

  • (#25208057) a spike in RDO_bytes plot TPX S11:4, power-cycled

  • (#25209003) ITPC S16:4, DOs power-cycled it

  • (#25209005) ITPC S07:1, DOs power-cycled it

  • (#25209007) ITPC S17:4, DOs power-cycled it

  • (#25209016) ITPC S04:1, DOs power-cycled it

  • (#25209019) ITPC S16:6, S16:3, DOs power-cycled them

• Environment alarm:

  • Had a temperature alarm again (13:30), followed by a series of similar alarms for different subsystems on July 22. Called MCR and Jameela. The CAS watch and AC people came and fixed the problem (~15:14). Jameela scheduled an AC maintenance on the next maintenance day.

Schedule & Plans:

• physics for all-day

 

Friday, July 26, 2024

Status & Issues:

• TPC: 

  • TPX: RDO S21:6, power-cycled

  • iTPC S02:1 power-cycled, still create problem, masked out  

  • TPX[28] [0xBA1C] died/rebooted -- restart a new run and it looks good

  • 25207049-25207052: ITPC: RDO S18:4 , many auto-recoverys, again in the late night (25207059), power-cycled the it

  • (22:48) TPC Anode sector-1 channel-5 tripped, shifters tried to clear the trip it didn't worked. So, individually cleared the trip following the manual.

  • ITPC: RDO S11:2 -- auto-recovery failed. Powercyle this RDO manually & restart run. (25208018, 25208019 )

• FCS:

  • fcs10 issues:  It gets stuck in fcs10 HCAL South FEE scan, Tonko increased the logging level to capture it in the log for the next occurence

New guide for FCS If the blinking issue happen again, try follows:

1) Powercycle FCS HCAL South FEEs in the FCS slow control.

2) "Reboot" FCS in the run control

3) Start a new run tun

4) If that failed, mask out the FCS[10] and record that in the shift log

• TOF:

  • (#25208020)  Several TOF trays in error and do the auto-recovery. got the a red alarm from TOF LV THUBNE at same time. After the auto-recovery done the red alarm disappeared.

The list of TOF tray in error:

TOF: tray # in error: 66 68 69 70 71 72 73 74 75 76 77 79 80 81 82 83 84 85 86 87 89 90 91 92 93 94 95 122 -- auto-recovery, wait 6 seconds…

• (#25208022)TOF THUB NE auto-recovery and triggered the red alarm. Alarm disappeared after the auto-recovery finished

Schedule & Plans:

 

 

• physics for all day and weekends

• Cosmic in next Monday (likely) requested by sPHENIX, Carl & Xiaoxuan & JH will work on the triggers during that time

• Works plans on the list: AC. FCS S 10, VME 62, BBC East 1 & bwoo6, the fan of TOF(east vpd); reboot scserv (Wayne)

Thursday, July 25, 2024

Beam until around 15:30 (extended since 7:00); We had a short access to fix bTOW problem after beam dump; APEX until midnight; running physics until this morning.

Status & Issues:

• TPC: 

  • (25206021 & 022) iS02:1, masked out; tpc.C:#621 RDO4: automatically masking FEE #7 error

• Laser: 

  • Jim showed shifters about how to operate Laser

  • Checked the magic crystals for the TPC lasers. The quantity of crystals is good and should last several more days.

  • Alexei and Jim decided to increase the amount of methane flowing to the TPC (slightly) to try to increase the drift velocity. (It has been falling in recent days). So I turned FM2 clockwise by 3mm at the end of the index needle.

• TOF gas: DOs switched from TOF Freon Line B to Line A

• BTOW: Oleg and Yu made an access, replaced blown fuses for crate 0x0b it is configuring OK. Powercycled PMT box 39 (on separate power supply) and restore communications with boxes 41,42 and 39. BTOW sysetm restored and ready to go.

• FCS: DEP10:6 in unmasked at 22:30 during fcs_led_tcd_only; but create problem when try to start the emc-check at the beginning of the fill (1:04). Tried try reboot trg and fcs, doesnt’t work; tried to only mask the 10:6, doesn’t work; masked the 10; - Tonko will look at it

• Run control:  Run control was frozen this morning right before the beam dump, couldn't close the windows at the beginning. Force it to close with the windows task manager, but couldn't bring it back after several try. Called Jeff, found vcx-server was not running in the background.  Run control is back after rebooted the vcx-server (xlaunch). Since it happened in the end of the fill when the beam is about to dump, the problem didn't affect any physics run.  - shifters can use the old shitcrew PC (in front of shift leader desk, RTS02) to start the run control if this happens and stop us to start/stop a physics run in the future

• Network: 

  • Any new host attempting (e.g. yesterday rebooted sc3) to connect to scserv initially fails in the same way. Wayne want to reboot scserv to see if it changes anything, but want to hold off until a maintenance period.

  • Temporatory solusion: if this issue is encountered again, please wait two minutes and try connecting again. 

• Others: 

  • #25207018: 

    • 06:03:03 1 tcd CRITICAL tExcTask mvmeIrqLib.c:#477 UNKNOWN EXCEPTION: Task 0x01DFE148 suspended, exception 0x00000400.

    • 06:03:03 1 tcd CRITICAL tNetTask mvmeIrqLib.c:#477 UNKNOWN EXCEPTION: Task 0x01DEDA70 suspended, exception 0x00000700.

  • #25207019: EPD West hit count shows two (relatively) not-very-efficient areas. Issues disappeared in the next urn;

Schedule & Plans:

 

• Machine development is cancelled, so physics for all day

• sPHENIX is addressing the suggestions got from the safety walkthrough for the isobutane, no clear schedule yet; Carl and JH will try to test the low luminosity trigger configurations on Friday morning (Carl & JH), Carl will send a guide to summarize trigger configuration exam did last time

• Works plans on the list: AC. FCS S 10, VME 62, BBC East 1 & bwoo6, the fan of TOF(east vpd); reboot scserv (Wayne)

Wednesday, July 24, 2024

Status & Issues:

• SC3: 

  • lost control on VPD,BBC,ZDC and VME crates due to sc3 CPU crash. David bought control of VPD/BBC/ZDC back at SC5; Wayne came and rebooted SC3

• BTOW: 

  • configuration failed error around 20:50; Tried restarting the run, but the caution persists. Then realized this might due to the crash of sc3 

  • Oleg T. found three BEMC PMT boxes (39, 41, 42) are dead, and they are masked out for now. 

  • Error at 05:21:09:  1 btow CAUTION btowMain emc.C:#467 BTOW: Errors in Crate IDs: 11;BTOW: configuration failed 

  • At similar time, VME-9 emcvme9_i4val made a red temperature alarm. (5:43), Oleg suspects that the issue with the BTOW is due to the blown fuse.  

  • Also have a problem on connecting to VME processes on the platform, for BTOW data collector and BTOW canbus, 

  • An access is requested after this fill for Oleg, and Wayne (if needed)

  • now running without BTOW+BSMD

• GMT:  trip at u3 channel. DOs performed a reset trip operation.

• Trigger: Hank points out the document about how to fix the trigger related problem for shifters (https://www.star.bnl.gov/public/trg/trouble)

• FCS: DEP10:6 failed again, masked from the component tree. To the shifters:

  • If it is DEP 10:6 problem, masked 10:6 and run (already masked)

  • If it is entire DEP 10 problem, take FCS out from the run, contact Tonko

• Others:

  • STAR control room door handle is fixed

  • An “umbrella” is installed to temporary fix the ceiling leaks

  • J.H. opened a BERT window for the beam-beam parameter. Now we can check the beam-beam parameter by it. 

Schedule & Plans:

• APEX for today (July 24) 8:00 - 00:00, - problem on AGS RF cooling water, the beam extended 

• Machine development assigned for tomorrow (July 25) 11:00-15:00

• Still no clear timeline about when sPHENIX will flow the isobutane / have access / low luminosity runs - exam the trigger configurations on Friday morning (Carl & JH), Cail will send an guide to summarize trigger configuration exam did last time

• Works plans on the list: AC. FCS S 10, VME 62, BBC East 1 & bwoo6, the fan of TOF(east vpd) 

 

Tuesday, July 23, 2024

• unexpected beam abort (~ 20:06)

• MCR had a fake ODH alarm, but based on the safety procedure, still dump the beam earlier (~06:20)

Status & Issues:

• TPC: 

  • TPX: RDO S09:5, recovered after start a new run

  • #25204040, The TPC went 99% dead, this indicates it is external to TPC.( By doing a reply of daq you will see that at 12:27 the JP1 SCA triger rate goes to 3 mHz)

  • #25204053, many ITPC RDO auto-recoveries and 100% TPX/iTPC dead time

  • RDO4: automatically masking FEE #7 

  • power-cycled TPX: RDO S15:6

• EPD: Mariia Stefaniak tried to fix the EPD problem,  reboot TRG and DAQ and take some pedestal_rhicclock_clean runs 

• sTGC: Before #25205016, shifters restarted the sTGC LV and found some empty lines in the sTGC hits/FOB and empty space in hits/Qudrant. Power cycled after this run, and thing back to normal in the next run.

• EEMC: 

  • (Day shift) red+blue indicator for sector 1, P1 (most inner circle) at the EEMC MAPMT FEE GUI. DOs followed the manual and solved the problem

  • a new noisy PMT in ESMD starting from Run# 25204041

• Tigger: 

  • (at 9:45am): 1) STP reset is failing. Runs will not work, please power cycle L0/L1 crate #62 2) STP reset finally worked. Do not power cycle L0/L1 crate

  • L0 and L1 got stuck on TRG + FCS, shifters rebooted all component; still fail to start the run, FCS keep blinking; called Jeff; take the fcs[10] out; run could start. But today’s morning it is working again

  • #25204066: There was a warning in daq monitor for L2. Event timed out by 97 ms, Token 861, Total Timeouts = 11, Timed Out Nodes = MIX_DSM2::dsm2-23. - Will be discussed in the trigger meeting

  • #25204068: BQ_QTD[trg] [0x801E] died/rebooted -- try restarting the Run. Shifters tried rebooting trigger, didn't work. Then rebooted all, run could be started.

• Others:

  • Takafumi brings it up that the reference QA plot is out of date (https://drupal.star.bnl.gov/STAR/content/reference-plots-and-instructions-shift-crew-current-official-version), will add a list of recent good run as an example of additional reference

  • The control room AC is still leaking, Jamella came and said will try to fix it ASAP

  • The door handle (white door) to enter STAR control room is loose - call MCR and maintenence team

Schedule & Plans:

• Physics for the rest of the day with 6 hours of fills 

• Possible chance to access after the sPHENIX isobutane safety walk-through (start at 11:00) in the afternoon. Works planed last time: AC. FCS S 10, VME 62, scaler board 5 (BBC E) & bwoo6 (Chris Perkins), the TOF east vpd -  we decide to wait for the next access

 

Monday, July 22, 2024

Status & Issues:

• TPC: power cycled TPX: RDO S02:3; RDO iS19:1 bad (#25203050, and a few runs after  25203051), powercycled this RDO, but did not work, masked it out

• #25203031 & 25203044 - The shift crew noticed in the QA plots that RDO_bytes have a spike around 75 (TPX Total bytes per RDO), - may related to the dead time

• FCS: Tim came and had access around 2 pm; Tonko with Tim checked the fPRE DAQ link for sector 10:6, the DEP board#13(from 1) in crate #3 (from 0). The issue remains after replacing the patch cable and SFP module...but in any case, from further evaluation, the issue seems to be w/ the DEP board. Time constraints for the access did not allow for enough time to replace + configure + test a new board in the system. DOs unmasked sector 10 RDO 6. Again not working at ~ 23:22, shifters took it out. But showing ok during the midnight shift.

• EEMC: Will reconfigured MAPMT box 1P1 (a.k.a ESMD crate @71) at ~ 10:05. OTOH it very simply responded to a reconfigure so appears ok. Then it again tripped many times during the day. Still shows the errors in Crate IDs: 71. Follow the manual can clear the trip, I will notice all the shifters about clear this trip manually. 

• VME: VME62 got stuck. DOs reset it (14:56)

• Environment alarm: 

  • TOF LV -> (East vpd) Terminal voltage triggering the yellow alarm from time to time over from ~16:00

  • Wide Angle Hall temperature is 30.7 degrees at 17:36 (yellow alarm); raise to 31.1 at 18:29 (red alarm). VME Crate 55 (no in-used) temperature yellow alarm at ~ 19:00; sTGC LV ROB# 10 current alarm at ~17:16 (yellow); VME Crate 51 PS Air temperature transient yellow alarm at ~19:26. Called MCR, and they sent CAS Watch to STAR to have a look. Looks like the AC in the IR is not running, so the 2nd platform shows a high-temperature alarm, but the original diagnosis is they need to have access to fix it. Since the temperature is still ok to run, we scheduled access for the CAS watch and AC guys to come, investigate, and fix it at the end of the fill (midnight). Then they found both ACs for WAH were down. They have successfully turned on one AC, and the temperature has started to decrease. Since the temperature is gradually back to normal, and running ok now, we will keep running until the next access

  •  

• Trigger: 

  • #25203026, “The run got stopped due to: L2[trg] [0x8201] died/rebooted -- try restarting the Run”, could not start a new run for 2 times, reboot the trigger and everything was running again

  • Jeff updated the low rate prescale setting for fcsDiJPAsy*EPDveto - Good so far

  • Hank power-cycled scaler board 5. Tim checked the patch cable for the BBC E PMT signal. The cable is connected and visually seems fine. But still no response. We will need to check further at scheduled maintenance access.

• Others: 

  • water leak at STAR control room, which seems to be from a bad sealing of the AC, AC team got informed and ordered new parts to fix the problem; 

  • Waters outside STAR assembly hall, maintenance team got informed. They shut the water down. 

  • If similar thing happened, called MCR first (and/or Jameela), and then maybe water group x4668

• BERT: the system freeze time to time so the notice doesn’t pop up, keep an eye on the BERT system, restart it if needed 

Schedule & Plans:

• Physics for the rest of the day with 6 hours of fills 

• We are now running with one AC on in the IR, looks fine so far, will try to schedule a work once there is a chance to have a long time access. So for the next access: AC. FCS S 10, VME 62, scaler board 5 (BBC E), the TOF east vpd

 

Sunday, July 21, 2024

It was quite a smooth day for our data-taking.

Status & Issues:

• TPC: #25202047 stopped due to TPC dead time (TPX: RDO S18:3 -- auto-recovery) 

• Laser: DO and shift crew should check both drift velocity and charge distribution vs phi plot. The latter should show spikes at about the sector centers.  Two examples are printed and left near the laser PC and shift leader's desk. 

• ETOF usually stocks about 3-5 minutes after the beginning of the run with the errors: ETOF has 1136>1000 EVB errors. It keeps happening. We are currently running without ETOF. Do we want to include it?

• FCS: FCS10 is ready to go after Tonko power-cycled the DEP crate. The DEP10:6 remains masked. -  request an access for 30 mins

• ESMD warning: "ESMD: Errors in Crate IDs: 71 -- refer to Endcap Operations Guide, Data Transfer Issues", run with this warning error for the rest of the shift

• EPD: Run 25202062 - The shift crew observed a new cold tile in the EPD West <ADC> plot.

• Trigger: Hank noticed the BBC East Scaler board 5 has problem

• Others: Ceiling leaks at STAR control room (at the top of the coffee maker table), called site maintenance, they are sending people here; another leaking is found in the assembling hall (in front of the gas room), called site maintenance

Schedule & Plans:

• Physics for the rest of the day with 6 hours of fills (Significant more down time now, need to discuss if longer fill is ok in tomorrow’s meeting)