04/07/2022 -Thursday

 

04/06/2022 - Wednesday

 

• Reduced AGS extraction currents, as requested by STAR, going well
• CeC studies continue but their schedule may change, be flexible

II. RHIC Schedule

• Today
  • Physics for STAR                    (Note change of schedule, next line)
  • CeC studies           16:35 – 00:35    (no beam, access is possible)

• Tomorrow (Wednesday)
  • Physics for STAR 
  • No scheduled access.   Jet needs work but risk analysis suggests don’t touch it.
  • p-Carbon spin direction measurements – 2 hours at end of a fill,  time TBD.

• Future
  • Thursday & Friday  CeC (Au Beam, 48 hours)   May be reduced to 2 shifts of 8 hrs, TBD, be flexible.

III.  Items from Shifts:

• Day Shift
  • Smooth running (except unexpected beam abort at 12:30)
  • FCS HCal, Trigger 100% dead, EVB-08, Run Control, Tonko@work - fixed
  • Multiple sTGC problems. Now missing two sectors of plane 3

• Evening Shift
  • Smooth running till ~6 PM. Beam dumped for CeC until 2:00 AM
  • STAR Access
    • TOF maintenance: replaced THUB-NW, CANBUS not working, replace microcontroller next access (today)
    • Prashanth & David checked sTGC reverse burn, no luck.  Stopped.
    • Akio loaded new HV file for FCS
  • Cosmics after 10 PM

• Overnight Shift
  • New shift crew, overlap day
  • Cosmics till 2:00 AM, Smooth production running after 02:00
  • ETOW, JEVP restart, Tonko@work
  • MTD and TOF, and sTGC are not happy

IV. To Do:

• Access on Tuesday afternoon (today); report to SL before entry
  • Access to start at 16:35 (Note change of schedule)
• No maintenance access on Wednesday. Develop plan to accommodate this
• Shift CeC studies to 4:35 PM, JH to negotiate, notify Ops list – Done
• Quick entry to unplug TOF cable (?)… recommend wait till CeC access
• TOF gas, Alexei and Geary to discuss
• Shift Leaders, please call Prashanth if sTGC trips
• Change sTGC HV and trip current … can be done between runs
  

   

04/04/2022 - Monday

I.  Summary of Operations:

• RHIC performance ‘pretty good’ over the weekend, some AGS injection issues
• AGS injection current lowered in response to lower effective cross-section used by STAR, generally good results
• Jet polarization avg of 54% - not in full agreement with AGS numbers, experts reviewing the numbers
• STAR took 18.5 hours of data on Saturday.  A new record for Run 22!  Sunday took 15 hours.
• CAD plans to work on AGS issues behind CeC activity

II. RHIC Schedule

• Today
  • Physics for STAR
  • CeC studies            16:00 – 00:00   (no beam, access is possible)

• Tomorrow
  • Physics for STAR 
  • CeC studies            16:00 – 00:00   (no beam, access is possible)

III.  Items from Shifts:

• Day Shift
  • Smooth running
  • Trigger 100% dead, VME crate 63 (MXQ), Tonko@work, sTGC, FCS, EQ2
  • Removed a few FCS triggers at 10:30 till end of fill
  • Access ~15:30, replaced LV power for splitter, TPC field cage currents - fixed

• Evening Shift
  • Smooth running
  • David T. tuned a few issues around the control room
  • VME crate 75 (with scaler board) fan failure at ~20:45, Christian access – fixed
  • L2 pedestals shifted at 20:00, expert analysis needed

• Overnight Shift
  • Smooth running
  • iTPC, TPX, Trigger 100% dead, Run Control, restart JEVP online plots

IV. To Do:

• Access possible Monday afternoon (today); report to SL before entry
  • TOF maintenance: Replace THUB-NW.  Magnet barrier to come down.
  • Ramp TOF LV down. Tune TOF gas. Estimate two hours to replace THUB. 
  • Prashanth to cycle sTGC – check reverse burn, check HV cables
• Another access possible on Tuesday afternoon.
  • Work with RHIC to give ~2 hours of overlap on Tues. afternon shift
• Likely reduced access to STAR on Wednesday … develop maintenance plans accordingly
• Experts to review L2 pedestal shifts
  

   

04/03/2022 - Sunday

I.  Summary of Operations:

• Access today (Sunday) at 3:00 PM
• CeC on Monday has been reduced to 8 hours, schedule TBD
• Polarization mixed;  Hi 40’s – Lo 50’s depending on when you look

II. Yesterday's News

• Lower effective pp cross-section (as seen by ZDCs) has been adopted across all platforms  (2.06 => 1.86)
• As a result, the Figure of Merit plots will look 10% higher than previous editions of these plots
• ZDC rates from MCR now to be 10% lower  ( 420 ± 100ish Avg  =>  380 ± 100ish Avg )

III. RHIC Schedule

• Today
  • Access at 3:00 PM to fix FCS
  • Physics for STAR

• Tomorrow
  • Monday April 4th       CeC for 8 hours    (no beams, access is possible)
  • Physics for STAR

IV.  Items from Shifts:

• Day Shift
  • Smooth running
  • TPX, sTGC, Run Control Froze, TOF gas, FCS, Tonko@Work - fixed

• Evening Shift
  • Smooth running
  • Online plots died, JPlots restarted
  • TOF LV and TPC FC IOCs, David@Work, BBC & BBQ – fixed
  • FCS needs work (lost Drell Yan and J/Psi triggers as a result)

• Overnight Shift
  • Smooth running
  • FST, sTGC Plane 4 Prashanth@work, TOF – fixed
  • EMC check at begining of fill – no FCS events (?) 
  •     [This is a configuration error, take FCS out or fix configuration; Tonko and Jeff to talk and determine best shift leader actions.]
  • Event building 3 left in a few runs early in the shift (?)
  •     [If it runs, its OK.  It ran so it seems it was OK.]

V. To Do:

• Access at 3:00 PM today to fix FCS;  fix Field Cage Currents
  • FCS pre-shower board needs work during the Sunday access
  • Shift Leader Contact  Oleg – Alexei – David – Tim  & Prashanth for Sunday access
  • Shift Leaders - take Drell Yan and J/Psi triggers out of the run until FCS is fixed
• Another access is possible on Monday
  • TOF maintenance: repair or replace THUB-NW.  Magnet barrier needs to come down.
  • Prashanth to cycle sTGC on Sunday and/or Monday
  • Please report to the shift leader before entering the STAR hall.
    

     

04/02/2022 - Saturday

I.  Summary of Operations:

• CeC on Monday has been reduced from 48 hours to 8 hours    (no beams on Monday, access is possible)
• More CeC in our future, TBD
• Polarization is good, overnight Hi 40’s, today Lo 50’s

II. Other News

• Lower pp cross-section has been adopted across all platforms   (2.06 => 1.86)
• Figure of Merit plots will look 10% higher than previous editions of these plots
• ZDC rates from MCR now to be 10% lower  ( 420 ± 100ish Avg  =>  380 ± 100ish Avg )

III. RHIC Schedule

• Today
  • Physics for STAR

• Tomorrow
  • Physics for STAR            

• Future
  • Monday April 4th       CeC for 8 hours    (schedule TBD, no beams, acesss is possible)

IV.  Items from Shifts:

• Day Shift
  • Smooth running till Noon, CeC till 8 PM, some cosmics taken
  • Polarization high 40’s low 50’s
  • MTD, sTGC & gas, Tim&Christian@Work on THUB-complete
  • TOF out most of the day
  • Shift log stopped recording, disks full, Jeff@work - repaired

• Evening Shift
  • Smooth running after 20:00
  • Polarization above 50%
  • iTPC RDOs, TOF trays, THUB masked/ quarter of detector out
  • TPC laser data taken with and without compensating resistor  (for TPC Calibrations in coordination with GVB)

• Overnight Shift
  • Smooth running
  • Polarization Hi 40’s – Lo 50’s, mid 50’s most recent fill
  • sTGC, BTOW, DAQ watch program, Tonko@work

V. To Do:

• TOF maintenance: repair or replace THUB-NW.  The Magnet barrier will need to come down.
  

   

04/01/2022 - Friday

I.  Summary of Operations:

• Many strange phenomena at RHIC yesterday, odd day.
• Ramp 33262 stopped ramp, many unusual issues, Vtx wide  (JH says good enough)
• OPPIS work overnight & behind CeC – successful after a few false starts
• Polarization in RHIC seems good, not sure if AGS numbers agree

II. RHIC Schedule

• Today
  • Physics for STAR              00:00 – 12:00
  • CeC no beam                  12:00 – 20:00      (access to STAR possible)
  • Physics for STAR              after   ~21:00

• Tomorrow
  • Physics for STAR            

• Future
  • Monday April 4th             CeC for 48 hours   (CeC with Au beams, no access to STAR)

III.  Items from Shifts:

• Day Shift
  • Beam dumped at ~08:30 for CeC
  • TOF, sTGC, Tonko@work, Christian-Prashanth-Oleg@work
  • CeC done at ~16:00 but start of odd problems in AGS & RHIC

• Evening Shift
  • Vacuum Valve failure and other unusual issues @ RHIC
  • RHIC ready ~22:00 …Ramp delayed…steering for beam…long wait
  • Minor TOF LV issues in last run of the evening
  • Polarization numbers in the low-mid 50’s

• Overnight Shift
  • Smooth running
  • TOF THUB, other TOF issues, Rongrong@work, TOF out of run
  • TPX, sTGC, EEMC, Polarimetry page & RunLog not updating
  • Reminder … Dump at Noon  (4/1/2022)

IV. To Do:

• Call Jeff if Run Logs aren’t updating … even late at night
• Additional details entered into the Shift Log would be useful as we come to the end of the run
• Update shift procedures, TPC Short Ops manual w/GG & Laser coming soon
• sTGC statistics for summer shutdown maintenance
• reflash TOF THUB micro-controller, suggest mask off RDO-1 and run TOF today
• ZDC cross-section discussion – JH to consider further.  Updates to CAD, Jamie & Vincent.

03/31/2022 - Thursday

I.  Summary of Operations:

• OPPIS source work overnight & behind CeC
• Beam dumped at ~08:30 for CeC
• Strong geomagnetic storm today; may affect electric grid.

II. RHIC Schedule

• Today
  • Physics for STAR              00:00 – 08:00
  • CeC no beam                  08:00 – 16:00      (access to STAR possible)
  • Physics for STAR              after   ~17:00

• Tomorrow
  • Physics for STAR              00:00 –  08:00
  • CeC no RHIC beams        08:00 – 16:00      (access to STAR possible)
  • Physics for STAR              after   ~17:00

• Future
  • Monday April 4th             CeC for 48 hours   (note impact on training shift crews)

III.  Items from Shifts:

• Day Shift
  • Beam dumped for APEX (w/protons) early in shift
  • sTGC, access to online, Magnet trip, MXQ[trg] – fixed
  • Alexei and David accessed the hall, power cycled TPC Field Cage Current monitor

• Evening Shift
  • APEX till ~22:45
  • iTPC & minor Trigger issue - fixed
  • new sTGC reference plot available

• Overnight Shift
  • Smooth running
  • Polarization > 50%, only Blue shows up on web page
  • BTOW, Jevp plots, EMC, EEMC, FCS, sTGC, Laser GUI - fixed
  • Trigger 100% dead

IV. To Do:

• Christian & team to make access
• Additional detail in the eLog would be useful as we come to the end of the run
• As always, try to maintain high shift efficiency – especially at the start of a fill
• ZDC cross-section discussion – JH to consider further

03/30/2022 - Wednesday

I.  Summary of Operations:

• Short stores, Blue lifetime low, AGS emittance high
• Polarization still a bit low (high 40s), injectors?
• Yellow abort kicker access & maintenance
• Strong geomagnetic storm on Thursday may affect electric grid.  (minor W & F)

II. RHIC Schedule

• Today
  • Physics for STAR              00:00 – 08:00
  • APEX studies w/proton   08:00 – 23:00
  • Physics for STAR              after   ~23:00

• Tomorrow
  • Physics for STAR              00:00 –  08:00
  • CeC no RHIC beams        08:00 – 16:00      (access to STAR possible)
  • Physics for STAR              after   ~17:00

• Future
  • Friday – CeC no beam     08:00-16:00         (access to STAR possible)
  • Monday April 4th             CeC for 48 hours   (note impact on training shift crews)

III.  Items from Shifts:

• Day Shift
  • Smooth running till end-of-shift (Pol high 40’s)
  • FCS, TPX, BTOW, iTPC, TOF – ordinary issues
  • Trigger 100% dead

• Evening Shift
  • Yellow abort kicker maintenance – no beam till ~20:00
  • TOF LV – disable and turn off tray 2, masked off
  • Relatively smooth running  (Pol high 40’s)
  • iTPC masked off  9-1, 16-3; other minor issues – fixed
  • Question – should we turn off FEE after masking off RDO’s?

• Overnight Shift
  • Smooth running
  • Polarimetry page not updating
  • sTGC, FST, BTOW, TOF, TPC Field Cage lost comms, Tonko@work
  • Trigger 100% dead

IV. To Do:

• Shift crews please add more detail regarding shift to eLog,  also record significant comms with MCR
• Access required to fix TPC field cage current logger
• Do not turn off FEE unless specifically asked to turn it off;  update to shift crew instructions needed
• Take IV scan on day shift – Akio to call STAR Control Room

03/29/2022 - Tuesday

I.  Summary of Operations:

• Yellow abort kicker, sextupole PS failure
• Polarization dropping overnight (50's to 40's)

II. RHIC Schedule

• Today
  • Physics for STAR

• Tomorrow
  • Physics for STAR   00:00 - 08:00
  • APEX studies        08:00 - 23:00
  • Physics for STAR   after 23:00

• Future
  • Thursday - CeC    08:00-16:00  (no RHIC beams)
  • Friday      - CeC    08:00-16:00  (no RHIC beams)
  • Monday April 4th – 48 hours for CeC (note impact on training shift crews)

III.  Items from Shifts:

• Day Shift
  • CeC Development & end effects   08:00-17:30
  • Cosmics most of the day, eTOF work, FCS, Tonko@work, HLT code rolled back
  • Access by experts: Laser, sTGC HV reverse burn continues

• Evening Shift
  • Smooth running after 17:30
  • TPX, BTOW, VME CPU Creighton5 (takes 3 minutes to boot, wait till complete)
  • Yellow abort kicker failure ~22:00, Sextupole PS failure same time, MCR access required

• Overnight Shift
  • Cosmics till ~ 03:00, then regular running
  • TOF lv/fixed, BEMC hv/fixed, Jeff magic@midnight, Tonko@work
  • Trigger 100% dead

IV. To Do:

• Can MCR work on pre-injectors during CeC?

03/28/2022 - Monday

I.  Summary of Operations:

• Today: Booster access 07:00-Noon; CeC development 08:00-16:00 work on injectors
• Booster cavity E6 repaired => 3 fills with low Pol, poor emittance, transverse position error, etc.

II. RHIC Schedule

• Today
  • CeC development 08:00-16:00  (Magnets at flattop, no ion beam)
  • Restricted Access to STAR Hall possible until 15:30 (no access to tunnels)
  • STAR will take cosmics during CeC work
  • Physics for STAR evening and overnight

• Tomorrow
  • Physics for STAR

• Future
  • 16 hours of APEX on Wednesday (no access at STAR)

III.  Items from Shifts:

• Day Shift
  • Smooth running, Pol low ~45%, beam position not ideal
  • Trigger 100% dead
  • JEVP server restart, TOF tray, power cycled scaler board 6 - fixed

• Evening Shift
  • Smooth running, Pol ~50%
  • sTGC FEEs power cycled, EEMC crate power cycled
  • L4 calib table errors, L4 rebooted ~22:00, VTX distributions improved, diagnostics improved

• Overnight Shift
  • Smooth running, Pol ~52%, beam position good
  • L4 error, TPX issues, FCS errors, RHIC monitors frozen/fixed, Tonko@work
  • sTGC chamber tripped, additional work today
  • ZDC rates ~550kHz new fill, L4 Vz histograms look good

IV. To Do:

• Alexei to make access, Prashanth to make access, Reverse polarity burn – update?
• Bill to walk through at Noon for property inventory
• Akio to load new voltage tables (not an access)
• RMS distributions on L4 undergoing review by experts

03/27/2022 - Sunday

I.  Summary of Operations:

• Booster rf problem, no estimate, trouble shooting continues
• Deer Management activities Evening and Overnight – call MCR if you have questions

II. RHIC Schedule

• Today - Sunday
  • Physics for STAR

• Tomorrow - Monday
  • CeC Development 08:00-16:00 (TBD)
  • Physics for STAR

• Future
  • TBD on Monday Morning

III.  Items from Shifts:

• Day Shift- Saturday
  • Boooster RF problem.  Held midnight fill till ~14:30, new fill ~16:30
  • Trigger 100% dead, screenshots saved
  • BBC crashed, lost comms to VME crates, SC3 crashed, etc.
  • Many good runs

• Evening Shift- Saturday
  • Trigger 100% dead
  • EEMC Radstone board problem/fixed
  • Smooth running (but beam quality not so good)

• Overnight Shift - Sunday
  • Trigger 100% dead, screen shots saved
  • Smooth running, but beam quality not ideal overnight, also current fill not ideal
  • Etow, sTGC, TRG, comms issues with MCR etc.
  • Discussion with MCR about when to dump beam due to issues at RHIC.  Held beam till 06:30, new fill ~07:30, JH suggests dump < 350 kHz

IV. To Do:

• Consider low luminosity fill if low quality beam continues - beam shifted low - poor polarization
• Scaler board 6 network issues - (last run day 83 run 61).  This impacts Lumi counting but board 5 is backup and should allow analysis to proceed.

03/26/2022 - Saturday

I.  Summary of Operations:

• Deer Management activities Evening and Overnight – call MCR if you have questions
• There still be daemons in RHIC ...

II. RHIC Schedule

• Today - Saturday
  • Physics for STAR

• Tomorrow - Sunday
  • Physics for STAR

• Future
  • Monday: CeC development 08:00-16:00 (to be confirmed)
  • Deer Management activities Evening and Overnight till Monday morning

III.  Items from Shifts:

• Day Shift- Friday
  • No beam all day
  • Tim worked on sTGC Blower.  Better?  (yes, so far)
  • Christian tuned RDO power, Tonko burned Proms, etc.

• Evening Shift- Friday
  • Beams in RHIC after 10:00 PM
  • Smooth running

• Overnight Shift - Saturday
  • Smooth running from RHIC
  • TPX & BTOW & TOF issues, EMC manual needs update re: alias for GUI
  • RF injector problem - not yet solved, no AGS beam, keep current beam till ~Noon

IV. To Do:

• Summer activity: check sTGC blower

03/25/2022 - Friday

I.  Summary of Operations:

• Deer Management activities Evening and Overnight – call MCR if you have questions
• CAD worked on Yellow abort kicker
• AGS calibrate pulse was found to be incorrect … explains (false) extraction losses
• Polarization has slipped slightly in both rings (< 55%), work on injectors proposed
• Last 3 fills for STAR were above Lumi target, but keep at this rate, tune later if necessary

II. RHIC Schedule

• Today - Friday
  • p-Carbon spin studies 09:00-14:00 during Day shift (no beam for STAR)
  • Physics for STAR remainder of evening and night

• Tomorrow - Saturday
  • Physics for STAR

• Future
  • Monday: CeC development 08:00-16:00 (to be confirmed)
  • Deer Management activities Evening and Overnight till Monday morning

III.  Items from Shifts:

• Day Shift- Thursday
  • Smooth running
  • TOF gas changed, TOF QA Plots funny/fixed, power cycle sTGC, etc.
  • Trigger 100% dead, screen shots taken and put in eLog

• Evening Shift- Thursday
  • Smooth running
  • Trigger 100% dead, Hank took some screen shots
  • L2 died, TOF, EPD, BTOW online plots not working, L2 was out of run - fixed

• Overnight Shift - Friday
  • Smooth running but ended with a double quench in yellow at 6:20 AM
  • sTGC minor issues/fixed, BEMC confusion/fixed, Tonko@work on RDOs, etc.
  • Trigger 100% dead, mystery continues, screen shots taken and put in eLog

IV. To Do:

• Experts, please double check health of systems in preparation for a good weekend
• Oleg to power up FCS radiation monitors

03/24/2022 - Thursday

I.  Summary of Operations:

• CANCELLED:  CeC e-beam development (no RHIC beam)  08:00 – 16:00
• Otherwise - no news is good news

II. RHIC Schedule

• Today
  • Physics for STAR

• Tomorrow
  • Friday: p-Carbon spin studies 09:00-13:00 during Day shift (no beam for STAR)
  • Physics for STAR remainder of Evening and Night

• Future
  • Monday: CeC development 08:00-16:00 (to be confirmed)

III.  Items from Shifts:

• Day Shift
  • STAR cooling tower was cleaned and put back into operations
  • Long access – many experts at work (e.g. TOF new HV board, sTGC blower bad sensor)
  • TOF HV glitch, repaired.  Some issues with default config needing update by DO
  • sTGC communications issues … alarm handler will have to be rebooted to fix the problem.  Shift crew can do this.

• Evening Shift
  • Late start ... but no major issues
  • Will Jacobs worked on ETOW hot tower issue; crate masked but still shows as hot channel in QA plots
  • Smooth running after 10:45 PM, minor issues. (e.g.  Frozen online histograms fixed)

• Overnight Shift
  • Smooth running most of shift
  • Shift Leader computer was erratic, fixed itself.
  • Lost communications with alarm handler, fixed itself.
  • BTOW PMT 10 empty bins, required cycling HV.
  • FCS issues, power cycle MPOD crate by expert
  • MTD had multiple errors on several occasions (54)
  • Trigger 100% dead a few times

IV. To Do:

• sTGC blower sensor – action plan?
• Work will continue on Trigger 100% dead issue; suggest reboot trigger whenever it happens
• Continue with negative voltage burn-in

03/23/2022 - Wednesday

I.  Summary of Operations:

• Relatively smooth data taking since yesterday; last 5 fills have lasted natural lifetime
• Siemens performance has been good for past 2 weeks, so confidence is growing that OK
• Luminosity a bit low, RHIC Ops hopes this will improve after today’s access (e.g. they will work on chromaticity tuning)

II. RHIC Schedule

• Today
  • AGS Access started at 07:00, RHIC Beam dumped 09:00 (on schedule)
  • AGS Access scheduled to complete by 14:00 so RHIC Beams by 15:00
  • This may slip but hopefully no later that 17:00 for RHIC beams
  • Physics for STAR remainder of Evening and Night

• Tomorrow
  • CANCELLED: CeC e-beam development (no RHIC beam)  08:00 – 16:00
  • Physics for STAR remainder of Evening and Night

• Future
  • Friday: p-Carbon spin studies 09:00-13:00 during Day shift (no beam for STAR)
  • Monday: CeC development 08:00-16:00 (To be confirmed)

III.  Items from Shifts:

• Day Shift
  • BEMC glitches
  • High bin at 97 in cr2Hot QA Plot for EEMC
  • STGC air blower temp not stable – sets off annoying alarm. (SlowC or STGC?)
  • Scaler rate for J/Psi were high in the middle of the day (not a continuing problem)

• Evening Shift
  • Smooth running

• Overnight Shift
  • High bin at 97 in cr2Hot QA Plot for EEMC – hot channel? (Will@work)
  • Long list of iTPC & TPX RDOs masked off  (~dozen),  could not unmask.
    (consultation with Tonko suggested before unmasking RDOs)
  • iTPC[9] prevented Run Stop.  Powered Down node in DAQ room & restart.
    (in general, contact Tonko or Jeff before power cycle nodes).
  • STGC air blower temp still unstable

Period coordinator change: Sooraj Radhakrishnan --> Jim Thomas

03/22/2022

I. Summary of operations:

• Good stable fills all through yesterday
• Vernier scans taken ~4.00 pm
• Updates on Trigger 100% dead issue ---- Experts looking into, some cases issue is from tokens lost, but not from busy detectors
• Maintenance day tomorrow. STAR Magnet off ~10.00 am
• Please inform on tasks/to do/plans. So far:
  • Cleaning of magnet cooling tower
  • Time for Tonko to work on TPC
  • Access for Prashanth+David (stgc cable reconnection)
  • Access for Xu (~hr) coolant for FST
  • Access for Alexei for TPC survey
  • Access for Oleg/Akio install radiation censors in front of FCS
  • TOF Rack 1C1 need to checked if cooling fans are working correctly (Alexei)

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • 07.00 - 17.00 :: Maintenance (RHIC from 9:00, Access at AGS behind store from 07.00), IP2 close at 14:00

III. Items from shifts:

• Evening shift:
  • Masked out iTPC Sector 5 RDO 1
  • New voltage files are loaded for both FCS Ecal and Hcal. Rates looks good. Production Ids for all FCS production triggers were incremented AFTER this run
  • For first time TOF noise seems to have shown up in production running. Also seems to be on BTOW TP in the same region (fuzzy TP pedestals and enhanced response around TP = 100) ---- origin of issue not clear. Jeff will look for large events in cosmic runs and if so could reconstruct and look into further
• Overnight shift:
  • Issue with FST and ETOW. FST Critical Recovery failed for RDO(s): 4. EEMC lost communication
  • DOs turned the FST off and back on that fixed FST, back in next run. ETOW out for a few runs. DOs recovered ETOW following manual. Details on shift log
  • Jpsi*HTTP Sca Hz, upcJpsiSN Sca Hz was a little to moderately high, noted from Run 23081008 ---- Do 'Prepare for Physics' on BEMC computer if this happens
  • STGC computer in control room has completely frozen. Switched to a different tty (using Ctrl+Alt+F3) and came back into the original tty (using Ctrl+Alt+F1) to fix ---- David will follow up
  • Because the STGC wasn't in a safe state before beam dump, missed the opprotunity to take the zdcPolarimetry_2022 run
  • Gas alarm for STGC for PT-1. It is rising and is now at 24
  • TOF LV East Sec 5 U5 temperature alarm of 42 (orange)

03/21/2022

I. Summary of operations:

• Unexpected beam loss 3 times. Some downtime bcos of this. Apparently still Yellow abort kicker issues
• Trigger going 100% dead a few times and needing to restart runs ---- Trigger experts are looking into the issue
• 8 hr maintenance day on Wednesday. Please inform on plans/tasks to do/requirements

II. RHIC Schedule:

• Today:
  • STAR Physics. Access for CeC taken, back to STAR
  • Vernier scans possibly today
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • FCS "FEE count 44, expected 55 -- restart run for two runs in a row. Contacted Akio
  • Shift crew pointed out extra spikes in FST Non-ZS noise vs channel for disk
  • Node 0xf001Is forcing the run to stop because: 494|rc_handler.c|L2 stopping run. Looks like BCW was the problem. Rebooted trigger after run stopped and starting a new run
• Evening shift:
  • Under ETOW cr2Hot shows hot tower at ~97. Earlier this week (day 77) this hot tower was gone
  • TPX and iTPC are regularly go 100% dead - probably 5-10x a run. Auto-recovery and auto powercylcing works 99.99% of the time, but it still seems like a lot of autorecoveries(?) ---- Tonko informs the occurrence is in fact less frequent for the past 5 days
• Overnight shift:
  • New spike in ETOW->Hot_Tower cr2, right next to the other spike
  • stgc pt-1 pressure just went above minor threshold. From online plot looks like a slow but steady trend upwards over last day or two
  • TOF HV was ramped to standby after unexpected beam abort, but DO noticed voltages stayed at full values (7k). Online status viewer confirms that TOF HV still at full. DOs working through restart TOF HV IOC procedure. ---- Call experts if happens

03/20/2022

I. Summary of operations:

• Got beams for STAR all yesterday. A few unexpected beam losses. Might be an access to work on some problems related to it at end of this fill
• No vernier scans taken
• Some recurring issues with ETOW during the shifts
• Trigger goes dead 100% several times. Dylan has put in a Python script on the SL computer to monitor to watch DAQ Monitor webpage and play sound if trigger is dead

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • EEMC crate issues, dont know the cause yet. Happened a few times, see below for other instances in all the shifts. Taken out of run and powercycled relevant crates and vme crates to fix
  • Caution message: "ETOW: failed crate Ids: 4". QA plot ETOW Hot_Tower cr4Hot mostly empty. EEMC GUI Tower FEE Crate 4 showed "Ready" light red. Most FEE crate voltages were in red. Monitoring plots related to that crate were nearly empty. After consulting the expert power cycled the corresponding CANBUS crate which restored the FEE crate voltages. After a master reload everything seems back to normal
  • Run would not start due to TPX 23-4 and 23-5 failed configuration. Masked out TPX 23-5
  • New sTGC caution messages:"2: FOB 0 (R#10:F#55) NOT found" and "2: FOB 5 (R#10:F#60) NOT found" (since changes David made yesterday, we always get "1: FOB 5 (R#05:F#30) NOT found", but these two are new). QA plots look fine ---- Prashanth will look into and ask Tonko to suppress these messages
  • FCS[1] stuck configuring and run could not start, power cycled fees twice to fix
• Evening shift:
  • ETOW: configuration failed -- watch ETOW triggers or restart run. Crates 3 and 4 are off and no tower response. Tried powercycling crates and following instructions for reconfigure but that didn't work. Cycled Crate 90 and started over and that worked
  • EEMC Crates 1&2 and MAPMT sectors 12-3 are bad. Monitoring plots show missing data. Problem with EEMC in last run was somehow HV was not applied to the cycled crates, even though they were all green and looked ok. Will fixed it but noticed there was an error with Crate 3 configuration
  • Monitoring plots interface not updating, restarted servers
• Overnight shift:
  • ETOW Hot_Tower spike in cr2 appears to be back, also new spike in cr4Critical_Shift_Plots->ETOW->Tower_ADC for cr2. We power cycled tower FEE cr1&cr2. Problem is resolved
  • Many fcs trigger rates are high (red). Stopped run and rebooted fcs from run control. Fixed
  • prom_check: trg + daq + itpc: “Incorrect Trigger clock 10.000 MHz -- restart run" --- not important, can be ignore. Also, prom_check can be taken with beam

03/19/2022

I. Summary of operations:

• Back from APEX around 12.00, but unexpected beam loss. Long downtime, beam back around 12.30 am. Did cosmic runs during downtime
• No vernier scan last night, might do later today

II. RHIC Schedule:

• Today:
  • STAR Physics, Vernier scan
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • West laser failed. Short access for Alexei to fix
  • Prashanth increased pressue PT1 as we were getting low alarms
  • David took out sTGC ROB #5 channels 3, 5, and 7. These should be manually removed when ramping up LV
  • Updated run control handler for support for new prom_burn and fcs_ivscan runs (Jeff)
• Evening shift:
  • Cosmic run: Noise in TOF and BEMC JP is back
• Overnight shift:
  • ETOW Hot_Tower spike in cr2 appears to be back, also new spike in cr4

03/18/2022
I. Summary of operations:

• Got one fill overnight. Unexpected beam loss just after fill after that. Yellow abort kicker issues during injection. MCR decided to go into pC spin direction measurements early, ~6.00 am
• Shift was smooth, no major issues

II. RHIC Schedule:

• Today:
  • 08.00 - 12.00 :: pC spin direction measurements. STAR Physics otherwise
  • Vernier scan during store after pC spin measurements
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Evening shift:
  • FST error code during pedestal. Disk1 module 12 alternate sensor current was fluctuating called Xu and he fixed. He will raise limit on this module so it doesn't happen again
  • Masked out iTPC S13:2
  • iTPC and TPX 100% dead during start of run, power cycled FEEs to fix. ---- likely from masked out RDO S13:2 in error state, power cycling fees was right way to fix
• Overnight shift:
  • Unmasked itpc S13-2 after instructions from Tonko

03/17/2022
I. Summary of operations:

• Dedicated time for APEX and CeC. Not much activity at STAR. Took btow_ht: trg + daq + btow every two hours

II. RHIC Schedule:

• Today:
  • CeC till 16.00, STAR Physics afterwards
• Tomorrow:
  • 08.00 - 12.00 :: pC spin direction measurements. STAR Physics otherwise

III. Items from shifts:

• Day shift:
  • TOF HV was turned to standby, but voltages not ramping down. DOs going through HVIOC restart procedure. Connection appears to be back, but gui does not seem to be responding still (voltages will not ramp up when HV turned on). David power cycled TOF HV crate, which fixed issue
  • Unmasked iTPC S13-2 and S20-4 following Tonko's instructions
  • Some delay in ramping up the magnet, was unable to turn on the main power supply

03/16/2022
I. Summary of operations:

• Access till ~12.00 to fix sextupole issues. Beam back around 1.00 pm
• Power dip and magnet trip at STAR ~4.00 pm. Lost connection with trigger detectors, TPC FEE GUI went completely white. Fixed by David. Run could not be stopped in run control. Needed to turn power back on to MIX, BE, EQ2, and LOL2 VMEs following instructions from Jeff. STAR back ~5.00 pm. Got next fill at 1.00 am

II. RHIC Schedule:

• Today:
  • APEX till 16.00 (STAR Magnet is off, Power supply repair behind APEX); CeC afterwards
• Tomorrow:
  • CeC till 16.00, STAR Physics afterwards

III. Items from shifts:

• Day shift:
  • Unmasked iTPC S2-3, 10-4, and 6-4 at Tonko's request
  • Updated fcs offline trigger ids to match fast offlines organization scheme (Jeff)
  • In shift critical plots, observe peaks in TOF ADC plots that were not present in previous cosmic data. Experts notified, but they don’t know the reason
  • Spike in BEMC DSM L0 Input - PatchSum plot. Oleg asked us to take a btow_ht run to debug, all looks ok
  • Online shift plots not updating so restarted JEVP server
• Evening shift:
  • pedestal_rhicclock_clean: trg + daq + fcs runs correctly but DAQ monitor but BCE gives errors. CRC Error on STP Build header. Event 1000, Token 1000, nPre 0, nPost 0
  • Cosmic runs: The unusual activity is back in the TOF and BEMC
• Overnight shift:
  • TOF THUBNW now staying at yellow alarm with terminal voltage 5.9, should be fine

03/15/2022
I. Summary of operations:

• Much smoother fills and shifts than yesterday
• Faster loss of intensity in the last fill, yellow beam has issues. Yellow sextupole issue
• Possible access for 2 hours from 10.00 am (work on sextupole)

II. RHIC Schedule:

• Today:
  • 09.00 - :: Fix RF (2 hrs behind the store)
  • STAR Physics afterwards
• Tomorrow:
  • 08.00 - 16.00 :: APEX
  • 16.00 - :: CeC dedicated time

III. Items from shifts:

• Day shift:
  • GMT was checked to be ok after the gas bottle change yesterday
  • "Reboot" Button for LV on sTGC sometimes does not complete correctly. Got the same error last week. It says "starting socket... / sudu: command not found” --- David will look into
  • High Sca Hz rates for BHT2*BBCTAC and BHT3 and JP2. iTPC and TPX at 100% dead and would not recover. If the Sca Hz for BHT2*BBCTAC and BHT3 are high, causing the iTPC and TPX to be 100% dead, click "perpare for physics" on the BEMC computer. Wait for terminal to say "Finished checking FEEs on all SMD Crates" ---- would be good idea to do "perpare for physics" on the BEMC computer between the fills
  • Masked out iTPC S2-3
  • The TOF PT-1 gas ran out, so we switched the valves
• Evening shift:
  • ETOW lost communication. Took out of run. Power cycled crate 90 (controls) and reconfigure all (from Will)
  • iTPC RDO 10-4 was masked out
• Overnight shift:
  • Shift crew noticed hole in BTOW ADC plot from runs 19 and 21. DOs determined it to be from crate 0x14 and the run was stopped. DOs reapplied HV

03/14/2022
I. Summary of operations:

• Unexpected beam loss 3 times, TOF and MTD HV alarms during beam loss
• Faster loss of intensity after beta squeeze during last night’s run. Yellow sextupole failure, fixed now, should be better from the current fill
• A few issues with MTD, EEMC, Trigger during the shifts

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • High Err numbers in many of the triggers on DAQ monitoring web page during one run (Run 23072031), stopped and restarted
  • dimuon trigger Sca Hz is red and sitting around 900 Hz with a Sca Dead of 18%,.The Daq Hz is low, (during Run 23072035)
  • EHT0*BBCTAC trigger had a high Sca Hz rate of 2500. EHT1 trigger had a high Sca Hz rate of 140 (during Run 23072044) --- could be from beam issues
• Evening shift:
  • While updating L2 pedestal reference for run23072042, L2 trigger crashed. Jeff asked Navagyan to restart L2 trigger and update the L2 pedestal reference for run23072043
  • Sudden beam loss and saw MTD low HV red alarms (value ~4600) on some sectors
• Overnight shift:
  • dimuon trigger Sca Hz is red and sitting around 1600 with a Sca Dead of 25%. Shift crew observed hot spot in MTD critical strips vs BL plot between 100-120 and 4-8 (Run 23073002+)
  • got TOF LV error (NW THUB current is red) so power cycling LV. Also itpc[19] preventing run from starting. Contacted Tonko for assistance ---- needed powercycling of one FEE causing issues
  • GMT gas bottle changed, but adjusted to have pressure at 14 psi. Expert might want to look. ---- Alexei will take a look. Shift crew are not supposed to alter anything than change the bottles
  • Unexpected beam loss and got TOF and MTD HV alarm
  • Temperature for crate 1 is at 0 on ETOW GUI. ETOW QA plots monitored closely but no issues seen. Continued runs with this caution messages. Expert instructed same and power cycle the crate and then issue a master reload at next beam dump
• Day shift (Today):
  • TPC high dead times, Powercycling fees didn’t help. From last pedestal run not completing ---- If the pedestal run doesnt complete and keep requesting more events, call the expert

03/13/2022
I. Summary of operations:

• Magnet trip around 5.30 pm, same issue as last Thursday. Couldn’t be ramped back till today morning
• Took zero field alignment runs with low luminosity beams
• Magnet ramped back around 8.00 am. At next opportune time possible, a more permanent fix to the circuit causing the issue will be done
• Online status viewer not loading ---- Dimitri was called to fix

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • BERT was saying we had Physics On for 5min after a beam dump ---- Communication issue, didnt repeat in later shifts
  • Took access for laser and interlock communication issue
  • Piezodrive froze and I came to WAH and restarted by AC power flipping — Alexei
  • Corrupt Trigger Data Event : Token 2213, Node qt32d-3 (Crate=5, Slot=6) (TCU Event Ctr 0x7f8b3343, Node Readout Ctr 0x7eb4aff2), similar msg in the next run also ---- not an issue
  • Noticed the Sca Hz associated with the FCS were high. Stoped Run. Rebooted FCS from run control
• Evening shift:
  • Made a controlled access and power cycled the crate #52 manually and interlock monitoring is working again — David
• Overnight shift:
  • "TOF: too many trays (111>5) require power cycle -- consider TOF/MTD CANBUS reset!" so will restart CANBUS
  • Unknown alarm at ~4 AM heard by entire shift crew, lasted only about 1 second. High-pitched train whistle like sound (not very loud) from area of gas/interlock alarm

03/12/2022
I. Summary of operations:

• Beam conditions are very good, peak luminosity touching around 600 KHz. ---- will keep watching, and will ask MCR to trim down a bit if needed
• Issue with the abort kicker, few hours till next fill during day shift
• Shifts were smooth, no major issues

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics
• Run officially extended by 2 weeks!

III. Items from shifts:

• Day shift:
  • Regained communication with FCS with help from David
  • Shift crew noted that the wide sTGC Diagonal Strips Plane 3 QA plot has been varying in intensity since run 23070017. STGC HV looks okay on GUI for Plane 3 ----- Powercycle LV to fix
  • Jeff made an update to the JEVP plots attempting to stop the crashes that have been happening periodically
• Evening shift:
  • iTPC: RDO S13:2 was masked out
  • The interlock alarm handler appeared white. Expert said that there seems to be some issues with Canbus 52, he was unable to bring the power back up, and suspects that it is a fan failure. Ignore and keep running for now, the electronics expert will try to go in and fix it at the next access ----- short access expected, so will take access at the end of this fill
• Overnight shift:
  • Gaps in sTGC hits/VMM, powercycled LV to fix
  • The laser cameras do not change when steering buttons are pressed. The laser is very poor for this run. ----- Alexei will check after the meeting

03/11/2022
I. Summary of operations:

• Beam conditions are very good after the switch to Siemens MG. Better luminosity, peak for last fill at 600 KHz, may want to reduce depending on STAR's preference. Better beam polarization (55% for both beams as per 9 am meeting)
• Downtimes due to STAR Magnet trip and Yellow abort kicker issues
• Magnet ramped back ~3.30 pm. Beam back around 4.00 pm, unexpected beam loss around 11.00 pm
• Shifts were smooth, some issues for FCS and ETOW

II. RHIC Schedule:

• Today:
  • STAR Physics (after the magnet is ramped back)
• Tomorrow:
  • STAR Physics
• CeC scheduled to have dedicated 4 days (not continuous) for run till April 4, starting March 21

III. Items from shifts:

• Day shift:
  • Oleg and Bill put ecal block on the east side
  • FST had a failure code 2, out of runs. FST Disk3 Module1 inner sensor shows a significantly higher current than other channels. seems like the inner HV cable of D3M1 broke. A 1 hour access is needed to investigate the cable issue. Currently running D3M1 with 120V and the current is around ~25muA. ---- Expert wanta an access, but not urgent, when available
• Evening shift:
  • iTPC RDO 14-4 masked out
  • ETOW out of 3 runs bcos of lost connection. Took a while with help from expert to bring back
• Overnight shift:
  • L3 event display only shows lasers on north side and none on south. During evening shift: laser was left on for longer than usual (>20min). Laser power was suddenly lost and the run was stopped ---- Likely from DOs not tuning correctly
  • Unmasked iTPC S4-2 and S16-4
  • There is a silenced IOC error for FCS LV (IOC_Monitor->forward-cr->FCS,FCS_LV). Online plots for FCS are normal. ---- Jarda will look into

03/10/2022
I. Summary of operations:

• Beams through all shifts for STAR physics
• Three vernier scans during day shift, took data
• Luminosity comparable to before, above 400 KHz for ZDCAnd_nokiller. Unexpected beam loss around 7.00 am today
• STAR magnet trip ~6.45 am. CAS tried to bring the magnet up three times without success. Trying to ramp again now.

II. RHIC Schedule:

• Today:
  • STAR Physics (after the magnet is ramped back)
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • Lost a HV board in the STGC. STGC and MTD out of run on expert recommendation. Back in after two runs
  • Stopped the run due to high rates in triggers associated with FCS. Called Akio and DO power cycled the LV
  • BTOW Pmt Box 10 had empty histogram entries. Cycled the HV of PMT box 10 after run stop. Looking back at the QA histograms, that PMT box had missing entries since run 27 today
  • Needed to reboot L4 to stop run for few runs --- likely from scheduled disc checks on Wednesday, shouldnt happen again
• Evening shift:
  • Laser run: The QA plot of TPC drift velocity is empty ---- from QA plots not updating and needing to reboot jevp servers
  • ITPC: Sector 4, RDO 2 masked out
  • QA plots were not updating; read through the manual for it, and followed the instructions for both the "SHIFT" and "L4" server restart processes
• Overnight shift:
  • Couple of time stgc QA plots showed missing hits, power cycled LV to bring back
  • Jeff adjusted trigger rates to minimize deadtime

03/09/2022
I. Summary of operations:

• MCR did switch from Westinghouse to Siemens MG. Went well. RHIC injection polarization ~61%
• Extended the store from morning till 6.00 pm. Yellow abort kicker issues after the beam dump, fixed and beam back around 3 am
• Luminosity comparable to before, unexpected beam loss around 7.00 am today
• Very smooth shifts. No major issues

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • Persistent FST failure with error code 2. Taken out of runs. Included back in runs after expert changing FST current trip limit and alarm limit from 20 to 30 muA
  • Jeff made rebalancing of trigger rates in production_pp500_2022 ---- Jeff will watch data rates and adjust if there is much deadtime
• Overnight shift:
  • Vernier scan at start of store ---- MCR will be asked to let STAR know in advance of Vernier scan plans/schedule
  • Multiple emcvme1_u5val low alarms, looked at GUI and everything turned white. Called Oleg who fixed the issue

03/08/2022
I. Summary of operations:

• Beams for STAR physics all day without much downtime
• Extended the last store yesterday till today morning to start a new fill ~7.00 am
• A few issues with BTOW, L2 error, but overall smooth runs

II. RHIC Schedule:

• Today:
  • Siemens MG Changeover (behind long store for STAR): Change to Siemens (7-11), Hold store until ready to re-inject (~19:00)
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • Trigger went 100% dead - due to high rates for FCS triggers. Couldn’t start runs due to FCS error "FEE count xx, expected 55”. DOs powercycled FEEs with help from Oleg
  • Got BTOW crate 15 failed to configure a few times at start of run. BTOW so that crate 15 changed to manual configuration by Oleg
  • Akio loaded new FCS tower gains, took test run near end of fill
• Evening shift:
  • iTPC S12:1 masked out
  • Much of BTOW is missing in critical plots. DOs called the expert: take it out of the run and power cycle LV, reapplied HV
  • Stopped the run: BERT unexpectedly says "Physics OFF" but there's still beam in RHIC. Seems a connection issue
• Overnight shift:
  • New missing hits in sTGC (hits/VMM) and sTGC (hits/Fob). Back in the new fill today morning ---- If the power cycling ROB didn't help, please ask the crew to power cycle entire LV. It seems crew power cycled wrong ROB (from Prasanth)
  • Error message mid run 23067012 that BC1 timed out, "L2 stopping run”. On trigger component tree BC1 was red, TCD flashing red. TCD sys-reset and powercycling TCD VME crate didn’t help. Tonko fixed remotely ---- Separate issues from BC1 and TCD. TCD issue from Tonko updating the codes. For BC1 issue, powercycle the BC1 crate
  • iTPC s12-1 unmasked

03/07/2022
I. Summary of operations:

• Got beams for STAR physics all day without much downtime
• Unexpected beam abort two out of three times
• Some issues with TPC, BTOW, ETOW, but nothing major during shifts

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics, Change to Siemens MG behind a long RHIC store

III. Items from shifts:

• Day shift:
  • ETOW and ESMD out for 2 runs due to configuration problem with ETOW crate 2. Back after powercycling of all crates and reconfigure
  • iTPC sector 9, RDO 1 masked after three failures
  • BTOW crate 15 required restart of run couple of times, also twice during evening shift ---- for crate 15, just restart the run
• Evening shift:
  • Several runs had to be stopped due to various iTPC RDOs, not necessarily consecutively: S14:1(x2), S13:2(x3), S12:1(x2), S12:3
  • iTPC RDO S13:2 is masked out
• Overnight shift:
  • iTPC: sector 12, RDO 1 masked
  • FCS started to show this error: FEE count 13, expected 26. This was fixed by Oleg after powercycling FEEs
  • Critical plots for SHIFT not being updated, JEVP server needed to be restarted
  • ETOW: configuration failed, DOs found that crates 5 and 6 were red, out of one run. Rebooted and reconfigured and back in run
  • BTOW: configuration failed -- watch BTOW triggers or restart run. We restarted the run and BTOW trigger were firing at a too high rate. Reconfigured

03/06/2022
I. Summary of operations:

• STAR Physics through yesterday, not much interruptions
• FCS: DEP10:2 failed, was taken off runs from ~8.30 pm. Fixed and back in ~6.00 am

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift:
  • Overall smooth shift, with a power-cycle of an iTPC RDO, and a few ROB's (often #15) in the sTGC every once and a while
  • iTPC sector 3, RDO 3 masked
  • Incorrect number of events shown on RunMarker for several runs. Issue of monitoring programs not updating properly, but files were sent to sdcc fine
• Evening shift:
  • btow crate 23 removed from automatic configuration, it is on manual control now (Oleg)
• Overnight shift:
  • sTGC (hits/VMM) plot shows a line that appeared from run# 23065006 (see shift log)
  • FCS fixed and included back in run. Masked RDOs iTPC sector 3, RDO 3 and iTPC sector 4 and RDO 2) also fixed by Tonko ---- Akio will follow up with Tonko so that there is more than one expert who can be contacted to fix the FCS issue

03/05/2022
I. Summary of operations:

• 56 x 56 bunches low luminosity run for alignment yesterday. STAR magnet was OFF. Got 6 hrs, till 8 pm
• CeC took access for 2 hrs. Back to high luminosity run ~11.30 pm
• Lost beam due to Quench Link Interlock around 3.00 am. MCR just got back a new fill
• Shifts were smooth, no major issues

II. RHIC Schedule:

• Today:
  • STAR Physics
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Overnight shift:
  • Masked TPC: Sector 4, RDO 2 ----- Tonko will look into
• Day shift (Today):
  • At start of run, got Caution messages about BTOW crate 23 (crate failed, config failed). Trigger scaler rates all look okay, except for Jpsi*HTTP which is ~400 Hz (expect 50) ------- Indicates misconfiguration of the crate, not to be ignored. Oleg will put this crate under manual configuration

03/04/2022
I. Summary of operations:

• AGS chiller work not finished yesterday. So was decided to provide fill for STAR and work behind it today
• Got beam around 04.00 am today (lost around 09.30 am), high intensity fill
• Shifts were smooth, no major issues

II. RHIC Schedule:

• Today:
  • No APEX today. 56 x 56 bunches low lumi fill for alignment run. STAR magnet off. 6 hr fill expected
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Overnight shift:
  • sTGC out of one run due to communication loss. DOs fixed the problem by powercycling the crate 1 and bringing back the LV modules. Happened once also on todays Day shift ----- No need to take sTGC out of run, fix and include back in the run
  • Getting messages ‘TPX TDBm: error setting up -- TPX might not work! ’. Been happening for all runs since ~05.30 am. DAQ and online plots for TPC looks ok ------- likely communication issue, need to powercycle TDM board. Tonko will follow up with shift

03/03/2022
I. Summary of operations:

• Low luminosity run yesterday night with vernier scans at beginning and end of fill
• MCR has injector issues, working on fixing those now. STAR is open for access. MCR expects at least 6 hour down time, possibly longer
• Shifts were smooth, no major issues

II. RHIC Schedule:

• Today:
  • STAR Physics (after beam is back)
• Tomorrow:
  • 08:00-14:00 APEX
  • 14:00-16:00 CeC Access
  • 16:00 - STAR Physics

III. Items from shifts:

• Smooth shifts, no major issues

03/02/2022
I. Summary of operations:

• Generally smooth data taking, but some issues with TPC
• Fills are lower luminosity (peak 400 kHz), mostly correlated with blue intensity
• Took runs with modified DX angle and low luminosity run with missteered beam at end of fill last night

II. RHIC Schedule:

• Today:
  • 7:00-11:30 CeC access
  • 12:00-17:00 APEX
  • 17:00 - STAR low lumi. fill
• Tomorrow:
  • STAR Physics

III. Items from shifts:

• Day shift: Got beam around 1.15 pm. Smooth shift
  • Had a problem with TPX[4] died/rebooted, was taken out for one run
• Evening shift: Smooth Physics runs, some TPC issues (19:00-)
  • The itpc20 node died. iTPC rdo S20:4 masked out
  • tpx and itpc were going 100% persistently dead, masked out iTPC: RDO S13:3 ----- Both channels to remain masked out. Tonko looking into it, suspects a hardware issue
• Owl shift: Smooth Physics runs
  • Issue with laser and cathode HV GUI, David restarted VME crate to bring back
  • Lost laser in the middle of the run (~06.00 am) ---- Alexei checked laser today morning, works fine

03/01/2022

Period coordinator change: Zaochen Ye ==> Sooraj Krishnan 

I. RHIC Schedule

• Today: Collision for Physics
  • Local pol test with DX angle (any opportune end of store)
• Tomorrow: 7:00--
  • 7:00-11:30    CeC access
  • 12:00-17:00  APEX
  • 17:00 -          STAR low lumi. fill

  II. Notable items/recap from past 24 hours: Smooth Physics

  III. Items from shifts:

  • Day shift: Smooth Physics runs (ended ~11:15am) + Cosmic runs
    • Masked out iTPC sector 13 RDO 3
    • Alexei restarted Laser PC, fixed laser GUI no-response issue
    • Akio uploaded a new electronic gain file for FCS ECal, Jeff updated associated production ids
  • Evening shift: Smooth Physics runs (19:00-)
    • sTGC lost communication, and both crates had to be rebooted and powercycled
    • BEMC sector 10 had hits missing for half a run. Had to be powercycled
  • Owl shift:       Smooth Physics runs 

  IV. Other items: may request to reduce CeC and APEX time to get more physics runs

  
  02/28/2022

  I. RHIC Schedule

  • Today: Collision for Physics
    • will have a local pol test with DX angle 14:00, nominal end of store 14:25
  • Tomorrow: Collision for Physics

  II. Notable items/recap from past 24 hours: Cosmics + Smooth Physics

  III. Items from shifts:

  • Day shift: (first a few hours) Cosmic runs + (11:15) Smooth Physics runs
    • many auto-recoveries in TPX, iTPC, TOF
    • sTGC error "ROB #01 bad FEB.”, powercycle ROB #1
    • FST 100% dead, no errors in scrolling DAQ messages. Power cycled FST.
  • Evening shift: Smooth Physics runs when beam available (no beam: 19:00-21:00)
    • TPC Anode Trip, GMT HV needed powercycling
  • Owl shift:       Smooth Physics runs 
    • missing bins in EEMC Tower ADC plots, expert power cycled crate 92, and performed master reload
    • laser control GUI (where the camera is controlled) froze, Alexei will investigate during day shift
    • ITPC: Sector 23, RDO 4 -- powercycle then restart run (several times)

  Other discussions: request low lumi runs (Wed or Thu ?) 
  

  02/27/2022

  I. RHIC Schedule

  • Today: Collision for Physics till Monday morning, exchange the cathodes (~ 2 hours) 

  II. Notable items/recap from past 24 hours: Cosmics + Physics + Cosmics

  III. Items from shifts:

  • Day shift:  Cosmic runs + 3 Physics runs
    • node TPX[19] removed / masked from run configuration
  • Evening shift: 13 Physics runs
    • node TPX[19] removed / masked from run configuration
  • Owl shift:       5 Physics runs + Cosmic runs
    • TPX: RDO S01:6, TPX: RDO S01:3, ITPC: Sector 1, RDO 4, ITPC: Sector 10, RDO 3 
    • partial trip of BTOW PMT 10
    • Beam lost ~2:40 (failure of a Yellow Abort Kicker, requires a Thyratron replacement) 

  Other discussions: CAD should do a better job to deliver beam  

  02/26/2022

  I. RHIC Schedule

  • Today: Collision for Physics till Monday morning, exchange the cathodes (~ 2 hours) 

  II. Notable items/recap from past 24 hours: Cosmic runs

  III. Items from shifts:

  • Day shift:  Cosmic runs
    • Alexei changed a burned mirror for East laser (camera 3), more laser tracks on East part of TPC
    • 12:37 Tonko fixed a bug in the ROB firmware which caused odd behavior under sparse data taking conditions (cosmics, low lumi runs). The new code version is 0x126 (old, up to now, was 0x124)
  • Evening shift: Cosmic runs
  • Owl shift:       Cosmic runs
    • sTGC air flow alarm went off, slow control issue
    • Tonko: “Now all STGC RDOs have the newest firmware version (0x126).” 4:40am

  
  02/25/2022

  I. RHIC Schedule

  • Today: 7:30 access to exchange garages, start bake load-lock chamber (~ 8 hours)
  • Collision for Physics till Monday morning, exchange the cathodes (~ 2 hours) 

  II. Notable items/recap from past 24 hours: Access + Low lumi + Smooth Physics

  III. Items from shifts:

  • Day shift:  Smooth physics till 13:30 + CeC Access
    • A few powercycles of iTPC RDO, sTGC ROB, BTOF, DAQ
  • Evening shift: Low lumi runs till 21:13 + normal physics runs started 23:59
    • testVernierScaler, forwardCrossSection, zdc_Polarimetry_2022
    • trigger was 100% dead. sTGC plots looked abnormal, and had missing tiles, stgc_n_sigma reset to 12 and 8, taking a production run to check what the issue is with sTGC, still open issue (days to solve) will request another low lumi runs after figure out sTGC issue
    • MXQ crate had to be powercycled
  • Owl shift:       Smooth Physics runs till 5:30 am (beam lost)
    • False magnet trip alarm, apparently due to communication or slow control glitch
    • BEMC PMT box 10 west partially tripped, solved by reapplied HV

  Others: if plan to make use of the access time, please bring up

  02/24/2022

  I. RHIC Schedule

  • Today: 14:00-24:00 STAR Low lumi. Fill with Field on (Forward experts should watch the online plots to make sure detectors are running good)
  • Friday: Aperture scan and crossing angle for local pol. measurement

  II. Notable items/recap from past 24 hours: Access + Smooth Physics

  III. Items from shifts:

  • Day shift: Access
  • Evening shift: Smooth Physics started from 18:33
  • Owl shift:       Smooth Physics
    • FCS: persisting errors “FEE count 44, expected 5”. FCS GUI no responding, Oleg power cycled FEEs manually cleared error message, however the GUI is still frozen, solved by restart IOCs
    • iTPC node 13 died at end of the evening fill, back online after powercycle corresponding computer

  
  02/23/2022

  I. RHIC Schedule

  • Wed: 7:00-13:00 Maintenance
  • Thu: 16:00-24:00 STAR Low lumi. Fill with Field on 

  II. Notable items/recap from past 24 hours: Smooth Physics

  III. Items from shifts:

  • Day shift: CeC access + Smooth physics + AGS Snake issue
    • TPC gating grid lost communication, solved by reboot software
  • Evening shift: Smooth Physics + a few issues
    • Dimuon triggers were not included in time
    • "Incorrect Trigger Clock 76 MHz" error due to stuck in PLL. Had to powercycle TCD and turn MXQ on
    • FCS PS channel U302 got stuck. Power cycling MPOD crate recovered that channel
  • Owl shift:       Smooth Physics

  IV. Access today:
  

  • Tonko: DAQ tests with TPX and iTPC, don't need access but no runs with Run Control, finished 10:15.
  • MTD group: 2.5 hrs with field cosmics after Tonko’s test. MTD HV 13.2 kV to check eff. Vs. 13 kV
  • Wayne:
    • replace a disk in the computer named deneb, on the 2nd floor of the South Platform
    • apply some routine software and OS updates to the shift-leader PC in the Control Room

   

  02/22/2022
  I. RHIC Schedule

  • Today: 10:00-12:00 CeC access, after will be Collision for physics
  • Wed: 8:00-16:00 Maintenance
  • Thu: 16:00-24:00 STAR Low lumi. Fill with Field on 

  II. Notable items/recap from past 24 hours: Smooth Physics

  III. Items from shifts:

  • Day shift: Smooth physics
    • BTOW: PMT box 10 west partially tripped. Solved by reapply HV
  • Evening shift: Smooth Physics 
    • MTD HV BL 24-30 HV cannot be set to full from standby at first try, second try succeed
    • ETOW crate Id 5 configuration failure, solved with expert’s help
  • Owl shift:       Smooth Physics 
    • BTOW: PMT box 10 west partially tripped. Solved by reapply HV
    • sTGC  error: 1: FOB 4 (R#13:F#77) NOT found, power-cycle ROB #13

   

  02/21/2022
  I. RHIC Schedule

  • Today-Tuesday morning: Collision for physics

  II. Notable items/recap from past 24 hours: Smooth Physics + a few issues

  III. Items from shifts:

  • Day shift: Smooth physics
    • GMT -> APV -> pedestals_APV_0-3 pulse height for all channels in ARM:0, APV0 shows a second band and scattered values all throughout
    • BTOW Crate Id 1: Oleg put under manual control -> Watch trigger rates and reconfigure if needed
  • Evening shift: Smooth Physics 
    • Global communication problem, fixed by David
    • "too many recoveries - powercycle TOF LV”, Tray 67 (dead, masked out) was accidentally enabled by DO, which caused the failure of auto recovery. Zaochen updated instruction to reduce the mistake
  • Owl shift:       Physics run + issues
    • DAQ issue: EVB[3] [0xF103] died/rebooted) EVB3 is suggested to be out of run for now. writing data issue (No space left on device)
    • error " Incorrect Trigger clock 10.000 MHz”,  solved by powercycle TCD and turn on MXQ from off. Tonko:”One TCD board was stuck and couldn’t lock its PLL, it happens after too many Trigger reboots”
    • high rates of JetPatch triggers, solved by reconfigure BEMC. Related to TCD powercycle.
    • 6:17am, TPC Gating-grid monitor lost connection, need access to powercycle hardware (8:50am, Tim cycled crate 50, gating grid control is back now)
    • IOC for sTGV LV lost connection, solved by David
    • Laser brightness has no response to control, drift velocity looks normal. Alexei fixed this. 
      
      

  02/20/2022
  I. RHIC Schedule

  • Today-Tuesday morning: Collision for physics

  II. Notable items/recap from past 24 hours: Smooth Physics
  Main discussions: Carl: "correct" bunch crossing has barely more hits than any of the visible out of time bunches. In this environment, it’s impossible to see if 2-5% of the triggers are late by a RHIC tick.  We’re going to need a 28 or 56 bunch fill to answer this question. ZDC_Polarimetry runs can go back to “TRG + DAQ only”, Will try “TRG + DAQ + FCS” on sometime in middle of next week (Wed-Fri, decide in schedule meeting?)

  III. More items from shifts:

  • Day shift: Smooth physics
    • FST plots were empty during run, but filled after run stops.
    • JP triggers hot, solved by restart run
    • Run stopped by EQ crate, solved by power cycle and reboot trigger then restarted run
  • Evening shift: Smooth Physics 
    • PMT box 10 west partially tripped, solved by reapplied HV
    • from run 23050064: "dimuon" and "hit_dimuon_mon" are on Enabled list
  • Owl shift:         Smooth physics run
    • BTOW Configuration failed for crate 1 (will mask out from auto configuration, do similar thing as crate 8)

  
  02/19/2022
  I. RHIC Schedule

  • Today-Tuesday morning: Collision for physics

  II. Notable items/recap from past 24 hours: Smooth Physics when beam available 

  • GMT HV alarm went off, HV has no response to “Physics” or “Standby” mode in GUI, Nikolai fixed the GMT. It had lost connection. Fixed by rebooting the crate and restarting the controls

  III. More items from shifts:

  • Day shift: Physics started 10 am
    

     

    • sTGC air blower, solved and discussed yesterday 10 am
    • Alexei fixed TPC air blower 
  • Evening shift: Smooth Physics 
  • Owl shifts: Smooth Physics run (till 5:30, injection issue, solved 7am)
    • FCS FEE LV caused missing hits in FCS ECal's North 18~24 and high rates of FCS ECAL triggers. Resolved by powercycling FCS FEE LV
    • BTOW Configuration failed for crate 1 (23050011, 23050013) and L2 issues (23050016)
    • iTPC errors (S2, RDO 1; S16 RDO 1). Tonko fixed

  IV. other items:

  • zdc_polarimetry runs please do with (TRG+DQA+FCS)

   

   

  02/18/2022
  I. RHIC Schedule

  • Today-Tuesday morning: Collision for physics

  II. Notable items/recap from past 24 hours: Snake scan + Physics + IssueMain issues: 

  • TPC gas alarm at 03:35 and called Prashanth and Alexei.
  • 4:20 there was a power dip, which caused magnet trip, problems with water system and sTGC blower trip.
  • Alexei and Prashanth fixed the TPC gas and water issues, CAS brought back the magnet
  • 7:10 Tim and Mike access to fix issue with sTGC air blower (6:00, MCR: a spark in ion source). Compressor not running, after replaced compressor contractor, Compressor is now running. Replaced LV relay for preventative maintenance. Mike connected aux contacts from compressor relay to monitor contractor state. Tim replaced sTGC AHU Contactor, Capotosto added remote monitoring for contactor state to sTGC interlock cabinet. Just fixed all issues, sTGC can be added in the run now.
  •  GMT HV alarm went off, HV has no response to “Physics” or “Standby” mode in GUI, issue is still open

  III. More items from shifts:

  • Day shift: snake scan for spin direction
    

     

    • BTOW is flashing blue (W) for >30 minutes
  • Evening shift: Smooth Physics while beam is available
    • Beam came at 19:35, beam lost at 21:12, beam back at 23:20
  • Owl Shift: Smooth physics run
    • Trigger: BBQ and EQ2 VME crates tripped, then L2 was stopping the run. Powercycling BBC and EQ2 crate and rebooting the trigger solved the problem.
    • Online plots issues of TPC, MTD and ESMD. 

  IV. other items:

  • Request for 2 polarimetry runs at the end the fill

   

  02/17/2022
  I. RHIC Schedule

  • Today:       8:00-14:00 Snake scan for spin direction
  • Tomorrow: Collision for physics

  II. Notable items/recap from past 24 hours: General: APEX + Smooth physics

  • Main issues: A mistake to assign channel 1 of the 2nd board to channel 1 of the 1st board. So the channel was not turned on when controlled by the GUI. David fixed it and HV is back on that sTGC chamber. Online plots back to normal. 

  III. More items from shifts:

  • Day shift: APEX
    

     

    • Alexei replaced GMT tank-1
    • David found HV channel was mis-mapped, online plots shows normal
  • Evening shift: Smooth Physics started 21:40
    • JP1 was high due to hot trigger patches on crates 0E, 0D, 05, solved by power cycle. However, crates 0C, 07, 0F was not get configurated at start of next run. Oleg Tsai helped manually configure these crates.
  • Owl Shift: Smooth physics run
    • high JetPatch trigger rates, Oleg manually configured BTOW crate 1E. Restarted run, JP rates are normal.
    • Twice problems when stopping run (EVB[5] [0xF105] died/rebooted; ITPC[23] [0xBF97] died/rebooted)

  IV. other items:

  • Remind shiftcrew to check more frequently on the critical plots

  02/16/2022
  I. RHIC Schedule

  • Today:        9:00-19:00 APEX (Au, 3.85 GeV/n)
  • Tomorrow: 8:00-19:30 Siemens test and Rotator scan?

  II. Notable items/recap from past 24 hours: General: Access + Smooth physics (Owl)

  • Main issues: David, Prashant, Mike, Tim installed sTGC HV boards to MTD CAEN HV crate, rerouted and reconnected HV and interlock cables. However, sTGC still looks strange in test pedestals and cosmic runs. Experts will check on RDO and others

  III. More items from shifts:

  • Day shift: Access
    • David, Prashant, Mike, Tim installed sTGC HV boards to MTD CAEN HV crate, rerouted and reconnected HV and interlock cables. However, sTGC still looks strange in test pedestals and cosmic runs
    • Jeff cleaned up and changed some cables for iTPC DAQ machines
  • Evening shift: Physics run started 23:50
  • Owl shift: smooth physics run
    • 2B quadrant was empty in sTGC (hits/Quadrant) online plots.
    • Masked ITPC: Sector 11, RDO 4

  IV. Other items?

   

  02/15/2022
  I. RHIC Schedule

  • Today:        7:00-15:30 maintenance, then collisions for physics
  • Tomorrow:  9:00-19:00 APEX (Au, 3.85 GeV/n)

  II. Notable items/recap from past 24 hours: General: Access + Smooth physics run

  • Main issues:
    • sTGC HV GUI lost connection, David and Tim checked sTGC CAEN HV crate, no solution, run without sTGC
    • Partial trip of BTOW PMT box 10 West, solved by reapplying HV to this PMT box

  III. More items from shifts:

  • Day shift: Access
    • sTGC bad chamber was checked by Prashanth and Tim, will order parts and replace later
    • FST coolant refilled by Xu and Yu
    • Wayne fixed issue with BBC_DSM2 and BCW_DSM2, if same problem happens, call trigger experts first, may call Wayne later.
  • Evening shift: Access + Smooth physics run
    • sTGC HV GUI lost connection, David and Tim checked sTGC CAEN HV crate, no solution, run without sTGC 
  • Owl shift: smooth physics run
    • no sTGC in the run

  IV. Other items?

  02/14/2022
  I. RHIC Schedule

  • Today:      7:00-15:30 maintenance, then collisions for physics
  • Tomorrow:  7:00-15:30 maintenance, then collisions for physics

  II. Notable items/recap from past 24 hours:

  • General: Smooth physics run 
  • Main issues:
    • Error configuring BBC_DSM2 Node and Error configuring BCW_DSM2 Node. Reboot all did not help. Powercycling the corresponding VME crates and reboot all did not help either. Had to take out BBC_DSM2 and BCW_DSM2 from the Component tree (experts are working on it)
    • Tonko masked Crate Id 8 from _any_ configuration, Oleg shows how to manually reconfigure BTOW crate 8, will need to keep watching the performance

  III. More items from shifts:

  • Day shift: smooth physics run
    • BTOW crate 8 config. Err., Tonko: disable à rates too high à keep old procedure
    • Online QA plots lost, solved by reset jevp server
    • Lost connection of sTGC LV, solved by David T.
  • Evening shift: smooth physics run
    • iTPC: Sector 4, RDO 3 powercycled
    • sTGC LV errors, solved by power-cycle
    • BTOW: PMT box 10 west partially tripped, solved by reapply HV 
  • Owl shift: smooth physics run
    • Error configuring BBC_DSM2 Node and Error configuring BCW_DSM2 Node, moved out from the Component tree

  IV. Other items?

  • Access plan on Monday-Tuesday (8 hrs each day)
    • Magnet maintenance: Water Supply Group will be cleaning the strainers in the STAR magnet power supplies during the maintenance on Monday.  This involves de-energizing STAR magnets.
    • FST: Refill the coolant for FST, need about 30 mins for the fill then 20~30 mins cosmic runs to verify
    • others?

   

  02/13/2022
  I. RHIC Schedule

  • physics run

  II. Notable items/recap from past 24 hours:

  • General: Smooth physics run 
  • Main issues:
    • DO had difficulty to reboot CANBUS to solve BTOF LV errors, due to the mistake in password list
    • Several EVB errors
    • BTOW configuration errors happen many times from crate 8 (Oleg will provide solutions for the shifts to help reduce the down time for the data taking)

  III. More items from shifts:

  • Day shift: smooth physics run
    • BTOW configuration issues solved by restarting run while waste a lot of beam time (experts will provide better solution)
    • 2: FOB 2 (R#14:F#81) NOT found -- power-cycle ROB #14
    • 2: FOB 2 (R#14:F#81) NOT found -- power-cycle ROB #14
    •  
  • Evening shift: smooth physics run
    • Power-cycle BTOF LV
    • sTGC "3: FOB 0 (R#07:F#37) NOT found" errors, and we power-cycled ROB #07
    • sTGC "4: FOB 3 (R#16:F#94) NOT found" errors, and we power-cycled ROB #16
    • "EVB[3] [0xF103] died/rebooted"
    • "BTOW: failed crate Ids: 1; BTOW: configuration failed - watch BTOW triggers or restart run."
  • Owl shift: smooth physics run
    • l4 got stuck at the end of several runs, “EVB[3] [0xF103] died/rebooted”, “EVB[4] [0xF104] died/rebooted
    • BTOF LV errors, get difficulty to reboot the CANBUS due to the mistake in password list
    • a few BTOW Configuration failed for crate Id 8 and once for crate Id 1
    • iTPC RDO S20:1 was masked

  Other items?
  

  • Access plan on Monday-Tuesday (8 hrs each day)
    • Magnet maintenance: Water Supply Group will be cleaning the strainers in the STAR magnet power supplies during the maintenance on Monday.  This involves de-energizing STAR magnets.
    • FST: Refill the coolant for FST, need about 30 mins for the fill then 20~30 mins cosmic runs to verify
    • others?

   

  02/12/2022
  I. RHIC Schedule

  • physics run

  II. Notable items/recap from past 24 hours:

  • General: smooth physics run 
  • Main issues:
    • Lasers had no respond to controls well. Brightness does not change when pressing control button. Called Alexei.
    • East laser was fine for two laser runs, West laser was far from normal brightness

  III. More items from shifts:

  • Day shift: physics run, short access, physics run
    • Issue with L2, solved by reboot trigger
    • Jeff fixed the TOF+MTD noise run rate, runs ~14 kHz now.
    • 13:00-14:30: David and Prashanth refilled the sTGC n-Pentane
  • Evening shift: smooth physics run
    • Power-cycle iTPC sector 14, RDO 3 (twice)
    • BTOW configuration failed, restarted the run
    • "tpc_caen_alive" alarm went off, fixed by following manual
    •  sTGC “3: FOB 2 (R#11:F63) NOT found”, fixed by power-cycle sTGC LV ROB #11
  • Owl shift: smooth physics run
    • iTPC RDO S14:3 and S16:3 were masked out, (had error in 2 consequent runs
    • FCS error in DAQ, solved by restarted the run twice
    •  sTGC LV: 2: FOB 3 (R#10:F#58) NOT found -- power-cycle ROB #10 if the problem persists.

  Other items?

  • Lumi was a little too high than usual during Owl 
  • sTGC cable(chamber) 9
  • Access plan for Monday-Tuesday

  02/11/2022
  I. RHIC Schedule

  • Plan from RHIC
    • Collisions for physics now to Sunday

   

  II. Notable items/recap from past 24 hours:

  • General: APEX and smooth run
  • Main issues:
    • - sTGC: HV cable 9 trip

  III. More items from shifts:

  • Day shift: physics run at the beginning, then APEX
    • -       10:00, Power dip, magnet tripped, lost slow control. Experts solved the problem.
  • Evening shift: APEX ended around 22:00, Physics from 22:30
    • -        sTGC HV cable 9 tripped, ran without this channel
  • Owl shift: very smooth physics run
    • -       TPX 36 config error, solved by powercyle TPC FEEs and Reboot All (Comments: Reboot All is not necessary but no any harm)

  IV. Other items?

  -

  
  02/10/2022
  I. RHIC Schedule

  • Plan from RHIC
    • 8:00-16:00: LEReC APEX: 3:85 GeV/n Au
    • Today 16:00-Tomorrow 16:00: Collision for physics

  II. Recap

  • General: Rotator study, low lumi run (18:00-4:15am), 7:00 am normal physics run, smooth
  • Main issues:
    • sTGC: LV unresponsive, solved by short access (18:55). 
    • sTGC HV cable 9 trip, experts will discuss for a solution today
    • Timing during alignment fill, “extra” peak in preshower, ECal and HCal 8 time bins before “right” peak… (Carl Gagliardi), experts will follow up later.

  III. More items from shifts

  •  Day shift: Rotator study
    • 12:43: Wayne replaced the monitor for EPD-CR
  •  Evening shift: low lumi run
    • sTGC: LV1 is unresponsive, after a short access, run smoothly for the rest of the shift.
  • Owl shift: low lumi run till 4:15am, then normal physics 7:00am
    • sTGC HV Cable 9 trip, still open
    • BTOW configuration cautions, Rebooting not help but rates and critical plots are good

  IV. Others items?

  • Battery for the clock replaced
  • fast offline data production request for forward detector calibrations with the low lumi runs.

  02/09/22
  

     I. RHIC Schedule

  • Today's plan:
    • 08:00 - 14:00: Rotator study
    • 14:00 - 23:59: STAR low luminosity run (for alignment; magnet off, 0.2e11/bunch, 8 hr, 56x56)
    • 00:01 -8:00:    collision forphysics 
  • Tomorrow’s plan:
    • 8:00-16:00: LEReC APEX: 3.85 GeV/n Au

     II. Recap

  • Main issue in past 24 hours:
    • 14:17: EEMC (ETOW, BTOW) config errors, “GUI is white” seen from SC3 but not seen in control room. (15:12) Dmitry K "Recovery & Restart", turned HV and MAPMT FEEs back on 
    • sTGC cable 9 HV trip, David fixed it, after 2 hrs run, tripped again, David T. checked on it (see his email). HV current frequent spikes even at a lower voltage may due to the dust in chamber. If this happen again, don’t need to take out sTGC from data taking. Just make a note which channel/cable tripped, i.e. is not at full voltage in the run. Tripped again during OWL, under David T’s control
    • 23:44, EPD GUI is saying that the current is too high for most channels, not affect data taking. Rosi pointed out that current values were fine, a problem with GUI, she will follow up with EPD maillist.
  • Open issues:
    • sTGC cable 9 HV trip, David T. will work on it this after noon, during the low lumi run.
  • Day shift: smooth run from the beginning to the end
    • Jeff fixed the issue effecting the sTGC critical plots (y-axis range).
    • Issues with BTOW and ETOW, Dmitry K worked on it.
    • EVB12 was out for a while in a run,  back in next run
    • sTGC LV issue, fixed by David.
    • sTGC cable 9 HV trip. After clearing trip and ramping up the channel tripped again. David advised a temporary solution of manually setting demand voltage to 2K until he fixes it. David fixed the issue. 16:18, same issue came again. 
  • Evening shift:
    • "GUI is white" when trying to access slow controls from SC3. Followed manual section "Recovery & Restart", turned HV and MAPMT FEEs back on – Dmitry Kalinkin
    • EPD GUI is saying that the current is too high for most channels, however not affect data taking. Rosi said current values were fine, a problem with GUI, she will follow up with EPD maillist. 
    • TOF critical errors around 8pm, POWERCYCLE TOF LV. Tried to power cycle at least twice, no luck. Geary instructed us to try one more time as the problematic trays are different.
    • Display for EPD computer went off. Michael will work on it in the morning
  • Overnight shift: Beam and physics for all time
    • Major issues with the sTGC Cable 9 HV supply.
    • sTGC cable 9 HV trip, slowly increasing HV by +10V or +5V, reached full, then tripped again, David worked on it. 
    • BTOW configuration cautions at the start of each run. Rebooting BTOW in Run cotrol or restarting run did not help. Rates and critical plots look OK, keep running.
    • TOF ADC critical plot: One run (23040009) had ADC entries at ADC = 15 and ADC = 30 for all channels. Restarting the run helped.
    • Relatively many TOF auto-recoveries. Apart from the run mentioned above, no indication of problems in the critical plots.
    • Had to powercycle a couple of iTPC RDOs manually which helped.

  02/08/22
  

     I. RHIC Schedule

  • Collisions for physics today
  • Tomorrow’s plan:
    08:00 - 14:00 Rotator study
    14:00 - 22:00 STAR low luminosity run (for alignment; magnet off)

    II. Recap

  • Machine development all day yesterday, beam back later in evening shift
  • EEMC QA plots having issues during day shift (first note @ 07:54), needed short access after MD (before physics) and replace HVSYS module A
  • In overnight shift: ETOW error messages in DAQ for crate 6 -> No success in fixing issue overnight; today’s morning shift advised to power cycle the crate and then issue a master reset; noted: this is the procedure, no need to contact on-call expert for this 
  • Period coordinator shift changeover Matt -> Zaochen

    III. Open issues/status

  •  

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006606.html)

  02/07/22

     I. RHIC Schedule

  • Machine development unitl around 17:00, back to physics after

    II. Recap

  • In terms of sampled luminosity, yesterday was best day so far: 6.1 pb-1 w/ 18.5 DAQ hrs

    III. Open issues/status

  •  Discussed prospects of dedicated aligment/forward cross section runs. Potential to do alignment run Wed and cross section run Fri. Will be discussed during sheduling meeting, etc.

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006594.html)

  02/05/22
     I. RHIC Schedule

  • Collisions for physics all weekend

    II. Recap

  • No beam 6:45 until around ~17:00
  • sTGC HV slowly ramped up during shifts; new temporary ramping procedure for cable 9 (see overnight shift log and https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006575.html)
  • Warning from TPC computer (chaplin): running low on GPU memory, usage at 99% (18:01)
  • Two TOF FM2 red alarms in overnight shift. FM2 plot shows sudden spikes (04:47)

    III. Open issues/status

  • Discussed on making the new temporary sTGC HV ramp the same for entire sTGC (i.e., automated)
  • BTOW crate 8 error - Noted shifts should just stop and restart run
  • Note on various reported high rates in logs, at beginning of fill this is expected, Jeff may change the baseline rate used to determine color-coding in monitor
  • Need to follow up with Wayne on TPC computer (chaplin) GPU memory usage warning
  • Alexie following up on current laser issue and TOF FM2 alarm

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006577.html)

  02/04/22
     I. RHIC Schedule

  • MCR access until 12:00, about 2 hrs to get back to physics

    II. Recap

  • Physics running started around 22:50 due to long injection tuning
  • Lots of errors overnight (see below), however no major downtime
  • sTGC HV trip for plane 2 cable 9. Powercycled the HV - Investigation still ongoing
  • sTGC LV (23:31) powercycled after getting following error: “ROB-15: FOB 1 (R#15:F#86) NOT found -- power-cycle ROB #15 if the problem persists” - This is also a correction note w.r.t. minutes sent durectly to star-ops

    III. Open issues/status

  •  Tentative plan to do rotator measurements/corrections Monday during day
  • When beam comes back today, add BBC back into run for testing
  • Investigation of sTGC HV ongoing.

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006569.html)

  02/03/22
     I. RHIC Schedule

  • APEX 9:00 - 17:00. Followed by physics.
  • Tomorrow: CeC access ~7:00 for 3 or 4 hrs; TBD if machine development with follow, or back to physics

    II. Recap

  • All contingent activities for APEX running were solved, so APEX will run today ~8hrs (started at 9)
  • Tomorrow’s afternoon activities still TBD
  • Identified incorrect terminating resistor on injection kicker (40 -> 25 ohm); will fix during tomorrow’s access, should improve performance
  • TOF HV turned off during evening shift, values frozen in GUI indicated “on” (fixed with "IOC reset FULL” and ramping HV)
     

    III. Open issues/status

  •  

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006560.html)

   

  02/02/22
     I. RHIC Schedule

  • 07:00 to 11:00: RHIC access -- CeC clean room, Booster access for foil, back to physics
  • Thur: 09:00-17:00: APEX (if ready). Followed by physics.
  • Fri: 07:00-12:00(?): RHIC access injection kicker module replacement + CeC access; imperfection study after(TBD)

    II. Recap

  • Two (last minute) Vernier scans in day and evening shifts - Will update crew checklist to note what to do if this happens again
  • Booster RF power amplifier leak, extended evening shift fill, no beam after 2:55
  • Major issue with DAQ starting @ 18:20. Jeff stepped in and was able to eventually pinpoint the problem. Noted computer disc may need to be changed - Done by Tonko
  • Alarm for VME crates, lost communication and QA plots. David worked on this for awhile. Decided access is needed (23:43). Accessed today at 9:30 - eveything should be OK now
     

    III. Open issues/status

  •  

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006549.html)

  02/01/22
     I. RHIC Schedule

  • Collisions for physics
  • Tomorrow: likely APEX from 9:00 - 17:00

    II. Recap

  • No beam from ~7:00 -~19:30 yesterday
  • Smooth running once beam returned
  • TOF gas changes twice in past 24 hrs
  • JH will bring up request for low intensity run for forward x-sec and alignment runs @ RHIC coordination meeting; alignment run will need magnet off, warmer ambient temperatures should be considered when doing this
  • David will increase current alarm for sTGC from 200 micro amps to 300
     

    III. Open issues/status

  • ETOF HV proceedure has been changed, shifters should change to OFF for all states
  • APEX runing for tomorrow contingent on a few ongoing RHIC/AGS activities. If postponed, tomorrow will be physics, and APEX will run Thursday

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006531.html)

  01/31/22
     I. RHIC Schedule

  • Collisions for physics

    II. Recap

  • Luminosity highest in recent fills; improvement in RHIC polarization (~50%)
  • Issues at linac, unclear when will be fixed
  • Afternoon access requests: CeC 4 hrs, Jet 1/2 hr; may be earlier in morning  -> Will hear back from MCR/Vincent et al. shortly on schedule
  • Relatively smooth running over past 24 hours
  • Sampled luminosity: 5.1 pb-1 w/ 15.5 DAQ hrs; today 1.5 pb-1
  • Forward x-sec strawman trigger config test with low lumi in run 23030047 - will request fastoffline production

    III. Open issues/status

  • Day shift will take a laser run ASAP to see if they can get West laser working; if problems Alexie may go in to make some adjustments
  • David will make changes to alarm handler (see FCS HCal PS issue in elog in overnight shift)

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006515.html)

  01/30/22
     I. RHIC Schedule

  • Collisions for physics

    II. Recap

  • Very smooth running over past 24 hours
  • Sampled luminosity (note with updated x-sec): 4.6 pb-1 w/ 16.5 DAQ hrs; today 2.2 pb-1
  • Forward x-sec strawman trigger config test in Run 23029038; todays test at end of fill should aim to have MCR displace beams to achieve ZDC rate between 10-20k (~10 min); request offline to produce runs ASAP

    III. Open issues/status

  • Will test forward x-sec strawman trigger config today with low lumi beams at end of next fill

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006495.html)

  01/29/22
     I. RHIC Schedule

  • Collisions for phyics over weekend

    II. Recap

  • Weather conditions very bad due to winter storm. So far no major issues reported
  • Sample luminosity: yesterday: ~4 pb-1 w/ 17 DAQ hrs; today: ~1 pb-1
  • Updated Vernier scan x-sec: 2.06 mb, all plots will be updated with new value
  • Very high JP1, JP2, ETH1, … rates noted in Run 23028049, had to power cycle and reconfigure EEMC (with experts), also next shift noticed high JP1 rates in one run
  • Ground fault of substation 6A @ 7:22 (another at 10 AM), addressed promptly by CAS watch
  • Forward x-section measurement strawman trigger configuration provided by Jeff, tested in Run 23028062 (in between fills). Test today @ end of fill - 5 min needed (note: another test tomorrow with beam steering for low lumi.)

    III. Open issues/status

  • Forward x-section measurement strawman trigger configuration will be tested today at end of fill. Should also be tested tomorrow with a lower luminosity
  • Shuai: will update dimuon timing window trigger configuration

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006485.html)

  01/28/22
     I. RHIC Schedule

  • Collisions for physics over weekend

    II. Recap

  • ZDC polarimetry done during rotator test
  • FCS gain tests after rotator test (some thresholds changed after)
  • Smooth running when beam came back during overnight shift
  • sTGC settings changed in evening shift, some “spikes” seen in QA plots (02:33)
  • New PROM check run added by Tonko to be done before; will update detector readiness checklist
  • Vernier scan was performed, preliminary cross section is lower than Run17
  • Carl proposed we take out dimuon trigger for first 1.5 hrs of a fill, and agreement was reached. This has to be done manually by shifters. Jeff will conveniently move where the SL can do this on the computer to be right next to laser run trigger.
     

    III. Open issues/status

  • We should start planning the low luminosity runs (as early as next week). Jeff is working on the configuration, once ready we will ask MCR to steer the beams to lower rates and take a run with this configuration at the end of a fill

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006475.html)

   

  01/27/22
     I. RHIC Schedule

  • CeC will take 4 hrs of dedicated time
  • 1/2 hr coupling test
  • Rotator test after CeC/coupling test (STAR will do zdc polarimetry runs as we did last rotator scan, one before and after rotator ramp, we will also do a FCS gain check at the end of store for ~30 min.)
  • Back to physics

    II. Recap

  • Blue injection efficiency is getting worse every fill, reducing luminosit
  • AGS polarization also seems to also be decreasing
  • Smooth running after maintenance, albeit with lower intensity
  • Various maintenance day activities compled yesterday, see elog for details
  •  

    III. Open issues/status

  • Jeff/Tonko: doing a few checks today, but may begin to have a daily PROM check run
  • Daniel: sTGC will have SL update configurations for pedestals
  • Hank: still investigating BBC crate issue
  • Xu: Current in problematic FST module seems to be stabilizing. Will keep an eye on for next 24 hrs, and may start to slowly ramp HV back up
  • David: Now has updated manuals ready for sTGC, also compressor monitoring 

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006456.html)

  01/26/22
     I. RHIC Schedule

  • Maintenance from 7 AM until around 3 (likely till 5 or so) PM, back to collisions for physics
  • Tomorrow: 4 hrs for CeC starting after natural end of a fill, some time between 9-12

    II. Recap

  • 3 good fills yesterday, one fill this morning with reduced luminosity
  • Yesterday: sampled lumi 4.1 pb-1 w/ 17 DAQ hrs; Today: 1.2 pb-1
  • Crew enable BBC readout on evening shift as requested by Hank
  • Overnight L2 timeout error was from MXQ, not BBC. Will work on making this more explicit for shifters
  • New items during maintenance time:
    Hank/Jeff, will need control system to test new tier 1 file, will convene around 15:00 to start
    Akio, work on EPD

    III. Open issues/status

  • Manual for sTGC will be updated with current procedures
  • Explicity logging for L2 timeouts to aid shifte

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006438.html)

  
  01/25/22
     I. RHIC Schedule

  • Collisions for physics today
  • Tomorrow maintenance from 7 AM until around 3 PM, likely to extend longer

    II. Recap

  • Consistent 2+1 ongoing fills since last meeting; Yesterday: 4 pb-1 w/ 15.5 DAQ hrs (best day for int. lumi/hr); today: ~1.5 pb-1
  • FCS taken out of run during day shift, put back in during overnight shift
  • Unusual fill pattern for sTGC shift crew plots, Jeff and Daniel B we notified, not immediately obvious is anything is wrong but Jeff will followup, and Daniel may change plotting scales
  • One more (noted) online plot crash during evening shift
  • Several activities planned for tomorrow's maintenance day. See detaIls in star-ops log linked below
  • Daniel requested a special ~30 min run for sTGC at end of fill at 3:25 PM today, will coordinate with PC & SL

    III. Open issues/status

  • FST still tripping, experts changing trip limits and HV, expect to be OK after tomorrow's maintenance
  • Manual for sTGC will be updated with current procedures

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006406.html)
  
  01/24/22

     I. RHIC Schedule

  • Collisions for physics
  • Maintenance schedule for Wed. (8 hrs?, depends on yellow rotators TBD)

     II. Recap

  • Consistent stores in terms of luminosity (slight improvement w.r.t. yesterday’s report); two beam dumps with ~3+~6 hr downtimes
  • Yesterday: 3.2 pb-1 & 13 DAQ hrs; today: ~1.2 pb-1
  • Some continuing FST HV trips in evening shift; Xu adjusted limits and HV again
  • Fire department had to check a (false) smoke alarm in the cave at 22:30 (during downtime so no interruptions to data-taking)
  • EEMC, TPC/iTPC out for a few runs during overnight shift; in both cases experts were consulted and eventually came back online (fixed with: rebooted crate 90, “reboot all” in run control, respectively)
  • GMT bottle B has been replaced (18:14)
  • Online plots crashed twice over 24 hrs
  • Noted that a full rack of TPC power supplies have browned out a few times over the past week (TPC/iTPC sectors 13,14,15). This is an open issue and Wed. during access we should check it out
     

     III. Open issues/status

  • TPC power supply rack experiencing brown-outs. Needs to be investigated during next access
  • FST: plan to access Wed to swap boards (see details above and previous 10 am meeting minutes)
  • Jeff is looking into the online plot crashes

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006388.html)

  01/23/22

    I. RHIC Schedule

  • Collisions for physics

    II. Recap

  • Generally good 24 hours of data taking with no major issues. 3.3 pb-1 & 15 DAQ hrs.
  • Besides low luminosity fill early yesterday following fills look better (2+1 ongoing fills)
  • Already accumulated ~1.5 pb-1 this morning
  • FST Disk 1 Module 12 outer sector: ISEG module board trips due to current fluctuation, current limit increased and HV decreased. ABC board needs to be swapped next access - Xu Sun (see entries 8:34 and 09:48 from today, and overnight shift starting from 06:46)
  • Online plots crashed again this morning. Shift crew have noted in the log book where to find instructions to restart
  • >2000k errors in VC1 in one particular run last evening shift (54). Not noted in log book but assumed shift followed instructions. Was noted during meeting just for information

    III. Open issues/status

  • Alexie will work on addressing gas alarms
  • See FST issue above, will wait till tomorrow to formulate a plan on when to access

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006380.html)
  01/22/22

    I. RHIC Schedule
  

  • Collisions for physics today and tomorrow

    II. Recap

  • CeC all day starting from 8 AM
  • Smooth running overnight when beam came back after CeC (1:30 AM)
  • Beam intensity low in first fill of the day
  • Few iTPC RDOs masked out during cosmics and overnight runs
  • David called and had VME slow controls reset (13:11)

    III. Open issues/status

  • JH will communicate with accelerator to use a 280k ZDC rate threshold to optimize data taking
  • Increased TPC dead time was noticed for an early run by Jeff (fixed by itself next run). SL can check for this under the “Current rates” tab on the DAQ page. If any noticeable changes in rates/dead times during run -> restart run
  • Transient alarm from TOF pressure-transmitter tof_gas_PT-2 (yellow warning, low pressure) - Alexie said he will change today
  • The online plots may crash and not update to latest run, SL needs to check with shift crew each run early

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006374.html)

   

  01/21/22

   
    I. RHIC Schedule
  

  • Cec Au beam dev. will happen today from 8 AM - 9 AM. End time may shift either way by one or two hours (likely end around 8 PM)
  • Collisions for physics after CeC and over the weekend

    II. Recap
  

  • Generally last 24 hours have been good, new run procedure and new beam sequence has nicely improved data taking (~17 DAQ hrs and sampled lumi = 4 pb^-1)
  • Jeff implemented the new production_pp500 configuration disabling BBC readout during day shift
  • Short access in between fills to check sTGC blower (~6 PM). It was running fine and also so was the compressor. Alexie also tuned the laser
  • The alarm handler is not connecting to the VME crates. That is something to be checked when there is no beam (see entry at 23:57)
  • Beam dump @ 01:13 overnight; no beam for rest of shift
  • Slight improvment in polarization and also new beam sequence with imediate first beta* squueze at begginging of fill; second at ~3hrs
  • Unclear when Siemens will (and if) come back online; at least another week of investigation 
     

   III. Open issues/status
  

  • David/Alexie will try to add more monitors for TPC to aid shifters
  • Jeff is working on the error logs to aid shifters remedy the configuration error messages that occurred over last few days; suggest maybe to run pedPhys run at end of extended downtime to preemptively identify any issues instead of during physics run.

  
  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006369.html)
  
  01/20/22

    I. RHIC Schedule
  

  • Physics running till 8 AM tomorrow then CeC Au beam devellopment until 9 PM tomorrow night

    II. Recap

  • Beam dump @ 8 AM; nothing for rest of morning shift due to RHIC issues with rotator scan; took calibration runs
  • Polarimetry runs start at 19:01, and continue until 22:52 (three 3M runs per setting [3x5])
  • Delay in run start due to “l4Disp” in component tree; recovered automatically in 7 min. (00:49) [not critical]
  • High temp. in sTGC air blower; sTGC removed from one run; was already in recovery after previous shift notified Prashanth (00:56)
  • Few TPC RDOs masked out during shift

   III. Open issues/status

  • sTGC reference plots now available
  • Today: Short access (~5 min.) in between fills to check sTGC air blower; current scheduled beam dump is 6 PM
  • Follow-up on BBC crate/L2 timeout discussion. Jeff will set new configuration to not read out BBC to remove this issue
  • Missing HV sections from TPC sectors 8,9,10,11 from yesterdays runs, pointed out by Flemming, will have some followup with potential plots/alarms from Flemming/Alexie

  More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006352.html)

  01/19/22

    I. RHIC Schedule

  • Rotator scan until 4 PM
  • Physics evening/overnight and tomorrow

    II. Recap

  • Beam dumps at 8:00 and 10:55 (beam resumption at 9:40, 16:00)
  • L2 error "More than 2000 timeouts... Suppressing further timeout error messages. STOP the run!” (16:15)
  • Magnet trip 17:22, followed by alarms in sTGC and TOF; fixed by David and Isaac, respectively (see log entry 18:05). EEMC w/ Radstone problem found by Will (see entry at 20:37); Troubles ramping until filter in magnet cooling tower replaced by CAS (20:20) - Following up on periodicity/consistency of filter cleaning schedule
  • In parallel to magnet problem, MCR reports cryo problems. No beam till end of shift
  • FST HV RDO issue at end of shift (00:12) ("Failure code 2”) while turning on), procedure followed to restore; fixed by expert
  • Problems with FCS ("FCS:DEP09:1 failed”); runs until 2:39 had no FCS until fixed by Tonko (see entry at 2:39); also around 3:53
  • Run 23019011 - scalar rate of BBC, VPD, ZDC and EPD are much higher than expected (plot in entry at 2:16) - Jeff looking into this (possible mis-config.)
  • TPC GUI computer frozen (5:18); had to restart computer ; Possible solution to be investigated - Wayne
     

   III. Open issues/status

  • Optimization of fills/beta squeezes - JH will discuss with Vincent
  • TPC GUI computer freezing will be looked at by Wayne
  • Large scaler rates for Run 23019011 will be checked by Jeff
  • sTGC critical plots for shifters

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006347.html)

  01/18/22

    I. RHIC Schedule

  • Schedule: physics until Wednesday 0800 (https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York)
  • Rotator scan on Wednesday 0800-1600
  • Low lumi runs tentatively scheduled Sat 9am - Sun 9am
  • Double beta squeeze test was a success, we will start doing this but during a polarization measurement.

    II. Recap

  • Tim fixed the compressor for the sTGC-FCS air blower after the meeting yesterday (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63743)
  • 19:30 Magnet trip
  • DAQ error "Trigger Bits corrupted: (0-5) set=0x00000000 read=0x00000040 addr=0x512114". This is currently unknown. Jeff is looking into it.
  • BBC taken out last night. The BBC TAC difference was empty since. Akio contends that this plot is empty whenever the BBC is out, so it's not indicative of a problem. Akio is changing his code so that it might be an indicator in the
  • 4 low lumi fills (2*10^10 bunch intensity -- 56 with STAR magnet off and 28 bunches) requested for later this week. Nominal schedule is 9am Saturday to 9am Sunday. Refer to Elke's slides for more detailed and current information

   

    III. Open issues/status

  • We need some clarification on the BBC crate issue. I thought this TAC difference plot was important. If not, the instructions need to be updated.
  • Short instructions for fixing holes in the BEMC need to be updated. They were proven wrong last night.
  • sTGC still has no critical plot instructions.
  • AH GUI still mislables the air blowers.

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006317.html)

   

   

  01/17/22

    I. RHIC Schedule

  • Schedule: physics until (at least) Wednesday 0800 (https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York)
  • Double beta squeeze test might happen this evening

    II. Recap

  • sTGC air blower AC compressor needs a fix, whenever there is access next time. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63692). Confusingly this is under the eTOF in the AH, but this one is serving the sTGC and FCS. The eTOF shares the TPC blower this year.
  • Work on the air blower is happening this morning
  • We can probably run without the error blower. The sTGC LV will drop if it hits a limit (currently 80F). Without the compressor we're still generally under this limit, but we're close (~77F).
  • Many issues with the trigger, as we discussed yesterday. bbc errors from the daqlog will be suppressed so that the operators should never see those ErrCnt messages. Not all events with BBC errors make it to L2, so not all are counted for the 200-error run stop limit. That's how we've been going beyond 200. Crews do not need to stop runs for this. Hank might get Chris to increase this limit from 200.
  • BBC issue might affect analyses that use the earliest TAC information. The TAC difference is from the earliest TAC, so the information is there. This should affect a very small fraction of events, but it might need to be considered by future analyzers (or maybe even the production?)

   

    III. Open issues/status

  • air blower status
  • sTGC critical plot information
  • rotator study (schedule not fixed)
  • Detector readiness checklist needs updating with new BEMC instructions and new BBC instructions

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006326.html)

   

   

  01/16/22

    I. RHIC Schedule

  • Schedule: physics until Monday 1600 (https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York)
  • Double beta squeeze test might happen Monday evening

    II. Recap

  • Updates on BBC instructions "Slow DMA error". Runs are going beyond 200 errors, which is a surprise.
  • TPC ran at near 100% dead from about 2200-0030. TThe node causing the problems was TPX35. We tried removing the node and the errors went away. I then tried masking the boards in these run (sector 21 RB 5&6 and sector 22 RB 5&6 respectively). The errors persisted whichever boards were masked. We then rebooted the tpx35 computer and the errors stopped. (Summary Report - Night Shift). We may need mass computer reboots during some down time (only ~10min)
  • Scalar board 6 is having unknown issues. It was out from run 29 yesterday to run 44. It's out again today.
  • We'll get some more detailed instructions on when to reapply voltage to the BEMC

   

    III. Open issues/status

  • sTGC shift crew manual
  • Instructions need to be updated for BEMC, BBC crate, and the detector readiness checklist.

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006303.html)

   

   

  01/15/22

    I. RHIC Schedule

  • Schedule: physics until Monday 0800 (https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York)

    II. Recap

  • Power dip in the afternoon, it took some time to get everything back.
  • Several accesses yesterday. Tim had to go in late to work on the TCIM.
  • One very-short fill overnight. Poor quality beam. High emittance and low polarization.
  • The ZDC scalar rate is not correct, while MCR use ZDC rate to tune the beam. J.H Lee has called MCR to use BBC rate instead of ZDC rate to tune the beam. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63575)
  • This ZDC problem came from a threshold change on a NIM crate after the power dip that depressed the values ~50% (can be seen by comparing to the BBC values). 
  • RICH scalars were out after the dip.
  • The sTGC crate issue seems to have been solved.

   

    III. Open issues/status

  • sTGC shift crew manual
  • Still talk of 2 beta squeezes
  • Plans for low lumi runs next week (Tuesday?)

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006286.html)

   

  01/14/22

    I. RHIC Schedule

  • See https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York
  • Injector dead until ~midnight tonight

    II. Recap

  • Injection plan is tentatively for midnight. Until then zero-field cosmics.
  • We're turning off the magnet now (ramping to zero current, no state change). Will be turned on ~midnight.
  • Access can be done. Tim went in yesterday to replace a flash card. Today the plan is Tim will replace an sTGC raspberry pi, Christian will replace two failed DEP boards for the FCS, and Prashanth will go in for a quick inspection.
  • Oleg will switch the ECal and work on the system after the DEP board replacement.
  • Jeff fixed instability with Configuration speedup. There should be a significant speedup for run starts now.
  • ^ We will change to 30 minute runs.
  • Tonko made a change in the ITPC processing in case of auto-recovery failure:

    Before) If the auto-recovery fails 4 times I force-stop the run with an appropriate message.

    Now) If the auto-recovery fails 4 times I raise iTPC BUSY with an appropriate message but I DON'T force-stop the run. In this case the forward program continues and it gives the Shiftcrew some time to figure things out.
    Crews are not expected to clear this busy.

  • A brown out this morning seems to have taken out a few sectors in the TPC and nothing else.
  • Carl has written detector requirements for the triggers in case, thus no detector -> disabled triggers. This email has been circulated to the trigger board and (will be) to the ops list. If you have expertise or opinions, review this.
  • Elke wants detector experts to get ready for the low luminosity runs. This is just a reminder.

   

    III. Open issues/status

  • sTGC shift crew manual
  • sTGC LV still has some issues

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006275.html)

   

   

  01/13/22

    I. RHIC Schedule

  • See https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York
  • Injector dead until late Friday at the earliest due to generator issue. We're holding on to the beam until 2pm.
  • CeC will take most of the time until then. Without the injector CeC is just with the electron beam. Thus we can take cosmics and go in for access.

    II. Recap

  • The collider will switch from the Siemens to the Westinghouse injector, which will reduce the quality of our fills.
  • Took out BBC from the triggers, as we always reached 200 BBC errors "Slow DMA error" (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63341) -- BBC not configured for the next 12 hours
  • Multiple problems with the FCS and FST stopping runs
  • We need some way to communicate to the shift leaders about what to do after repeated errors.
  • Jeff will work on run control during CeC
  • We should take advantage of the low luminosity fills from the Westinghouse generator to do some work which should be detailed later.
  • Tim will go in to check the LED control box for HCAL after the beam dump

   

    III. Open issues/status

  • No injection into RHIC, no beam
  • Daniel wants to do a few quick runs for the sTGC (one with and one without beam)

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006267.html)

  01/12/22

    I. RHIC Schedule

  • See https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York
  • Access during meeting -> 5 h CeC -> physics

    II. Recap

  • Access right now (until 11am) -> 5 hours of CeC (4pm) -> Physics
  • BCE had to be power cycled because the 0x1d000000 board failed to configure.
  • The BBC hit 200 errors and was out of the trigger for the rest of the night.
  • Hongwei reported that he thinks he fixed the issue of l4 hanging run stop requests.
  • Jeff is working on making run start faster. Perhaps we'll switch to 30 min runs tomorrow.
  • Carl reported that the forward trigger is reading out the MTD and TOF. TOF mult may be useful, but the MTD likely is not. That should be removed.

   

    III. Open issues/status

  • Still no sTGC critical plot instructions
  • 30-minute runs soon
  • We're currently at ~ 80% of requested luminosity from RHIC

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006258.html)

   
   

  
  01/11/22

    I. RHIC Schedule

  • See https://calendar.google.com/calendar/u/0/embed?src=boaudkh2sbs1lqsqgsjal563k0@group.calendar.google.com&ctz=America/New_York
  • Access planned for 0700-1100 on Wednesday. (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006226.html)

    II. Recap

  • Jeff: As per trigger board meeting adjusted rates in production_pp500_2022:

    Write ADC from every 101 events to every 201 events
    JP0 - removed trigger
    JP1 - reduce rate from 70 -> 35hz
    BT0 - reduce rate from 180 -> 100hz
    dimuon - reduce rate from 300 -> 250hz
    (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63133)

  • Magnet trip at ~6:42pm. Unfortunately the eTOF LV was forgotten again. This is a difficult thing to communicate.
  • Jeff added the logscales for the trigger plots, added l4_prim_nDedx/nHits plots to the primary tracks tab, disabled the BES & fixed target tabs in the l4 display, and added sTGC to the 'critical plots' tab.
  • EEMC was out all night
  • DAQ went 100% dead last night. Tonko came on and helped the crew fix the problem after some confusion (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63187). JP1 and JP2 were firing at 10 MHz because of the EEMC. Run came back to normal after removing these triggers.
  • The EEMC was fixed at ~9am.
  • Carl is working on a calorimeter matrix so that clicking out the EEMC will disable these triggers (among other such trigger/subsystem connections)
  • Chris implemented a gain correction for the QT boards.
  • Tonko proposed a color change for the BBC errors in the DAQ monitor, so that they don't drown out other errors.
  • Jeff moved some ethernet cables, increasing the data speed by ~15%. There are a few cables that might still be moved.

   

    III. Open issues/status

  • sTGC LV is constantly disconnecting. The solution to this is still not known.
  • We need sTGC critical shift plot instructions for the shift crew
  • We need higher luminosity
  • Daniel's 2 5-minute sTGC runs

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006245.html)

   
   

   

  01/10/22

    I. RHIC Schedule

  • TBD

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Communication with MCR about luminosity should probably be about ZDC and (without killer bit) rates. MCR has more rates to look at, but we want the communication to be clear. Angelika will make something like Jamie's singles-corrected value for them to look at too.
  • Daniel wants to do a short sTGC test today - only 5 min run x2
  • Tonko recoded the ETOF TCD so that it fires the ETOF whenever TOF is in the run and gets a trigger.
  • MTD bias scan: 23009043 (13.1 kV), 23009044 (13.2 kV), 23009045 (13.3 kV), 23009046 (13.4 kV), 23009047 (13.5 kV), 23009048 (13.6 kV), 23009049 (12.8 kV).
  • Jeff and Tonko are working on speeding up the DAQ. Jeff will spend a few hours during the CeC today about this.
  • We've had a lot of L4 crashes at run stop. The run won't actually stop until L4 is rebooted. Some events might be too big for L4 and the buffer may need an increase. Jeff and Hongwei are working on this.

   

    III. Open issues/status

  • sTGC LV is constantly disconnecting. The solution to this is still not known.
  • We need to cull some QA plots
  • We need sTGC plots in the critical shift plots (this is underway)
  • We need sTGC and FST critical shift plot instrucitons for the shift crew
  • We need higher luminosity
  • l4 crashing at run end
  • Daniel's 2 5-minute sTGC runs

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006222.html)

   

  01/09/22

    I. RHIC Schedule

  • Physics until 0800 Monday.

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Tim replaced a control card for sTGC LV crate #2 during access at the end of a fill. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62992)
  • MCR had some issues leading to low luminosity fills and they haven't really managed to hit the luminosity they had a few days ago.
  • MCR might be able to give us two beta squeezes/fill.
  • sTGC LV lost communication frequently. Every time when we need to change the status we will have to restart the sTGC LV IOC and re-build connection. But the communication would be lost again within 5-10 minutes. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63042). This seems to be ongoing. It isn't stopping data taking, but it is an inconvenience for the shift crew. Once Tim thinks he understands it the fix will likely require an access.
  • Bill has proposed a sort of triangle cut to remove high-rapidity clusters in the TPC to help the DAQ. The number of primary tracks looks reasonable, but the number of globals is huge. He will send out a document on this to be discussed during the triggerboard meeting (Monday 1pm). It may need some additional information from the TPC group and the spin PWG.
  • The trigger group has implemented code to skip these problematic BBC issues. This should ease the issues it causes for STAR. They're still working on the source.
  • We have a lot of outdated QA plots. If you are/were a subsystem expert please check (see https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006211.html)
  • Tuesday will be cold. This might be a concern for the sTGC gas vent.

   

    III. Open issues/status

  • sTGC LV is constantly disconnecting
  • We need to cull some QA plots
  • We need higher luminosity

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006204.html)

   

  01/08/22

    I. RHIC Schedule

  • Physics until 0800 Monday.

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Yesterday MCR desteered the beams to check how much of our background was coming from single beams. Angelika: "The result from our quick separation test earlier today was that the BBC singles rates came down to 25-25 kHz with full separation (1.8mm). This corresponds to 0.24-0.32% of the singles rates at full overlap. This does not look like background (consistent with it not responding to collimator settings) and appears to be associated with collisions."
  • ^ Thus we cannot collimate out the background. There is ongoing discussion why the BBC singles rates are higher than in 2017.
  • Special FCS gain calibration runs taken yesterday afternoon (23007067, 68, 69, 75, and 76). We've requested production for these.
  • We asked RHIC a few days ago to reduce the luminosity ~20% to ~250k ZDC and rates. After this meeting JH requested we go back up to ~300k. We plan on asking for ~20% higher again in the future.
  • We would like 2 beta squeezes/fill. JH will ask about this.
  • Jeff set approximate scalar ranges for the DAQ monitoring page. Bright red means the scalar rate for the trigger is too high. Black means the scalar rate for the trigger is about right. Brown means the scalar rate for the trigger is too low. These are only preliminary numbers. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62904)
  • Geary would like to do an HV scan for the MTD. This would comprise 6 20-min runs. These are normal physics runs, they'll just have different MTD biases.
  • It looks like any issues with the TPC automatic restart from Jeff and Tonko were ironed out. This will save the last DAQ restart from clusters of runs that cannot start because of a TPC RDO.
  • Hongwei fixed some issues that looked like L4 was stopping the run. Jeff is still working on an issue where a window will pop up saying "configuration not sent" (I think the quote is right) after everything configures and will require a run restart.

   

    III. Open issues/status

  • FCS end-of-fill runs
  • 2 beta squeezes/fill request
  • We will start requesting more luminosity

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006174.html)

   

   

  01/07/22

    I. RHIC Schedule

  • Physics until 0800 Monday.

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • MCR will do some background tests by unsteering the beam at the end of the current fill
  • We will do some FCS gain scans on either side of this desteering
  • First fill of the night had about as high luminosity as we have seen this run
  • Several problems occurred at once. At flat top the TPC cathode
       wouldn't come up, the sTGC had connection errors, and the EEMC GUI went
       white. It took a long time (1.5h?) to get the first two back. The EEMC
       wasn't back in runs for ~5h.
  • VPD got new TAC offsets. Jeff updated the trigger IDs (
       https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62817 +
       https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62820(5am) "
  • Tonko called in and made us enable all the triggers, more than
       half of them were not enabled since previous shifts." -- I suspect this is
       from triggers being taken out for VPD TACs
  • Daniel suggests we change the max TAC for the VPD. There will be an
       email about this. There also seems to be an issue with ch14 on the west
       VPD. This will be masked out.
  • Shuai updated the MTD timing window after the VPD was changed. We
       should change the trigger ID.

   

    III. Open issues/status

  • RHIC desteer beams to check background
  • FCS end-of-fill runs

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006174.html)

   

   

  01/06/22

    I. RHIC Schedule

  • CeC from now until they finish in the early afternoon. After that there are some beam tests and another hour-long access (~5 or 6pm). Then the focus is on physics until Monday morning.

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • No beam since yesterday. Yellow cavity issue overnight -> CeC took over starting around midnight.
  • Cosmic data taken around midnight was triggering at an extremely high rate due to TOF sector 3. Hank wonders if this is due to the BBC-MIX CPU swap.
  • Jeff and Tonko implemented the TPC autorecovery
  • Carl has a proposal for taking four 30-minute FCS runs at the end of a fill. Hopefully we can do this tomorrow.
  • We will get a new configuration for Tonko to consistently work on the TPC FEEs during the day

   

    III. Open issues/status

  • No beam = no VPD timing, no test of BBC crate, no test of TPC autorecovery, etc.
  • After the splitter was taken out and put back in the VPD pedestals look okay, but they continue to be very wide. This is true on both E/W. It's not known why.
  • FCS end-of-fill runs  + configuration
  • no-TPC configuration for Tonko 

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006154.html)

  
  01/05/22

    I. RHIC Schedule

  • After access we go into physics until 0800 tomorrow morning. 0800-1600 tomorrow is CeC.
  • Long access from 0700-1700 today.

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • 11:15 yesterday access. Christian disconnected DSMI boards. Prashanth checked sTGC gas lines. I think we can do a bit better at people who come in for access writing shift entries.
  • MCR called and said that they have not received the BBC delayed signals since Sunday. Tim will replace a PS on a nim crate tomorrow. This doesn't need access. I called MCR to let them know that they should get them back then.
  • We had issues with the BBC crate not being configured after TRG reboot/reboot all.
  • The CPUs for the BBC and MIX crates were swapped. 1M events were taken without error (w/out beam either). If the CPU is the culprit we'll start seeing the same issues with the MIX crate.
  • Christian tested the VPD splitter. It was found to be fine. Hopefully just reconnecting solves the problem and we can redo the timing.
  • Christian did not have time to look at the eTOF cable while the poletip was open, so this problem is not fixed.
  • sTGC HV was not put on an NPS. If there is a parasitic access opportunity Tim can do this quickly, but we won't request an access.
  • As a proof of principle Tim was able to reprogram 2 TPC RDOs, so the test discussed yesterday was a success.
  • A cable for the TPC gating grid was found to be disconnected. This was reconnected and things can go back to normal in run control.
  • The sTGC gas vent on the roof had partly frozen over. Prashanth is working on a solution, but we may see sTGC gas issues in cold weather.
     

   

    III. Open issues/status

  • We're watching the BBC and MIX crates for issues.
  • The VPD will need new runs to reset the timing
     

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006140.html)

   
   

  01/04/22

    I. RHIC Schedule

  • Schedule: physics beam until 0700 tomorrow. Access from 0700-1700 (01/05). Physics from 1700(01/05)-0800(01/06). CeC from 0800(01/06)-1600(01/06)
  • Long (10h -- 0700-1700) access tomorrow. Check PDF attached to (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006107.html) for the list of tasks.

   

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Beam energy change from 254.87 to 254.21 yesterday
  • Long (10h) access planned tomorrow.
     

   

    III. Open issues/status

  • Today we will have a 15min access for Christian to disconnect the DSMIs from the backplane of the BBC crate and for Prashanth to inspect the sTGC gas lines.
  • Jiangyong will cover day shift for the next few days followed (possibly) by Ben Schweid. Next week is unclear.
  • Tomorrow's 10am meeting will, instead, be held at 5pm BNL time.
     

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006107.html)

   

  01/03/22

    I. RHIC Schedule

  • 0900-1700: CeC
  • 1700 (Mon)-1600 (Tues): physics beam

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Beam energy change from 254.87 to 254.21 yesterday
  • Long (10h) access planned on Wednesday. This includes opening the East poletip. If you would like to enter please respond to the star-ops message.
     

   

    III. Open issues/status

  • VPD max TAC not changed since yesterday
  • The Trigger group needs a 1 hour access for Christian to swap
       processors for the BBC and MIX crates. This should happen tomorrow after a
       beam dump.
  • VMEs lost connection (white in GUIs) -- This requires the main CANBUS
       to be restarted in the IR. This needs a 10 minute access for Tim to go in
       and turn it off and on. We'll wait until right after CeC (maybe 5:30?) or
       any other available time today. In the future (Wednesday?) we need an NPS
       on this.
     

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006078.html)

   

  01/02/22

    I. RHIC Schedule

  • 0800-1600: injector polarization work + physics
  • 1600-0800: physics beam
  • 0800-1600 (monday 1/3) CeC
  • Beam energy change from 254.87 to 254.21 planned today

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Beam energy change from 254.87 to 254.21 planned today

   

    III. Open issues/status

  • Access yesterday at noon. sTGC was fixed by a crate powercycle (will be put on NPS in next major access) while DSM 0x17 was replaced
  • Lost a lot of time from beam aborts last night
  • Clusters of L2 issues happened again. Trigger group is looking into this again.
  • TOF gas freon bottle changeover was neglected for several hours.
  • Carl noted that the fast offline is just looking at the StPhysics stream (MinBias and HighTower) and not at jet events or the forward stream. I'll contact them.

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006057.html)

  01/01/22

    I. RHIC Schedule

  • 0800-1600: injector polarization work + physics
  • 1600-0800: physics beam

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • After midnight (year change) the DAQ monitor messages, the JEVP plots, and all of the scalar algorithms went down. The DAQ monitor messages came back with Tonko ~6am and the servers for the plots weren't restarted until ~7:45am. Jamie messaged Jinlong about the scalars.
  • BBC DSM board stopped many runs, even after yesterday's access to replace a DSM board.
  • sTGC lost connection to a HV board, this needs access.
  • Access at 12pm today for both systems
  • Typically alarms are masked when detectors are ramping. Given the recent issues of HV going down for both the sTGC and TOF without alarms David has proposed removing these masks.
  • Tonko needs a procedure every day to take time to bring back masked RDOs. He also needs accesses periodically to fix ones that cannot be brought back via software.

   

    III. Open issues/status

  • Access today at 12pm to replace BBC crate 0x17 board and check sTGC HV board
  • Tonko's time to bring back RDOs. In the short term we'll just stop the run for him to do this. In the long term we'll get a special configuration for forward measurements not including TPC.

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006066.html)

   

  12/31/21

    I. RHIC Schedule
  

  • 0800-1600 - injector polarization work, 1600-0800 collision  for physics.

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • BBC DSM board issue was a really bad problem starting yesterday
       afternoon. We'll plan on a 15-minute access when Akio/Christian are ready
       to go in and replace the board. This can be done in the 0800-1600 block. It
       looks like it will be 1:15pm.
  • Many TPC errors hurt running
  • L4 had a few issues that Hongwei fixed.
  • We're planning a beam energy change (254 GeV beams) for better
       polarization perhaps on Tuesday (01/04).
  • Polarization: B~40% Y ~50% until some time in the overnight shift when
       blue suddenly dropped to 11% (
       https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62215).
       This is primarily due to them changing the fill pattern at the end of the
       fill, and was not actually representative of the polarization. Elke has
       mentioned that the offline analysis isn't nearly so bad (~40%?)

   

    III. Open issues/status

  • Access at 12:30 for the BBC VME crate

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006046.html)

   

  12/30/21

    I. RHIC Schedule

  • Physics beam all day

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • iTPC caused many issues last night
  • BBC DSM board stopped one run, but not many like before.
  • Several hours of sTGC data were lost from confusion on shift.

   

    III. Open issues/status

  • Does iTPC need some change so that it doesn't stop so many runs in preparation and doesn't hang the system?
  • BBC DSM board issue is hard to understand, but trigger group may replace the board or even the CME CPU

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006028.html)

  
  12/29/21

    I. RHIC Schedule

  • Work on polarimetry in the afternoon (~4pm).
  • Physics at night (~midnight).

    II. Recap

  • production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
  • zdcPolarimetry_2022: trg+daq
  • CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
  • Switched to 45 min runs
  • Laser back to every 4 hours (from 2h cycle) since the TPC methane pressure back to (or slightly higher than) normal.

    III. Tasks for commissioning

  •  sTGC voltage scan planned for tomorrow morning when Daniel and Tonko are both awake

    IV. Open issues/status

  • L2 (which is really the BBC DSM board) stopped many runs in evening and overnight. Hank + trigger group will look into it.
  • sTGC caused many issues in the evening perhaps related to a loss of connection to a raspberry pi
  • crate 7, board 4 of the BTOW complains in run control. This can be ignored, but Tonko will stop it to avoid confusion.
  • L4 seems to be aborting VPD TACs (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006006.html). The rates don't make sense. This is being looked into.
  • Polarizations looked to be about B ~ 30, Y ~ 50 last night.

   More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006016.html)

  
  
  
  12/28/21

    I. RHIC Schedule

      Work on blue injection during the day to prevent increase in emittance
      Collisions later afternoon and overnight
      Maintenance day is rescheduled to Jan 5th, no planned access tomorrow

    II. Recap

      production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      zdcPolarimetry_2022: trg+daq
      CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst

    III. Tasks for commissioning

      a) Local polarimetry

      b) sTGC noise and HV scan and FST HV scan finished yesterday

      c) MTD HV scan, after avalanche/streamer analysis

      d) VPD splitter board (Christian, maintenance day)

      e) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    IV. Open issues

      a) FCS Mpod slot-1 looks dead, no alarm for LV

      b) Eemc-pwrs1 NPS has a network interface failure, spare is available with NEMA 5-20 plug (maintenance day, Wayne)

      c) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th

    V. Plan of the day/Outlook

      a) Work on blue injection during the day

      b) Collisions later afternoon and overnight

      c) ETOF by expert operation

  12/27/21

    I. RHIC Schedule

      Diagnostic for quench detector and ramp development during the day
      Snake settings to compensate for partial snake
      Collisions for physics with store-to-store change in emittance in the afternoon and collisions overnight

    II. Recap

      production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      zdcPolarimetry_2022: trg+daq

    III. Tasks for commissioning

      a) Local polarimetry

      b) FCS gain calibration, full FastOffline for HCal

      c) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (call Daniel + Prashanth)

      d) FST HV scan, sw update without beam, call Xu when physics, together with sTGC, dedicated production configuration

      e) MTD HV scan, after avalanche/streamer analysis

      f) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      g) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    IV. Open issues

      a) Leaking valve replaced for TPC gas, methane concentration from 9% to nominal 10% over these two days, more frequent laser runs (2 hours)

      b) TPX automatic power-cycling, ongoing

      c) Eemc-pwrs1 NPS has a network interface failure, spare is available with NEMA 5-20 plug (maintenance day 29th, Wayne)

      d) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th

    V. Plan of the day/Outlook

      a) Ramp development during the day

      b) Collisions with emittance changes store-to-store later afternoon and collisions overnight

  12/26/21

    I. RHIC Schedule

      Slower ramp rate (x5) due to problem with quench detectors, work scheduled for tomorrow
      Ramp development during the day
      Collisions afternoon with intensity steps and overnight

    II. Recap

      production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      zdcPolarimetry_2022: trg+daq

    III. TPC gas

      a) Fluctuations in PI8 and CH4-M4 since yesterday afternoon, interlock overnight

    IV. Tasks for commissioning

      a) Local polarimetry

      b) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (Daniel + Prashanth)

      c) MTD HV scan, after avalanche/streamer analysis

      d) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      e) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    V. Open issues

      a) Temperature increase in WAH, yellow alarms for several VMEs

      b) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th

    VI. Plan of the day/Outlook

      a) Ramp development during the day, also stores for physics, MCR will inform

      b) Collisions with intensity steps afternoon and overnight

   

  12/25/21
  
    I. RHIC Schedule

      Collisions for commissioning
      Energy scan shall resume on 12/26

    II. Recap

      production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4

    III. Updates

      a) BEMC PMT trips

      b) Set of triggers elevated to physics (entry for Run 22359013)

    IV. Plan of the day/Outlook

      a) Collisions for commissioning

      b) Energy scan tomorrow 12/26

  12/24/21

    I. RHIC Schedule

      Energy scan was interrupted by QLI in blue and power dip (2 out of 6 points done), access ongoing for recovery from the quench (~4 hours)
      Collisions afternoon, intensity steps, and overnight
      Energy scan shall resume on 12/26

    II. Recap

      zdcPolarimetry_2022: trg+daq for part of energy scan

    III. Tasks for commissioning

      a) Local polarimetry

      b) FCS gain calibration, FastOffline finished, ECal ok (pi0), HCal

      c) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (Daniel + Prashanth)

      d) MTD HV scan, after avalanche/streamer analysis

      e) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      f) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    IV. Open issues

      a) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th

      b) TPX automatic power-cycling

      c) Readiness and detector states

    V. Plan of the day/Outlook

      a) Access ongoing

      b) Collisions afternoon, intensity steps and overnight

      c) Energy scan 12/26, call Ernst

  12/23/21

    I. RHIC Schedule

      Energy scan, low intensity, afternoon: intensity steps, overnight: collisions

    II. Recap

      Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      zdcPolarimetry_2022: trg + daq

    III. Tasks for commissioning

      a) Local polarimetry, scan will start later because of a quench (11:30 EST)

      b) Spin direction at STAR, longitudinal in blue is a part of systematic error

      c) Scaler bits timing ok now

      d) FCS gain calibration

      e) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (Daniel + Prashanth)

      f) MTD HV scan, after avalanche/streamer analysis

      g) FastOffline, new request for FCS finished

      h) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      i) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    IV. Open issues

      a) BTOW LV + FCS LV alarm, minor -> major for channel trip

      b) sTGC LV

      c) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th

      d) TPX automatic power-cycling

      e) Mailing lists to inform about any changes + logbook

      f) BTOW PMT recovery when opportunity for access, call Oleg (daytime/evening)

      g) Readiness and detector states

      h) ZDC-SMD pedestal for west horizontal #4

    V. Plan of the day/Outlook

      a) Energy scans, ZDC polarimetry, all detectors for machine commissioning

      b) Collisions overnight

  12/22/22

    I. RHIC Schedule

      Vernier scan, cross section compatible with run 17, energy and squeeze ramps
      Local component in blue beam, possibilities include use of existing snakes or phenix rotator, orbit imperfection tuning and energy scan

    II. Recap

      Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      Polarimetry runs zdcPolarimetry_2022: trg + daq

    III. Tasks for commissioning

      a) Local polarimetry

      b) Scaler bits timing

      c) Trigger for Vernier scan

      d) FCS gain calibration

      e) sTGC data volume

      f) sTGC noise thresholds

      g) MTD gas, more SF6, HV scan, trigger config

      h) FastOffline re-running to include EPD

      i) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      j) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    IV. Open issues

      a) Temperature in WAH

      b) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V

      c) Anode HV for sector 15, channel 3 at 1000 V as default

      d) TPC Chaplin frozen (gui available also on sc3 or on alarm handler)

      e) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)

      f) BEMC critical plots checked by shift crews (holds in general)

      g) Reference plots for critical plots

      h) SL on star-ops list

    V. Plan of the day/Outlook

      a) Scans related to longitudinal component and intensity steps during the day

      b) Collisions overnight

  12/21/21

    I. RHIC Schedule

      9 MHz RF cavity adjusted, can go to full intensity, alignment for yellow abort kicker, IPMs configured
      Snake current increased from 300 to 320 A, blue polarization improved to ~42%
      Stores during the day with intensity steps and overnight

    II. Recap

      Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      Polarimetry runs zdcPolarimetry_2022: trg + daq
      Run with 0 V at TPC 21 outer, 400 V after that

    III. Tasks for commissioning

      a) FCS rates, x10 - 20 higher, tests runs with change in gain/masks, beam position?

      b) sTGC data volume, firmware update

      c) Local polarimetry, spin angle

      d) FastOffline re-running to include EPD

      e) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      f) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

      g) BEMC tolerable tripped boxes, 1 out till 29th, DOs follow procedure to recover, run flag as questionable, note in shift log (specific for crate)

      h) Vernier scan, low number of bunches

    IV. Open issues

      a) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out, meeting today 3:30pm

      b) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)

    V. Plan of the day/Outlook

      a) Stores during the day with intensity steps

      b) Collisions overnight

  ­­­12/20/21

    I. RHIC Schedule

      Blue snake re-wired for correct polarity (coil #3)
      Timing alignment for abort kicker in yellow beam
      Access at 10am for 9 MHz cavity
      Ramp development after the access, then collisions after 5pm till tomorrow day

    II. Recap

      Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      Polarimetry run zdcPolarimetry_2022: trg + daq
      Blue polarization at 30%

    III. Tasks for commissioning

      a) FCS closing

      b) ZDC-SMD hot channel, daughter card to be replaced (Christian)

      c) Local polarimetry, scaler bits (Hank, Chris)

      d) FastOffline completed for previous 3 stores

      e) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      f) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day

    IV. Open issues

      a) Increase in magnet current, east ptt, Monday morning

      b) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out, meeting to determine the risks tomorrow

      c) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)

    V. BEMC operation

      a) Shift crew should star looking at critical plots, they are the same for BTOW as last many years. 2d hit map is main indicator of HV status. There are four boxes tripped since probably Sat., this was not noticed.

      b) Det. operator please don't hit wrong button in HV GUI, that can lead to a long recovery of HV, as it was today ~3 hours.

      c) For operation instruction:

          (a) during long downtime shift should run btow_ht configuration just to check HV was not tripped, looks like during Sat. evening shift no one exercise the system at all.

          (b) given that recovering one PMT box may lead to trip and then long recovery of entire BEMC HV, we better not to do such thing  during overnight shifts for example. Instead, may be, barrel jet triggers should be disabled, and live HT triggers only. Then recover HV between fills?

    VI. Plan of the day/Outlook

      a) Access 10am, beam development after

      b) Collisions after 5pm

  12/19/21

  I.RHIC Schedule for today-tomorrow

  Ramp-up intensity (up to 1.5*10^11) (limited by yellow RF)

  (partial) blue snake ramp-up 

  Collisions with luminosity likely with blue+yellow snakes overnight (111 bunches)

   

  II.Recap

   

  Collisions 111 bunches since 2am, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4

  Polarimetry run zdcPolarimetry_2022: trg + daq

  Abort gap at 2/8

  Intensity ~1*10^11 

  BBC/VPD/ZDC :  0.9 / 0.4 / 0.07M  

  ~55% polarization for yellow  ~0% for blue 

   

  Access: ZDC scaler / TCMI (Zhangbu,Tim) – fixed  

                ZDC SMD E-V 2 hot channel (Aihong) - ongoing 

   

  III.Tasks for commissioning

  a)          Detector performance at higher luminosity / issues

  b)         Any issues with “Beam loss”? (6:43 am) 

  c)          Trigger rates vs beam (ex: BHT3 rate lower ~x2 vs Run17) 

  d)         ZDC SMD hot channel 

  e)          Local polarimetry

  f)           FCS closing Monday morning?

   

  IV.Open issues

  a)         Increase in magnet current, east ptt, Monday morning

  b)        NPS for BC1 for 208V, power cord over two racks (Tim)

  c)         Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out

  d)          Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)

   

  V.Plan of the day/Outlook

  a)             Ramp development (intensity, snake) during the day

           b)             Collisions (run production) in owl shift

   

  12/18/21

    I. RHIC Schedule

      Ramps with higher intensity, abort gaps to be aligned, work for UPS for blue RF 9 MHz cavity
      Collisions with larger luminosity overnight (111 bunches)
      Tomorrow: Snake ramp up, intensity recommissioning, polarized collisions overnight

    II. Recap

      Collisions 56x56 bunches since midnight, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
      Polarimetry run zdcPolarimetry_2022: trg + daq
      60% polarization for yellow from RHIC

    III. Tasks for commissioning

      a) sTGC mapping

      b) FST status

      c) FastOffline requested for st_fwd

      d) ZDC east channel 2 on in QT, no coincidence in RICH scalers after TCIM reboot, incorrect discriminator level, access 2pm - 3pm, SMD to be checked also

      e) Local polarimetry

      f) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      g) FCS closing Monday if blue RF ok

    IV. Open issues

      a) BBC is ok (no trigger on previous xing on east) after power cycle to BBQ, bit check to be monitored (Akio)

      b) Increase in magnet current, east ptt, Monday morning

      c) NPS for BC1 for 208V, power cord over two racks (Tim)

      d) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out

      e) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)

    V. Plan of the day/Outlook

      a) Ramp development during the day, access for ZDC afternoon

      b) Collisions in owl shift

  12/17/21

    I. RHIC Schedule

      Potential controlled access till 1pm, ramp development after (squeeze ramp, blue tune kicker, intensity ramp up)
      Collisions in owl shift

    II. Recap

      Collisions 12x12 bunches since 4am, sTGC and FST voltage scans with field ON, tuneVertex_2022: trg + daq + tpx + itpc + fcs + stgc + fst + l4

    III. Tasks for commissioning

      a) FST (nominal voltages as before) + sTGC voltage scan (sTGC done, 2900 V is default for now)

      b) BBC lost earliest TAC on east, EPD was used for voltage scan instead

      c) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)

      d) Local polarimetry, results west ZDC only, code issue? (Jinlong), polarimetry runs tonight

      e) FCS mapping to be checked after cable swap

    IV. Open issues

      a) Increase in magnet current, east ptt

      b) BC1 fan tray swap, no alarm when ongoing, no on/off via slow controls, NPS? (Tim, David)

      c) Gating grid sector 21 outer disconnected, anode at sector 21 outer at 800 V, RDOs are masked, capacitance consistent with cable alone, 12h min + risk of need to remove parts in front, maintenance 29th

      d) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      e) sTGC has no data in first run after LV power up, under investigation

      f) star-ops mailing list is slow in delivery, also other lists (stgc)

      g) AC in control room

    V. Plan of the day/Outlook

      a) Potential controlled access till 1pm, ramp development after

      b) Collisions in owl shift, production configuration (prod ids except mtd), ZDC polarimetry, FCS closing Sat/Sun

      c) Forward detectors by experts only, sTGC mapping (Daniel)

      d) Saturday: ramp development during the day, collisions in owl shift

  12/16/21

    I. RHIC Schedule

      Blue snake reconfigured for coils #1 and #3, tests for abort kicker UPS
      CeC till 8pm, beam development after
      Collisions in owl shift

    II. Recap

      No collisions because of water flow problem at beamstop, caused by incorrect orifice
      Cosmics, tune configuration

    III. Tasks for commissioning

      a) Magnet on/off? -> feedback from FST by 4pm EST

      b) FST + sTGC voltage scan, procedure will be set by magnet on or off case

      c) MTD, no dedicated commissioning run?

      d) VPD slew parameters loaded, TAC windows set, investigation ongoing for splitter board

      e) Scalers board, signals ok, more than 6 bunches needed

      f) FCS status

      g) ZDC status ok

    IV. Open issues

      a) BC1 multiple power-cycle on crate, SysReset, on/off in slow controls? Fan tray swap when possible (Tim)

      b) Gating grid sector 21 outer disconnected, anode at sector 21 outer at 800 V for no gain, fix at maintenance day

      c) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      d) sTGC has no data in first run after LV power up, under investigation

      e) star-ops mailing list is slow in delivery, also other lists (stgc)

    V. Resolved issues

      a) EPD mapping at the splitter

      b) Magnet monitoring ok after maintenance yesterday, alarm limits ok

    VI. Plan of the day/Outlook

      a) CeC till 8pm, beam development after

      b) Collisions in owl shift

  12/15/21

    I. RHIC Schedule

      Maintenance for CeC and blue snake re-wiring, ramp development after 4pm
      Collisions late afternoon / overnight

    II. Recap

      Collisions with 12x12 bunches with forward detectors, production_pp500_2022, tuneVertex_2022

    III. Open issues

      a) sTGC voltage scan, another scans today (Prashanth + David to be called), in sync with FST

      b) FST voltage scan, looks ok from last night, another scans today

      c) tuneVertex_2022 for sTGC and FST voltage scans, runs for target number of events + add FCS, use BBC trigger

      d) Lists of tasks for collisions from experts passed to SL

      e) FCS status, trigger list

      f) VPD one channel to be checked for max slew - mask out this one for now, TAC window, need feedback on pedestals while still in access, cabling check (Christian)

      g) EPD calibrated now

      h) Cal scan, ESMD PMT voltages updated, ETOW phase to be applied

      i) ZDC towers check ok (Tomas), signal ok

      j) One run with ZDC-SMD HV off, signal cables checked ok on side patch (Aihong)

      k) Cabling check today (Christian)

      l) Scalers board, SMD counts still at RHIC clock (Jinlong)

      m) MTD commissioning (Shuai), VPD trigger and cal needed, instructions for SL by Shuai

      n) BC1 power cycled on crate (Tim), booted ok, CAN address 73 will be set (Christian)

      o) Gating grid status (Tim), sector 21 timing

      p) Laser runs every 4 hours

      q) TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), log affects, delay does not, new tier1 fixed it, readback added

      r) Magnet alarm limits

      s) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      t) sTGC has no data in first run after LV power up, under investigation

    IV. Resolved issues

      a) Commissioning done for: BBC, EPD, BTOW, ZDC

    V. Updates

      a) production_pp500_2022, BBC, BBCTAC, BHT3 and BHT3-L2W elevated (Jeff) + ETOW, VPD (almost)

      b) Contact Jeff when a trigger can elevate to physics

    VI. Plan of the day/Outlook

      a) Restricted access now

      b) Cosmics for gating grid, magnet up preferred

      c) Beam development after 4pm, detectors in proper safe state

      d) Collisions in the evening / overnight

      e) SL tasks shift crew based on what we're running

  12/14/21

    I. RHIC Schedule

      Damage in blue snake after power dip on Sunday evening, could use coils #1 and #3, access to rewire for these coils
      UPS was disabled for abort kicker
      Access now for kicker, snake and CeC, ramp development afternoon, collisions overnight

    II. Recap

      VPD, EPD and Cal scans
      Magnet trip yesterday evening
      Controlled access now (~4 hours)

    III. Open issues

      a) VPD commissioning (Isaac, Daniel), non-VPD trigger (Jeff), slew test with beam

      b) EPD commissioning (Rosi)

      c) ZDC SMD bits in scalers fire at RHIC clock (9.38 MHz), test with HV off, pedestal issue, cabling (Jinlong + Hank)

      d) ZDC commissioning (Tomas, Zhangbu), signal seen, work for 1n peak

      e) Cal scan (Oleg, Will J), BTOW 4ns shift, crate-by-crate scans

      f) MTD commissioning (Shuai), VPD trigger and cal needed, instructions for SL by Shuai

      g) Local polarimetry (Jinlong)

      h) BC1 crate off? fails during boot, spot crash in startup file, power-cycle now (Tim)

      i) Spike in 1st gating grid time bin (David), test now with cosmics

      j) TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), test today, log affects, delay not

      k) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      l) sTGC has no data in first run after LV power up, under investigation

    IV. Resolved issues

      a) Commissioning done for: BBC

      b) Cards in EQ1, EQ2 and EQ3 replaced yesterday (Christian)

    V. Updates

      a) Separate trigger configurations for commissioning (Jeff)

      b) File stream name for forward detectors: st_fwd

    VI. Plan of the day/Outlook

      a) Access now, beam development during afternoon, collisions overnight

      b) Production configuration with final prescales, start with BBC, BTOW, production_pp500_2022

      c) Forward commissioning with low intensity beam, Xu, Prashanth, David, VPD and EPD needed before

      d) Magnet work tomorrow

      e) Scalers need to run

  12/13/21

    I. RHIC Schedule

      Polarization development and ramp development during the day, collisions with rebucketed beam late afternoon or overnight
      Access at IP2
      Low intensity now because of mistime abort in both rings at the power dip
      Cogging depends on snake availability, needed for correct longitudinal position of vertex
      Lossy blue injection

    II. Recap

      Collisions yesterday after 8pm, BBC HV scan, ended by power dip
      Next collisions 5 am, ZDC polarimetry with singles at 2 kHz, VPD HV scan, EPD timing scan

    III. Open issues

      a) VPD HV 13.01 didn’t turn on, at lower voltage (1627 V) now ok, might need to swap the channel

      b) Non-VPD trigger needed (BBC coincidence in L4) for VPD slewing correction, Jeff will make separate configuration file, instructions for SL by Daniel

      c) Separate configuration for local polarimetry (Jeff)

      d) EPD commissioning (Rosi)

      e) ZDC commissioning (Tomas, Zhangbu)

      f) Every trigger detector sends a message over star-ops when done with commissioning

      g) Cal scan (Oleg)

      h) MTD commissioning (Shuai), VPD trigger needed, instructions for SL by Shuai

      i) Spike in 1st gating grid time bin, seen as perpendicular planes in event display, should fix after new pedestal, open/close test after beam dump, IOC restart (David)

      j) TAC windows for BBC, EPD, ZDC, VPD in investigation (Eleanor, Jeff), monitoring to check the registers

      k) L4 was not present because of incorrect R and z vertex cuts, ok now

      l) Collision triggers in tune_22 for calibration and tune configuration

      m) Dead QT32B daughter card for EPD (daughter A in EQ3 slot 10), also cards in EQ1 and EQ2, access needed to replace (Christian), controlled access (SL), SL calls Rosi after done to check

      n) Local polarimetry in progress (Jinlong), not yet from scalers

      o) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      p) sTGC has no data in first run after LV power up, under investigation

      q) iTPC Sector 13, RB 3 masked out and powered off, keep like this

      r) FCS LV was overheating, rack backside was opened (Tim), 1 deg drop, not critical now

      s) No ETOF

    IV. Resolved issues

      a) BBC commissioning done for run 22, Akio not in call list for collisions

    V. Updates

      a) Call list for collisions, SL informs over star-ops

      b) File stream name for forward detectors: st_fwd

    VI. Plan of the day/Outlook

      a) Potential access

      b) Tune configuration with beam development, detectors in proper safe state

      c) Could get collisions later afternoon or overnight

  12/12/21

    I. RHIC Schedule

      Collisions later afternoon (4/5pm), likely 6 bunches rebucketed
      Magnet quenches were caused by temperature problem at 1010A, not beam induced
      Lossy blue injection, work needed on Y2A RF cavity
      Rebucketing successful yesterday with 6 bunches
      Scans and ramp development till 4pm, stores with collisions after
      
    II. Recap

      tune_22: trg + daq + btow + etow + esmd + fcs

    III. Open issues

      a) Global timing with collisions

      b) TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), test with rebucketed collisions

      c) Dead QT32B daughter card for EPD (daughter A in EQ3 slot 10), access needed to replace (Chris)

      d) Local polarimetry (Jinlong)

      e) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      f) First run after LV power up sTGC has no data, under investigation

      g) iTPC Sector 13, RB 3 masked out and powered off, keep like this

      h) FCS LV was overheating, rack backside was opened (Tim), 1 deg drop, not critical now

      i) No ETOF

    IV. Resolved issues

      a) Phones were out yesterday night due to update, fixed early morning

    V. Updates

      a) New Readiness checklist, cosmics with 8+ hours without beam

    VI. Plan of the day/Outlook

      a) Tune configuration with beam development, detectors in proper safe state

      b) Could get collisions later afternoon or overnight, call list for shift leaders

  12/11/21

    I. RHIC Schedule

      Polarized scans and rebucketing tests till 8pm, then CeC until tomorrow morning

    II. Recap

      Collisions at 3am, 28 bunches, both snakes ramped, polarization 44% blue, 54% yellow, beam abort after 20 minutes
      Next collisions 8am, ended by blue quench near the snake (but not the snake)
      Cosmics, tune_22, trg + daq + btow + etow + esmd + fcs

    III. Open issues

      a) Global timing with collisions

      b) Phones out at STAR and MCR due to update to phone system, fake (?) magnet trip in west trim at the same time, now back (9am)

      c) Investigation in DSMs on TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), affects triggers which use TAC, read from registers is different from write, access will be good to test the VME board-by-board (Jeff)

      d) sTGC gas pressure increased after yellow alarm (Prashanth)

      e) Timing for scaler board with beam (Chris), expect to be ok, needed for local polarimetry (Jinlong)

      f) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      g) First run after LV power up sTGC has no data, under investigation

      h) iTPC Sector 13, RB 3 masked out and powered off, keep like this

      i) FCS LV was overheating, rack backside was opened (Tim), 1 deg drop, not critical now

      j) No ETOF

    IV. Resolved issues

      a) L4 plots missing from Jevp, fixed (Jeff)

    V. Updates

      a) New Detector States, 12/10, sTGC for both HV & LV is OFF for PHYSICS and Vernier scan, FST HV is OFF for PHYSICS and vernier scan

      b) Output from individual ZDC towers tested (Tomas)

    VI. Plan of the day/Outlook

      a) Tune configuration with beam development, detectors in proper safe state

      b) No collisions overnight (CeC instead)

      c) Cosmics only if there will be 8+ hours without beam

  12/10/21

    I. RHIC Schedule

      Blue9 snake ramps today till 4pm, there was shorted diode against spikes from transient current

      Recommissioning after that if blue snake is available, or rebucketing and ramp development if not

      Stores with collisions during owl shift if ready by 10pm today

    II. Recap

      Collisions at 4am for short time, ended by multiple beam aborts, access ongoing now
      tune_pp500_2022 with collisions, tune_22 or cosmics, field on

    III. Open issues

      a) Jpsi*HTTP at 1 kHz without beam, hot/warm tower ETOW/BTOW, leave out until calorimeters commissioned

      b) Update in TAC min/max for ZDC, EPD, BBC (Jeff)

      c) BBC HV adjusted to lower values (initial), need to finish HV scan (Akio)

      d) FCS LV overheating, rack backside to be opened (Tim), 1 deg drop, not critical now

      e) iTPC Sector 13, RB 3 masked out and powered off, keep like this

      f) Timing for scaler board with beam (Chris)

      g) Mask from L0 to L1 for a trigger patch

      h) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      i) First run after LV power up sTGC has no data, under investigation

      j) No ETOF

    IV. Resolved issues

      a) Remote access to scalers for polarimetry on cdev for Jinlong, was related to 64bit/32bit change, ok now

      b) Fan tray for EEMC CANbus, crate #70 replaced (Tim), also reboot to main CANbus, gating grid restored

    V. Updates

      a) VPD voltages HV changed to 2013 values (Isaac)

    VI. Plan of the day/Outlook

      a) Schedule from rhic is largely uncertain, could get collisions in owl shift

      b) tune_pp500_2022 with collisions, tune_22 or cosmics, field on, safe state when beam development

      c) FST keep off until very nice beam, expert present for any operation (Xu)

      d) sTGC by expert only (Prashanth)

      e) Commissioning starts with collisions on, state of experimental setup now

  12/09/21

    I. RHIC Schedule

      Possible collision setup in upcoming owl shift, progress on collimator, kicker alignment and timing, vertical injection matching
      and yellow injection damper, safe state important for detectors during beam development.

      blue9 snake: beam induced quench without substantial beam loss, question on magnet training or real problem,
      access today for a p.s. related to the snake

      Today after p.s. access: beam development without blue snake

    II. Recap

      Cosmic runs with field on, tune_22 with beams, trg + daq + btow + etow + esmd + fcs

    III. Open issues

      a) EEMC CANbus fan failure, crate #70, few minutes access to replace the tray (Tim)

      b) sc5 reboot by mistake by DO when trying to reboot crate #70 caused by incomplete instructions

      c) FCS LV overheating, rack backside to be opened (Tim), ½ hour to observe temperatures

      d) Level for yellow alarm for sTGC pentane gas, done

      e) Online database not visible yesterday ~2pm → ~5pm, Dmitry was called

      f) sTGC HV IOC having multiple instances (red alarm), ok now

      g) EEMC and EQ2, MXQ, and BBQ in alarm handler (David, input from experts on what to unmask in alarm handler)

      h) iTPC Sector 13, RB 3 was asserting busy even masked out, was powered off, Tonko + Jeff will take a look

      i) BCE DSM2, new algorithm uploaded, in test yesterday, in trigger, L0 to be checked by Chris

      j) Instructions on recovery for BBC/ZDC/VPD HV system (LeCroy1440) communication after power failure, pwd to bermuda needed

      k) Remote access to scalers for polarimetry on cdev for Jinlong

      l) Add instructions to recover forward detectors after power dip (sTGC call experts), Oleg T will add inst for FSC,  FST call experts

      m) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      n) sTGC auto-recoveries (is a place-holder for final message), empty plots for a few runs → on hold for commissioning

      o) Disk that stores TPC sector 8 pedestals needs to be replaced by Wayne (not urgent)

      p) No ETOF

    IV. Resolved issues

      a) Scaler board replaced during access yesterday (Chris), SCLR48 in trigger since run 22342037

    V. Updates

      a) Update in sTGC HV and LV GUI (channel numbering), instructions are updated

      b) Magnet current limit alarm, Flemming + David for default limits

    VI. When collisions are delivered

      a) Commissioning plan

      b) Time scan for BEMC together with ETOW and ESMD

      c) ETOW and ESMD basic QA with collisions to test if its configuration is ok

    VII. Plan of the day/Outlook

      a) beam development with detectors in correct safe states, tune_22 or tune_pp500_2022, cosmics when possible

      b) p.s. access for blue snake, beam work till midnight, possible collisions setup during owl shift

      c) ETOF may be turned over to SC for a few weeks during the run

  12/08/21

    I. RHIC Schedule

      Test for blue9 snake ok (partial snake, ongoing work), beam work till 10:00, access 10:00 -> 12:00,
      then injection,  ramps and rebucketing till tomorrow (12/09 4pm)

    II. Recap

      Cosmic runs with field on, tune_22 with beams, trg + daq + btow + etow + esmd + fcs

    III. Open issues

      a) Restricted access 10am today, scaler board (Chris, finished), also for EPD (finished)

      b) Add instructions to recover forward detectors after power dip

      c) EEMC and EQ2, MXQ, and BBQ in alarm handler

      d) iTPC Sector 13, RB 3 was asserting busy even masked out, was powered off, Tonko + Jeff will take a look

      e) sTGC HV at 2900 V for now

      f) sTGC auto-recoveries (is a place-holder for final message), empty plots for a few runs → on hold for commissioning

      g) BCE DSM2, new algorithm uploaded, ready to test (during today), not in trigger now

      h) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)

      i) Disk that stores TPC sector 8 pedestals needs to be replaced by Wayne (not urgent)

      j) No ETOF

      k) Instructions on recovery for BBC/ZDC/VPD HV system (LeCroy1440) communication after power failure, pwd to bermuda needed

      l) Access to scalers for polarimetry on cdev for Jinlong

    IV. Resolved issues

      a) BTOW crate Id 8 failed configuration fixed, (disconnected 0x08 board 1 and put it back)

      b) Replaced the problematic DSM1 in BCE crate, hole in trigger patch 250-259 seems gone from btow_ht run, 22340037

    V. Updates

      a) Two screens for sc3 (VPD/BBC/ZDC HV)

      b) 30 new mtd plots to the JevpPlots

      c) evb01/evb07 added to the DAQ default

      d) New firmware in BE004 DSM2

      e) sTGC LV IOC to follow the procedure

      f) To power cycle a EEMC follow the operation guide, power off and on not enough, follow manual strictly

      g) TPC current calibration should be done once per day

      h) Magnet current limit alarm, Flemming + David for default limits, sampling frequency?

    VI. When collisions are delivered

      a) Global timing, tune_pp500_2022 trigger definition

      b) Time scan for BEMC together with ETOW and ESMD, files from DAQ by Tonko, min bias trigger, time interval and steps to be set

      c) ETOW and ESMD basic QA with collisions to test if its configuration is ok, first reference plots will be available with collisions

    VII. Plan of the day/Outlook

      a) beam work till tomorrow afternoon, cosmics when possible

      b) no collisions are expected till tomorrow 12/9 4pm at least

      c) exercise for BBC/VPD/ZDC lecroy recovery after power failure (David)

      d) ETOF may be turned over to SC for a few weeks during the run

  11/17/21 to 12/07/21 Zilong Chang

  11/16/21

  RHIC schedule: no new info: “2-3 weeks” start-up delay (incomplete cryo controls upgrade).

  11/15 Blue 4K cool-down, starting 1/2 (12-6),  11/29 <  for Yellow

  11/15: magnet polarity change RFF -> FF

  11/15: Calibration sets taken: Long Laser run,  polarity flip, long laser run + laser with resistors in the chain  

  Currently 1.5MOhm in the chain, How long?

  will learn from the analysis (Gene) of the data set on the short in TPC

  Magnet stable

  All detectors are included and currently running (except ETOF)

  gmt trigger is enabled

  Issues and resolved:

    MTD:  issue with LV.   RDO masked out (1 out of 2): running / Geary

    BTOW: configuration fail. Fix by resetting board / Oleg

  Plan for today

    new shift crew + period coordinator (Zilong)

    NO access 07am-12pm tomorrow (11/17) for access controls test
   
    cosmic with all available detectors with Forward FF

    run until Thursday morning with FFF

    Flip the polarity back to RFF the polarity on Thursday morning (combined with BBC installation, MTD work)

    let crew know the detector is not ready to be included

    laser / 4 hours (separate run)

    pedestal / shift

    TOF,MTD  noise run / day 

  11/15/21

   

  RHIC schedule: the same: “2-3 weeks” start-up delay (incomplete cryo controls upgrade).

  11/15 Blue 4K cool-down,  11/29 <  for Yellow

  Short term plan:

  11/15: Flipping magnet polarity to RFF -> FF  (BBC installation postponed)

  11/15: Long Laser run (done),  polarity flip (ongoing), long laser run + laser with resistors in the chain (to be done)

  11/18: evaluate the short in TPC with data taken with two field settings,  and decide on the need to open East pole-tip to fix the if necessary

  magnet stable

  a trip yesterday 4:30pm with “daily” power dip

  TPC GG issue resolved: with correctly reloaded value

  Issues, detectors not included:

    FST: Error with HV ramping / 7am

    STGC running, included but HV off

    MTD: too many recoveries.LV control / 3am

  Shift procedure

  FST, STGC under shifter control?

  pedestal after “warm up” time

  Plan for today

    cosmic with field on with all available detectors with RFF -> FF

    long Laser runs

    TPC, BTOW, ETOW, ESMD,TOF, FCS, sTGC, FST, MTD

    let crew know the detector is not ready to be included

    laser / 4 hours (separate run)

    pedestal / shift

    TOF,MTD  noise run / day

  
  11/13/21

  RHIC schedule: the same: “2-3 weeks” start-up delay (incomplete cryo controls upgrade).

  11/15 Blue 4K cool-down,  11/29 <  for Yellow

  Any beam activities with only Blue cold?

  Short term plan:

  11/12 - 11/15: continue with cosmic data taking at Reverse Full Field

  11/15 Monday morning: Magnet polarity flip, BBC (West) installation

  11/15 - cosmic (+laser) data taking at Forward Full Field.

  11/18: evaluate the short in TPC with data taken with two field settings,  and decide on the need to open East pole-tip to fix the if necessary

  magnet stable

  trip yesterday likely from power dip

  alarm: set value, range to reduce false alarm from fluctuation

  ETOW: cable fixed. DAQ error. trigger/hardware/DAQ issue?

  FST overheating module 3-11. not resolved. masked-out.  Still out of run?

  STGC: DAQ 0. Still out

  shift QA plots, online QA, event display: lagging

  laser run: separate

  Plan for today

    cosmic with field on with all available detectors

    TPC, BTOW, ETOW, ESMD,TOF, FCS, sTGC, FST, (MTD)

    let crew know the detector is not ready to be included

    laser / 4 hours (separate run)

    pedestal / shift

    TOF noise run / day