Tomorrow: 7:00--
- 7:00-11:30 CeC access
- 12:00-17:00 APEX
- 17:00 - STAR low lumi. fill
II. Notable items/recap from past 24 hours: Smooth Physics
III. Items from shifts:
- Day shift: Smooth Physics runs (ended ~11:15am) + Cosmic runs
- Masked out iTPC sector 13 RDO 3
- Alexei restarted Laser PC, fixed laser GUI no-response issue
- Akio uploaded a new electronic gain file for FCS ECal, Jeff updated associated production ids
- Evening shift: Smooth Physics runs (19:00-)
- sTGC lost communication, and both crates had to be rebooted and powercycled
- BEMC sector 10 had hits missing for half a run. Had to be powercycled
- Owl shift: Smooth Physics runs
IV. Other items: may request to reduce CeC and APEX time to get more physics runs
02/28/2022
I. RHIC Schedule
- Today: Collision for Physics
- will have a local pol test with DX angle 14:00, nominal end of store 14:25
- Tomorrow: Collision for Physics
II. Notable items/recap from past 24 hours: Cosmics + Smooth Physics
III. Items from shifts:
- Day shift: (first a few hours) Cosmic runs + (11:15) Smooth Physics runs
- many auto-recoveries in TPX, iTPC, TOF
- sTGC error "ROB #01 bad FEB.”, powercycle ROB #1
- FST 100% dead, no errors in scrolling DAQ messages. Power cycled FST.
- Evening shift: Smooth Physics runs when beam available (no beam: 19:00-21:00)
- TPC Anode Trip, GMT HV needed powercycling
- Owl shift: Smooth Physics runs
- missing bins in EEMC Tower ADC plots, expert power cycled crate 92, and performed master reload
- laser control GUI (where the camera is controlled) froze, Alexei will investigate during day shift
- ITPC: Sector 23, RDO 4 -- powercycle then restart run (several times)
Other discussions: request low lumi runs (Wed or Thu ?)
02/27/2022
I. RHIC Schedule
- Today: Collision for Physics till Monday morning, exchange the cathodes (~ 2 hours)
II. Notable items/recap from past 24 hours: Cosmics + Physics + Cosmics
III. Items from shifts:
- Day shift: Cosmic runs + 3 Physics runs
- node TPX[19] removed / masked from run configuration
- Evening shift: 13 Physics runs
- node TPX[19] removed / masked from run configuration
- Owl shift: 5 Physics runs + Cosmic runs
- TPX: RDO S01:6, TPX: RDO S01:3, ITPC: Sector 1, RDO 4, ITPC: Sector 10, RDO 3
- partial trip of BTOW PMT 10
- Beam lost ~2:40 (failure of a Yellow Abort Kicker, requires a Thyratron replacement)
Other discussions: CAD should do a better job to deliver beam
02/26/2022
I. RHIC Schedule
- Today: Collision for Physics till Monday morning, exchange the cathodes (~ 2 hours)
II. Notable items/recap from past 24 hours: Cosmic runs
III. Items from shifts:
- Day shift: Cosmic runs
- Alexei changed a burned mirror for East laser (camera 3), more laser tracks on East part of TPC
- 12:37 Tonko fixed a bug in the ROB firmware which caused odd behavior under sparse data taking conditions (cosmics, low lumi runs). The new code version is 0x126 (old, up to now, was 0x124)
- Evening shift: Cosmic runs
- Owl shift: Cosmic runs
- sTGC air flow alarm went off, slow control issue
- Tonko: “Now all STGC RDOs have the newest firmware version (0x126).” 4:40am
02/25/2022
I. RHIC Schedule
- Today: 7:30 access to exchange garages, start bake load-lock chamber (~ 8 hours)
- Collision for Physics till Monday morning, exchange the cathodes (~ 2 hours)
II. Notable items/recap from past 24 hours: Access + Low lumi + Smooth Physics
III. Items from shifts:
- Day shift: Smooth physics till 13:30 + CeC Access
- A few powercycles of iTPC RDO, sTGC ROB, BTOF, DAQ
- Evening shift: Low lumi runs till 21:13 + normal physics runs started 23:59
- testVernierScaler, forwardCrossSection, zdc_Polarimetry_2022
- trigger was 100% dead. sTGC plots looked abnormal, and had missing tiles, stgc_n_sigma reset to 12 and 8, taking a production run to check what the issue is with sTGC, still open issue (days to solve) will request another low lumi runs after figure out sTGC issue
- MXQ crate had to be powercycled
- Owl shift: Smooth Physics runs till 5:30 am (beam lost)
- False magnet trip alarm, apparently due to communication or slow control glitch
- BEMC PMT box 10 west partially tripped, solved by reapplied HV
Others: if plan to make use of the access time, please bring up
02/24/2022
I. RHIC Schedule
- Today: 14:00-24:00 STAR Low lumi. Fill with Field on (Forward experts should watch the online plots to make sure detectors are running good)
- Friday: Aperture scan and crossing angle for local pol. measurement
II. Notable items/recap from past 24 hours: Access + Smooth Physics
III. Items from shifts:
- Day shift: Access
- Evening shift: Smooth Physics started from 18:33
- Owl shift: Smooth Physics
- FCS: persisting errors “FEE count 44, expected 5”. FCS GUI no responding, Oleg power cycled FEEs manually cleared error message, however the GUI is still frozen, solved by restart IOCs
- iTPC node 13 died at end of the evening fill, back online after powercycle corresponding computer
02/23/2022
I. RHIC Schedule
- Wed: 7:00-13:00 Maintenance
- Thu: 16:00-24:00 STAR Low lumi. Fill with Field on
II. Notable items/recap from past 24 hours: Smooth Physics
III. Items from shifts:
- Day shift: CeC access + Smooth physics + AGS Snake issue
- TPC gating grid lost communication, solved by reboot software
- Evening shift: Smooth Physics + a few issues
- Dimuon triggers were not included in time
- "Incorrect Trigger Clock 76 MHz" error due to stuck in PLL. Had to powercycle TCD and turn MXQ on
- FCS PS channel U302 got stuck. Power cycling MPOD crate recovered that channel
- Owl shift: Smooth Physics
IV. Access today:
- Tonko: DAQ tests with TPX and iTPC, don't need access but no runs with Run Control, finished 10:15.
- MTD group: 2.5 hrs with field cosmics after Tonko’s test. MTD HV 13.2 kV to check eff. Vs. 13 kV
- Wayne:
- replace a disk in the computer named deneb, on the 2nd floor of the South Platform
- apply some routine software and OS updates to the shift-leader PC in the Control Room
02/22/2022
I. RHIC Schedule
- Today: 10:00-12:00 CeC access, after will be Collision for physics
- Wed: 8:00-16:00 Maintenance
- Thu: 16:00-24:00 STAR Low lumi. Fill with Field on
II. Notable items/recap from past 24 hours: Smooth Physics
III. Items from shifts:
- Day shift: Smooth physics
- BTOW: PMT box 10 west partially tripped. Solved by reapply HV
- Evening shift: Smooth Physics
- MTD HV BL 24-30 HV cannot be set to full from standby at first try, second try succeed
- ETOW crate Id 5 configuration failure, solved with expert’s help
- Owl shift: Smooth Physics
- BTOW: PMT box 10 west partially tripped. Solved by reapply HV
- sTGC error: 1: FOB 4 (R#13:F#77) NOT found, power-cycle ROB #13
02/21/2022
I. RHIC Schedule
- Today-Tuesday morning: Collision for physics
II. Notable items/recap from past 24 hours: Smooth Physics + a few issues
III. Items from shifts:
- Day shift: Smooth physics
- GMT -> APV -> pedestals_APV_0-3 pulse height for all channels in ARM:0, APV0 shows a second band and scattered values all throughout
- BTOW Crate Id 1: Oleg put under manual control -> Watch trigger rates and reconfigure if needed
- Evening shift: Smooth Physics
- Global communication problem, fixed by David
- "too many recoveries - powercycle TOF LV”, Tray 67 (dead, masked out) was accidentally enabled by DO, which caused the failure of auto recovery. Zaochen updated instruction to reduce the mistake
- Owl shift: Physics run + issues
- DAQ issue: EVB[3] [0xF103] died/rebooted) EVB3 is suggested to be out of run for now. writing data issue (No space left on device)
- error " Incorrect Trigger clock 10.000 MHz”, solved by powercycle TCD and turn on MXQ from off. Tonko:”One TCD board was stuck and couldn’t lock its PLL, it happens after too many Trigger reboots”
- high rates of JetPatch triggers, solved by reconfigure BEMC. Related to TCD powercycle.
- 6:17am, TPC Gating-grid monitor lost connection, need access to powercycle hardware (8:50am, Tim cycled crate 50, gating grid control is back now)
- IOC for sTGV LV lost connection, solved by David
- Laser brightness has no response to control, drift velocity looks normal. Alexei fixed this.
02/20/2022
I. RHIC Schedule
- Today-Tuesday morning: Collision for physics
II. Notable items/recap from past 24 hours: Smooth Physics
Main discussions: Carl: "correct" bunch crossing has barely more hits than any of the visible out of time bunches. In this environment, it’s impossible to see if 2-5% of the triggers are late by a RHIC tick. We’re going to need a 28 or 56 bunch fill to answer this question. ZDC_Polarimetry runs can go back to “TRG + DAQ only”, Will try “TRG + DAQ + FCS” on sometime in middle of next week (Wed-Fri, decide in schedule meeting?)
III. More items from shifts:
- Day shift: Smooth physics
- FST plots were empty during run, but filled after run stops.
- JP triggers hot, solved by restart run
- Run stopped by EQ crate, solved by power cycle and reboot trigger then restarted run
- Evening shift: Smooth Physics
- PMT box 10 west partially tripped, solved by reapplied HV
- from run 23050064: "dimuon" and "hit_dimuon_mon" are on Enabled list
- Owl shift: Smooth physics run
- BTOW Configuration failed for crate 1 (will mask out from auto configuration, do similar thing as crate 8)
02/19/2022
I. RHIC Schedule
- Today-Tuesday morning: Collision for physics
II. Notable items/recap from past 24 hours: Smooth Physics when beam available
- GMT HV alarm went off, HV has no response to “Physics” or “Standby” mode in GUI, Nikolai fixed the GMT. It had lost connection. Fixed by rebooting the crate and restarting the controls
III. More items from shifts:
- Day shift: Physics started 10 am
- sTGC air blower, solved and discussed yesterday 10 am
- Alexei fixed TPC air blower
- Evening shift: Smooth Physics
- Owl shifts: Smooth Physics run (till 5:30, injection issue, solved 7am)
- FCS FEE LV caused missing hits in FCS ECal's North 18~24 and high rates of FCS ECAL triggers. Resolved by powercycling FCS FEE LV
- BTOW Configuration failed for crate 1 (23050011, 23050013) and L2 issues (23050016)
- iTPC errors (S2, RDO 1; S16 RDO 1). Tonko fixed
IV. other items:
- zdc_polarimetry runs please do with (TRG+DQA+FCS)
02/18/2022
I. RHIC Schedule
- Today-Tuesday morning: Collision for physics
II. Notable items/recap from past 24 hours: Snake scan + Physics + IssueMain issues:
- TPC gas alarm at 03:35 and called Prashanth and Alexei.
- 4:20 there was a power dip, which caused magnet trip, problems with water system and sTGC blower trip.
- Alexei and Prashanth fixed the TPC gas and water issues, CAS brought back the magnet
- 7:10 Tim and Mike access to fix issue with sTGC air blower (6:00, MCR: a spark in ion source). Compressor not running, after replaced compressor contractor, Compressor is now running. Replaced LV relay for preventative maintenance. Mike connected aux contacts from compressor relay to monitor contractor state. Tim replaced sTGC AHU Contactor, Capotosto added remote monitoring for contactor state to sTGC interlock cabinet. Just fixed all issues, sTGC can be added in the run now.
- GMT HV alarm went off, HV has no response to “Physics” or “Standby” mode in GUI, issue is still open
III. More items from shifts:
- Day shift: snake scan for spin direction
- BTOW is flashing blue (W) for >30 minutes
- Evening shift: Smooth Physics while beam is available
- Beam came at 19:35, beam lost at 21:12, beam back at 23:20
- Owl Shift: Smooth physics run
- Trigger: BBQ and EQ2 VME crates tripped, then L2 was stopping the run. Powercycling BBC and EQ2 crate and rebooting the trigger solved the problem.
- Online plots issues of TPC, MTD and ESMD.
IV. other items:
- Request for 2 polarimetry runs at the end the fill
02/17/2022
I. RHIC Schedule
- Today: 8:00-14:00 Snake scan for spin direction
- Tomorrow: Collision for physics
II. Notable items/recap from past 24 hours: General: APEX + Smooth physics
- Main issues: A mistake to assign channel 1 of the 2nd board to channel 1 of the 1st board. So the channel was not turned on when controlled by the GUI. David fixed it and HV is back on that sTGC chamber. Online plots back to normal.
III. More items from shifts:
- Day shift: APEX
- Alexei replaced GMT tank-1
- David found HV channel was mis-mapped, online plots shows normal
- Evening shift: Smooth Physics started 21:40
- JP1 was high due to hot trigger patches on crates 0E, 0D, 05, solved by power cycle. However, crates 0C, 07, 0F was not get configurated at start of next run. Oleg Tsai helped manually configure these crates.
- Owl Shift: Smooth physics run
- high JetPatch trigger rates, Oleg manually configured BTOW crate 1E. Restarted run, JP rates are normal.
- Twice problems when stopping run (EVB[5] [0xF105] died/rebooted; ITPC[23] [0xBF97] died/rebooted)
IV. other items:
- Remind shiftcrew to check more frequently on the critical plots
02/16/2022
I. RHIC Schedule
- Today: 9:00-19:00 APEX (Au, 3.85 GeV/n)
- Tomorrow: 8:00-19:30 Siemens test and Rotator scan?
II. Notable items/recap from past 24 hours: General: Access + Smooth physics (Owl)
- Main issues: David, Prashant, Mike, Tim installed sTGC HV boards to MTD CAEN HV crate, rerouted and reconnected HV and interlock cables. However, sTGC still looks strange in test pedestals and cosmic runs. Experts will check on RDO and others
III. More items from shifts:
- Day shift: Access
- David, Prashant, Mike, Tim installed sTGC HV boards to MTD CAEN HV crate, rerouted and reconnected HV and interlock cables. However, sTGC still looks strange in test pedestals and cosmic runs
- Jeff cleaned up and changed some cables for iTPC DAQ machines
- Evening shift: Physics run started 23:50
- Owl shift: smooth physics run
- 2B quadrant was empty in sTGC (hits/Quadrant) online plots.
- Masked ITPC: Sector 11, RDO 4
IV. Other items?
02/15/2022
I. RHIC Schedule
- Today: 7:00-15:30 maintenance, then collisions for physics
- Tomorrow: 9:00-19:00 APEX (Au, 3.85 GeV/n)
II. Notable items/recap from past 24 hours: General: Access + Smooth physics run
- Main issues:
- sTGC HV GUI lost connection, David and Tim checked sTGC CAEN HV crate, no solution, run without sTGC
- Partial trip of BTOW PMT box 10 West, solved by reapplying HV to this PMT box
III. More items from shifts:
- Day shift: Access
- sTGC bad chamber was checked by Prashanth and Tim, will order parts and replace later
- FST coolant refilled by Xu and Yu
- Wayne fixed issue with BBC_DSM2 and BCW_DSM2, if same problem happens, call trigger experts first, may call Wayne later.
- Evening shift: Access + Smooth physics run
- sTGC HV GUI lost connection, David and Tim checked sTGC CAEN HV crate, no solution, run without sTGC
- Owl shift: smooth physics run
IV. Other items?
02/14/2022
I. RHIC Schedule
- Today: 7:00-15:30 maintenance, then collisions for physics
- Tomorrow: 7:00-15:30 maintenance, then collisions for physics
II. Notable items/recap from past 24 hours:
- General: Smooth physics run
- Main issues:
- Error configuring BBC_DSM2 Node and Error configuring BCW_DSM2 Node. Reboot all did not help. Powercycling the corresponding VME crates and reboot all did not help either. Had to take out BBC_DSM2 and BCW_DSM2 from the Component tree (experts are working on it)
- Tonko masked Crate Id 8 from _any_ configuration, Oleg shows how to manually reconfigure BTOW crate 8, will need to keep watching the performance
III. More items from shifts:
- Day shift: smooth physics run
- BTOW crate 8 config. Err., Tonko: disable à rates too high à keep old procedure
- Online QA plots lost, solved by reset jevp server
- Lost connection of sTGC LV, solved by David T.
- Evening shift: smooth physics run
- iTPC: Sector 4, RDO 3 powercycled
- sTGC LV errors, solved by power-cycle
- BTOW: PMT box 10 west partially tripped, solved by reapply HV
- Owl shift: smooth physics run
- Error configuring BBC_DSM2 Node and Error configuring BCW_DSM2 Node, moved out from the Component tree
IV. Other items?
- Access plan on Monday-Tuesday (8 hrs each day)
- Magnet maintenance: Water Supply Group will be cleaning the strainers in the STAR magnet power supplies during the maintenance on Monday. This involves de-energizing STAR magnets.
- FST: Refill the coolant for FST, need about 30 mins for the fill then 20~30 mins cosmic runs to verify
- others?
02/13/2022
I. RHIC Schedule
II. Notable items/recap from past 24 hours:
- General: Smooth physics run
- Main issues:
- DO had difficulty to reboot CANBUS to solve BTOF LV errors, due to the mistake in password list
- Several EVB errors
- BTOW configuration errors happen many times from crate 8 (Oleg will provide solutions for the shifts to help reduce the down time for the data taking)
III. More items from shifts:
- Day shift: smooth physics run
- BTOW configuration issues solved by restarting run while waste a lot of beam time (experts will provide better solution)
- 2: FOB 2 (R#14:F#81) NOT found -- power-cycle ROB #14
- 2: FOB 2 (R#14:F#81) NOT found -- power-cycle ROB #14
-
- Evening shift: smooth physics run
- Power-cycle BTOF LV
- sTGC "3: FOB 0 (R#07:F#37) NOT found" errors, and we power-cycled ROB #07
- sTGC "4: FOB 3 (R#16:F#94) NOT found" errors, and we power-cycled ROB #16
- "EVB[3] [0xF103] died/rebooted"
- "BTOW: failed crate Ids: 1; BTOW: configuration failed - watch BTOW triggers or restart run."
- Owl shift: smooth physics run
- l4 got stuck at the end of several runs, “EVB[3] [0xF103] died/rebooted”, “EVB[4] [0xF104] died/rebooted
- BTOF LV errors, get difficulty to reboot the CANBUS due to the mistake in password list
- a few BTOW Configuration failed for crate Id 8 and once for crate Id 1
- iTPC RDO S20:1 was masked
Other items?
- Access plan on Monday-Tuesday (8 hrs each day)
- Magnet maintenance: Water Supply Group will be cleaning the strainers in the STAR magnet power supplies during the maintenance on Monday. This involves de-energizing STAR magnets.
- FST: Refill the coolant for FST, need about 30 mins for the fill then 20~30 mins cosmic runs to verify
- others?
02/12/2022
I. RHIC Schedule
II. Notable items/recap from past 24 hours:
- General: smooth physics run
- Main issues:
- Lasers had no respond to controls well. Brightness does not change when pressing control button. Called Alexei.
- East laser was fine for two laser runs, West laser was far from normal brightness
III. More items from shifts:
- Day shift: physics run, short access, physics run
- Issue with L2, solved by reboot trigger
- Jeff fixed the TOF+MTD noise run rate, runs ~14 kHz now.
- 13:00-14:30: David and Prashanth refilled the sTGC n-Pentane
- Evening shift: smooth physics run
- Power-cycle iTPC sector 14, RDO 3 (twice)
- BTOW configuration failed, restarted the run
- "tpc_caen_alive" alarm went off, fixed by following manual
- sTGC “3: FOB 2 (R#11:F63) NOT found”, fixed by power-cycle sTGC LV ROB #11
- Owl shift: smooth physics run
- iTPC RDO S14:3 and S16:3 were masked out, (had error in 2 consequent runs
- FCS error in DAQ, solved by restarted the run twice
- sTGC LV: 2: FOB 3 (R#10:F#58) NOT found -- power-cycle ROB #10 if the problem persists.
Other items?
- Lumi was a little too high than usual during Owl
- sTGC cable(chamber) 9
- Access plan for Monday-Tuesday
02/11/2022
I. RHIC Schedule
- Plan from RHIC
- Collisions for physics now to Sunday
II. Notable items/recap from past 24 hours:
- General: APEX and smooth run
- Main issues:
III. More items from shifts:
- Day shift: physics run at the beginning, then APEX
- - 10:00, Power dip, magnet tripped, lost slow control. Experts solved the problem.
- Evening shift: APEX ended around 22:00, Physics from 22:30
- - sTGC HV cable 9 tripped, ran without this channel
- Owl shift: very smooth physics run
- - TPX 36 config error, solved by powercyle TPC FEEs and Reboot All (Comments: Reboot All is not necessary but no any harm)
IV. Other items?
-
02/10/2022
I. RHIC Schedule
- Plan from RHIC
- 8:00-16:00: LEReC APEX: 3:85 GeV/n Au
- Today 16:00-Tomorrow 16:00: Collision for physics
II. Recap
- General: Rotator study, low lumi run (18:00-4:15am), 7:00 am normal physics run, smooth
- Main issues:
- sTGC: LV unresponsive, solved by short access (18:55).
- sTGC HV cable 9 trip, experts will discuss for a solution today
- Timing during alignment fill, “extra” peak in preshower, ECal and HCal 8 time bins before “right” peak… (Carl Gagliardi), experts will follow up later.
III. More items from shifts
- Day shift: Rotator study
- 12:43: Wayne replaced the monitor for EPD-CR
- Evening shift: low lumi run
- sTGC: LV1 is unresponsive, after a short access, run smoothly for the rest of the shift.
- Owl shift: low lumi run till 4:15am, then normal physics 7:00am
- sTGC HV Cable 9 trip, still open
- BTOW configuration cautions, Rebooting not help but rates and critical plots are good
IV. Others items?
- Battery for the clock replaced
- fast offline data production request for forward detector calibrations with the low lumi runs.
02/09/22
I. RHIC Schedule
- Today's plan:
- 08:00 - 14:00: Rotator study
- 14:00 - 23:59: STAR low luminosity run (for alignment; magnet off, 0.2e11/bunch, 8 hr, 56x56)
- 00:01 -8:00: collision forphysics
- Tomorrow’s plan:
- 8:00-16:00: LEReC APEX: 3.85 GeV/n Au
II. Recap
- Main issue in past 24 hours:
- 14:17: EEMC (ETOW, BTOW) config errors, “GUI is white” seen from SC3 but not seen in control room. (15:12) Dmitry K "Recovery & Restart", turned HV and MAPMT FEEs back on
- sTGC cable 9 HV trip, David fixed it, after 2 hrs run, tripped again, David T. checked on it (see his email). HV current frequent spikes even at a lower voltage may due to the dust in chamber. If this happen again, don’t need to take out sTGC from data taking. Just make a note which channel/cable tripped, i.e. is not at full voltage in the run. Tripped again during OWL, under David T’s control
- 23:44, EPD GUI is saying that the current is too high for most channels, not affect data taking. Rosi pointed out that current values were fine, a problem with GUI, she will follow up with EPD maillist.
- Open issues:
- sTGC cable 9 HV trip, David T. will work on it this after noon, during the low lumi run.
- Day shift: smooth run from the beginning to the end
- Jeff fixed the issue effecting the sTGC critical plots (y-axis range).
- Issues with BTOW and ETOW, Dmitry K worked on it.
- EVB12 was out for a while in a run, back in next run
- sTGC LV issue, fixed by David.
- sTGC cable 9 HV trip. After clearing trip and ramping up the channel tripped again. David advised a temporary solution of manually setting demand voltage to 2K until he fixes it. David fixed the issue. 16:18, same issue came again.
- Evening shift:
- "GUI is white" when trying to access slow controls from SC3. Followed manual section "Recovery & Restart", turned HV and MAPMT FEEs back on – Dmitry Kalinkin
- EPD GUI is saying that the current is too high for most channels, however not affect data taking. Rosi said current values were fine, a problem with GUI, she will follow up with EPD maillist.
- TOF critical errors around 8pm, POWERCYCLE TOF LV. Tried to power cycle at least twice, no luck. Geary instructed us to try one more time as the problematic trays are different.
- Display for EPD computer went off. Michael will work on it in the morning
- Overnight shift: Beam and physics for all time
- Major issues with the sTGC Cable 9 HV supply.
- sTGC cable 9 HV trip, slowly increasing HV by +10V or +5V, reached full, then tripped again, David worked on it.
- BTOW configuration cautions at the start of each run. Rebooting BTOW in Run cotrol or restarting run did not help. Rates and critical plots look OK, keep running.
- TOF ADC critical plot: One run (23040009) had ADC entries at ADC = 15 and ADC = 30 for all channels. Restarting the run helped.
- Relatively many TOF auto-recoveries. Apart from the run mentioned above, no indication of problems in the critical plots.
- Had to powercycle a couple of iTPC RDOs manually which helped.
02/08/22
I. RHIC Schedule
- Collisions for physics today
- Tomorrow’s plan:
08:00 - 14:00 Rotator study
14:00 - 22:00 STAR low luminosity run (for alignment; magnet off)
II. Recap
- Machine development all day yesterday, beam back later in evening shift
- EEMC QA plots having issues during day shift (first note @ 07:54), needed short access after MD (before physics) and replace HVSYS module A
- In overnight shift: ETOW error messages in DAQ for crate 6 -> No success in fixing issue overnight; today’s morning shift advised to power cycle the crate and then issue a master reset; noted: this is the procedure, no need to contact on-call expert for this
- Period coordinator shift changeover Matt -> Zaochen
III. Open issues/status
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006606.html)
02/07/22
I. RHIC Schedule
- Machine development unitl around 17:00, back to physics after
II. Recap
- In terms of sampled luminosity, yesterday was best day so far: 6.1 pb-1 w/ 18.5 DAQ hrs
III. Open issues/status
- Discussed prospects of dedicated aligment/forward cross section runs. Potential to do alignment run Wed and cross section run Fri. Will be discussed during sheduling meeting, etc.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006594.html)
02/05/22
I. RHIC Schedule
- Collisions for physics all weekend
II. Recap
- No beam 6:45 until around ~17:00
- sTGC HV slowly ramped up during shifts; new temporary ramping procedure for cable 9 (see overnight shift log and https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006575.html)
- Warning from TPC computer (chaplin): running low on GPU memory, usage at 99% (18:01)
- Two TOF FM2 red alarms in overnight shift. FM2 plot shows sudden spikes (04:47)
III. Open issues/status
- Discussed on making the new temporary sTGC HV ramp the same for entire sTGC (i.e., automated)
- BTOW crate 8 error - Noted shifts should just stop and restart run
- Note on various reported high rates in logs, at beginning of fill this is expected, Jeff may change the baseline rate used to determine color-coding in monitor
- Need to follow up with Wayne on TPC computer (chaplin) GPU memory usage warning
- Alexie following up on current laser issue and TOF FM2 alarm
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006577.html)
02/04/22
I. RHIC Schedule
- MCR access until 12:00, about 2 hrs to get back to physics
II. Recap
- Physics running started around 22:50 due to long injection tuning
- Lots of errors overnight (see below), however no major downtime
- sTGC HV trip for plane 2 cable 9. Powercycled the HV - Investigation still ongoing
- sTGC LV (23:31) powercycled after getting following error: “ROB-15: FOB 1 (R#15:F#86) NOT found -- power-cycle ROB #15 if the problem persists” - This is also a correction note w.r.t. minutes sent durectly to star-ops
III. Open issues/status
- Tentative plan to do rotator measurements/corrections Monday during day
- When beam comes back today, add BBC back into run for testing
- Investigation of sTGC HV ongoing.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006569.html)
02/03/22
I. RHIC Schedule
- APEX 9:00 - 17:00. Followed by physics.
- Tomorrow: CeC access ~7:00 for 3 or 4 hrs; TBD if machine development with follow, or back to physics
II. Recap
- All contingent activities for APEX running were solved, so APEX will run today ~8hrs (started at 9)
- Tomorrow’s afternoon activities still TBD
- Identified incorrect terminating resistor on injection kicker (40 -> 25 ohm); will fix during tomorrow’s access, should improve performance
- TOF HV turned off during evening shift, values frozen in GUI indicated “on” (fixed with "IOC reset FULL” and ramping HV)
III. Open issues/status
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006560.html)
02/02/22
I. RHIC Schedule
- 07:00 to 11:00: RHIC access -- CeC clean room, Booster access for foil, back to physics
- Thur: 09:00-17:00: APEX (if ready). Followed by physics.
- Fri: 07:00-12:00(?): RHIC access injection kicker module replacement + CeC access; imperfection study after(TBD)
II. Recap
- Two (last minute) Vernier scans in day and evening shifts - Will update crew checklist to note what to do if this happens again
- Booster RF power amplifier leak, extended evening shift fill, no beam after 2:55
- Major issue with DAQ starting @ 18:20. Jeff stepped in and was able to eventually pinpoint the problem. Noted computer disc may need to be changed - Done by Tonko
- Alarm for VME crates, lost communication and QA plots. David worked on this for awhile. Decided access is needed (23:43). Accessed today at 9:30 - eveything should be OK now
III. Open issues/status
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006549.html)
02/01/22
I. RHIC Schedule
- Collisions for physics
- Tomorrow: likely APEX from 9:00 - 17:00
II. Recap
- No beam from ~7:00 -~19:30 yesterday
- Smooth running once beam returned
- TOF gas changes twice in past 24 hrs
- JH will bring up request for low intensity run for forward x-sec and alignment runs @ RHIC coordination meeting; alignment run will need magnet off, warmer ambient temperatures should be considered when doing this
- David will increase current alarm for sTGC from 200 micro amps to 300
III. Open issues/status
- ETOF HV proceedure has been changed, shifters should change to OFF for all states
- APEX runing for tomorrow contingent on a few ongoing RHIC/AGS activities. If postponed, tomorrow will be physics, and APEX will run Thursday
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-February/006531.html)
01/31/22
I. RHIC Schedule
II. Recap
- Luminosity highest in recent fills; improvement in RHIC polarization (~50%)
- Issues at linac, unclear when will be fixed
- Afternoon access requests: CeC 4 hrs, Jet 1/2 hr; may be earlier in morning -> Will hear back from MCR/Vincent et al. shortly on schedule
- Relatively smooth running over past 24 hours
- Sampled luminosity: 5.1 pb-1 w/ 15.5 DAQ hrs; today 1.5 pb-1
- Forward x-sec strawman trigger config test with low lumi in run 23030047 - will request fastoffline production
III. Open issues/status
- Day shift will take a laser run ASAP to see if they can get West laser working; if problems Alexie may go in to make some adjustments
- David will make changes to alarm handler (see FCS HCal PS issue in elog in overnight shift)
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006515.html)
01/30/22
I. RHIC Schedule
II. Recap
- Very smooth running over past 24 hours
- Sampled luminosity (note with updated x-sec): 4.6 pb-1 w/ 16.5 DAQ hrs; today 2.2 pb-1
- Forward x-sec strawman trigger config test in Run 23029038; todays test at end of fill should aim to have MCR displace beams to achieve ZDC rate between 10-20k (~10 min); request offline to produce runs ASAP
III. Open issues/status
- Will test forward x-sec strawman trigger config today with low lumi beams at end of next fill
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006495.html)
01/29/22
I. RHIC Schedule
- Collisions for phyics over weekend
II. Recap
- Weather conditions very bad due to winter storm. So far no major issues reported
- Sample luminosity: yesterday: ~4 pb-1 w/ 17 DAQ hrs; today: ~1 pb-1
- Updated Vernier scan x-sec: 2.06 mb, all plots will be updated with new value
- Very high JP1, JP2, ETH1, … rates noted in Run 23028049, had to power cycle and reconfigure EEMC (with experts), also next shift noticed high JP1 rates in one run
- Ground fault of substation 6A @ 7:22 (another at 10 AM), addressed promptly by CAS watch
- Forward x-section measurement strawman trigger configuration provided by Jeff, tested in Run 23028062 (in between fills). Test today @ end of fill - 5 min needed (note: another test tomorrow with beam steering for low lumi.)
III. Open issues/status
- Forward x-section measurement strawman trigger configuration will be tested today at end of fill. Should also be tested tomorrow with a lower luminosity
- Shuai: will update dimuon timing window trigger configuration
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006485.html)
01/28/22
I. RHIC Schedule
- Collisions for physics over weekend
II. Recap
- ZDC polarimetry done during rotator test
- FCS gain tests after rotator test (some thresholds changed after)
- Smooth running when beam came back during overnight shift
- sTGC settings changed in evening shift, some “spikes” seen in QA plots (02:33)
- New PROM check run added by Tonko to be done before; will update detector readiness checklist
- Vernier scan was performed, preliminary cross section is lower than Run17
- Carl proposed we take out dimuon trigger for first 1.5 hrs of a fill, and agreement was reached. This has to be done manually by shifters. Jeff will conveniently move where the SL can do this on the computer to be right next to laser run trigger.
III. Open issues/status
- We should start planning the low luminosity runs (as early as next week). Jeff is working on the configuration, once ready we will ask MCR to steer the beams to lower rates and take a run with this configuration at the end of a fill
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006475.html)
01/27/22
I. RHIC Schedule
- CeC will take 4 hrs of dedicated time
- 1/2 hr coupling test
- Rotator test after CeC/coupling test (STAR will do zdc polarimetry runs as we did last rotator scan, one before and after rotator ramp, we will also do a FCS gain check at the end of store for ~30 min.)
- Back to physics
II. Recap
- Blue injection efficiency is getting worse every fill, reducing luminosit
- AGS polarization also seems to also be decreasing
- Smooth running after maintenance, albeit with lower intensity
- Various maintenance day activities compled yesterday, see elog for details
-
III. Open issues/status
- Jeff/Tonko: doing a few checks today, but may begin to have a daily PROM check run
- Daniel: sTGC will have SL update configurations for pedestals
- Hank: still investigating BBC crate issue
- Xu: Current in problematic FST module seems to be stabilizing. Will keep an eye on for next 24 hrs, and may start to slowly ramp HV back up
- David: Now has updated manuals ready for sTGC, also compressor monitoring
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006456.html)
01/26/22
I. RHIC Schedule
- Maintenance from 7 AM until around 3 (likely till 5 or so) PM, back to collisions for physics
- Tomorrow: 4 hrs for CeC starting after natural end of a fill, some time between 9-12
II. Recap
- 3 good fills yesterday, one fill this morning with reduced luminosity
- Yesterday: sampled lumi 4.1 pb-1 w/ 17 DAQ hrs; Today: 1.2 pb-1
- Crew enable BBC readout on evening shift as requested by Hank
- Overnight L2 timeout error was from MXQ, not BBC. Will work on making this more explicit for shifters
- New items during maintenance time:
Hank/Jeff, will need control system to test new tier 1 file, will convene around 15:00 to start
Akio, work on EPD
III. Open issues/status
- Manual for sTGC will be updated with current procedures
- Explicity logging for L2 timeouts to aid shifte
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006438.html)
01/25/22
I. RHIC Schedule
- Collisions for physics today
- Tomorrow maintenance from 7 AM until around 3 PM, likely to extend longer
II. Recap
- Consistent 2+1 ongoing fills since last meeting; Yesterday: 4 pb-1 w/ 15.5 DAQ hrs (best day for int. lumi/hr); today: ~1.5 pb-1
- FCS taken out of run during day shift, put back in during overnight shift
- Unusual fill pattern for sTGC shift crew plots, Jeff and Daniel B we notified, not immediately obvious is anything is wrong but Jeff will followup, and Daniel may change plotting scales
- One more (noted) online plot crash during evening shift
- Several activities planned for tomorrow's maintenance day. See detaIls in star-ops log linked below
- Daniel requested a special ~30 min run for sTGC at end of fill at 3:25 PM today, will coordinate with PC & SL
III. Open issues/status
- FST still tripping, experts changing trip limits and HV, expect to be OK after tomorrow's maintenance
- Manual for sTGC will be updated with current procedures
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006406.html)
01/24/22
I. RHIC Schedule
- Collisions for physics
- Maintenance schedule for Wed. (8 hrs?, depends on yellow rotators TBD)
II. Recap
- Consistent stores in terms of luminosity (slight improvement w.r.t. yesterday’s report); two beam dumps with ~3+~6 hr downtimes
- Yesterday: 3.2 pb-1 & 13 DAQ hrs; today: ~1.2 pb-1
- Some continuing FST HV trips in evening shift; Xu adjusted limits and HV again
- Fire department had to check a (false) smoke alarm in the cave at 22:30 (during downtime so no interruptions to data-taking)
- EEMC, TPC/iTPC out for a few runs during overnight shift; in both cases experts were consulted and eventually came back online (fixed with: rebooted crate 90, “reboot all” in run control, respectively)
- GMT bottle B has been replaced (18:14)
- Online plots crashed twice over 24 hrs
- Noted that a full rack of TPC power supplies have browned out a few times over the past week (TPC/iTPC sectors 13,14,15). This is an open issue and Wed. during access we should check it out
III. Open issues/status
- TPC power supply rack experiencing brown-outs. Needs to be investigated during next access
- FST: plan to access Wed to swap boards (see details above and previous 10 am meeting minutes)
- Jeff is looking into the online plot crashes
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006388.html)
01/23/22
I. RHIC Schedule
II. Recap
- Generally good 24 hours of data taking with no major issues. 3.3 pb-1 & 15 DAQ hrs.
- Besides low luminosity fill early yesterday following fills look better (2+1 ongoing fills)
- Already accumulated ~1.5 pb-1 this morning
- FST Disk 1 Module 12 outer sector: ISEG module board trips due to current fluctuation, current limit increased and HV decreased. ABC board needs to be swapped next access - Xu Sun (see entries 8:34 and 09:48 from today, and overnight shift starting from 06:46)
- Online plots crashed again this morning. Shift crew have noted in the log book where to find instructions to restart
- >2000k errors in VC1 in one particular run last evening shift (54). Not noted in log book but assumed shift followed instructions. Was noted during meeting just for information
III. Open issues/status
- Alexie will work on addressing gas alarms
- See FST issue above, will wait till tomorrow to formulate a plan on when to access
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006380.html)
01/22/22
I. RHIC Schedule
- Collisions for physics today and tomorrow
II. Recap
- CeC all day starting from 8 AM
- Smooth running overnight when beam came back after CeC (1:30 AM)
- Beam intensity low in first fill of the day
- Few iTPC RDOs masked out during cosmics and overnight runs
- David called and had VME slow controls reset (13:11)
III. Open issues/status
- JH will communicate with accelerator to use a 280k ZDC rate threshold to optimize data taking
- Increased TPC dead time was noticed for an early run by Jeff (fixed by itself next run). SL can check for this under the “Current rates” tab on the DAQ page. If any noticeable changes in rates/dead times during run -> restart run
- Transient alarm from TOF pressure-transmitter tof_gas_PT-2 (yellow warning, low pressure) - Alexie said he will change today
- The online plots may crash and not update to latest run, SL needs to check with shift crew each run early
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006374.html)
01/21/22
I. RHIC Schedule
- Cec Au beam dev. will happen today from 8 AM - 9 AM. End time may shift either way by one or two hours (likely end around 8 PM)
- Collisions for physics after CeC and over the weekend
II. Recap
- Generally last 24 hours have been good, new run procedure and new beam sequence has nicely improved data taking (~17 DAQ hrs and sampled lumi = 4 pb^-1)
- Jeff implemented the new production_pp500 configuration disabling BBC readout during day shift
- Short access in between fills to check sTGC blower (~6 PM). It was running fine and also so was the compressor. Alexie also tuned the laser
- The alarm handler is not connecting to the VME crates. That is something to be checked when there is no beam (see entry at 23:57)
- Beam dump @ 01:13 overnight; no beam for rest of shift
- Slight improvment in polarization and also new beam sequence with imediate first beta* squueze at begginging of fill; second at ~3hrs
- Unclear when Siemens will (and if) come back online; at least another week of investigation
III. Open issues/status
- David/Alexie will try to add more monitors for TPC to aid shifters
- Jeff is working on the error logs to aid shifters remedy the configuration error messages that occurred over last few days; suggest maybe to run pedPhys run at end of extended downtime to preemptively identify any issues instead of during physics run.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006369.html)
01/20/22
I. RHIC Schedule
- Physics running till 8 AM tomorrow then CeC Au beam devellopment until 9 PM tomorrow night
II. Recap
- Beam dump @ 8 AM; nothing for rest of morning shift due to RHIC issues with rotator scan; took calibration runs
- Polarimetry runs start at 19:01, and continue until 22:52 (three 3M runs per setting [3x5])
- Delay in run start due to “l4Disp” in component tree; recovered automatically in 7 min. (00:49) [not critical]
- High temp. in sTGC air blower; sTGC removed from one run; was already in recovery after previous shift notified Prashanth (00:56)
- Few TPC RDOs masked out during shift
III. Open issues/status
- sTGC reference plots now available
- Today: Short access (~5 min.) in between fills to check sTGC air blower; current scheduled beam dump is 6 PM
- Follow-up on BBC crate/L2 timeout discussion. Jeff will set new configuration to not read out BBC to remove this issue
- Missing HV sections from TPC sectors 8,9,10,11 from yesterdays runs, pointed out by Flemming, will have some followup with potential plots/alarms from Flemming/Alexie
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006352.html)
01/19/22
I. RHIC Schedule
- Rotator scan until 4 PM
- Physics evening/overnight and tomorrow
II. Recap
- Beam dumps at 8:00 and 10:55 (beam resumption at 9:40, 16:00)
- L2 error "More than 2000 timeouts... Suppressing further timeout error messages. STOP the run!” (16:15)
- Magnet trip 17:22, followed by alarms in sTGC and TOF; fixed by David and Isaac, respectively (see log entry 18:05). EEMC w/ Radstone problem found by Will (see entry at 20:37); Troubles ramping until filter in magnet cooling tower replaced by CAS (20:20) - Following up on periodicity/consistency of filter cleaning schedule
- In parallel to magnet problem, MCR reports cryo problems. No beam till end of shift
- FST HV RDO issue at end of shift (00:12) ("Failure code 2”) while turning on), procedure followed to restore; fixed by expert
- Problems with FCS ("FCS:DEP09:1 failed”); runs until 2:39 had no FCS until fixed by Tonko (see entry at 2:39); also around 3:53
- Run 23019011 - scalar rate of BBC, VPD, ZDC and EPD are much higher than expected (plot in entry at 2:16) - Jeff looking into this (possible mis-config.)
- TPC GUI computer frozen (5:18); had to restart computer ; Possible solution to be investigated - Wayne
III. Open issues/status
- Optimization of fills/beta squeezes - JH will discuss with Vincent
- TPC GUI computer freezing will be looked at by Wayne
- Large scaler rates for Run 23019011 will be checked by Jeff
- sTGC critical plots for shifters
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006347.html)
01/18/22
I. RHIC Schedule
II. Recap
- Tim fixed the compressor for the sTGC-FCS air blower after the meeting yesterday (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63743)
- 19:30 Magnet trip
- DAQ error "Trigger Bits corrupted: (0-5) set=0x00000000 read=0x00000040 addr=0x512114". This is currently unknown. Jeff is looking into it.
- BBC taken out last night. The BBC TAC difference was empty since. Akio contends that this plot is empty whenever the BBC is out, so it's not indicative of a problem. Akio is changing his code so that it might be an indicator in the
- 4 low lumi fills (2*10^10 bunch intensity -- 56 with STAR magnet off and 28 bunches) requested for later this week. Nominal schedule is 9am Saturday to 9am Sunday. Refer to Elke's slides for more detailed and current information
III. Open issues/status
- We need some clarification on the BBC crate issue. I thought this TAC difference plot was important. If not, the instructions need to be updated.
- Short instructions for fixing holes in the BEMC need to be updated. They were proven wrong last night.
- sTGC still has no critical plot instructions.
- AH GUI still mislables the air blowers.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006317.html)
01/17/22
I. RHIC Schedule
II. Recap
- sTGC air blower AC compressor needs a fix, whenever there is access next time. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63692). Confusingly this is under the eTOF in the AH, but this one is serving the sTGC and FCS. The eTOF shares the TPC blower this year.
- Work on the air blower is happening this morning
- We can probably run without the error blower. The sTGC LV will drop if it hits a limit (currently 80F). Without the compressor we're still generally under this limit, but we're close (~77F).
- Many issues with the trigger, as we discussed yesterday. bbc errors from the daqlog will be suppressed so that the operators should never see those ErrCnt messages. Not all events with BBC errors make it to L2, so not all are counted for the 200-error run stop limit. That's how we've been going beyond 200. Crews do not need to stop runs for this. Hank might get Chris to increase this limit from 200.
- BBC issue might affect analyses that use the earliest TAC information. The TAC difference is from the earliest TAC, so the information is there. This should affect a very small fraction of events, but it might need to be considered by future analyzers (or maybe even the production?)
III. Open issues/status
- air blower status
- sTGC critical plot information
- rotator study (schedule not fixed)
- Detector readiness checklist needs updating with new BEMC instructions and new BBC instructions
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006326.html)
01/16/22
I. RHIC Schedule
II. Recap
- Updates on BBC instructions "Slow DMA error". Runs are going beyond 200 errors, which is a surprise.
- TPC ran at near 100% dead from about 2200-0030. TThe node causing the problems was TPX35. We tried removing the node and the errors went away. I then tried masking the boards in these run (sector 21 RB 5&6 and sector 22 RB 5&6 respectively). The errors persisted whichever boards were masked. We then rebooted the tpx35 computer and the errors stopped. (Summary Report - Night Shift). We may need mass computer reboots during some down time (only ~10min)
- Scalar board 6 is having unknown issues. It was out from run 29 yesterday to run 44. It's out again today.
- We'll get some more detailed instructions on when to reapply voltage to the BEMC
III. Open issues/status
- sTGC shift crew manual
- Instructions need to be updated for BEMC, BBC crate, and the detector readiness checklist.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006303.html)
01/15/22
I. RHIC Schedule
II. Recap
- Power dip in the afternoon, it took some time to get everything back.
- Several accesses yesterday. Tim had to go in late to work on the TCIM.
- One very-short fill overnight. Poor quality beam. High emittance and low polarization.
- The ZDC scalar rate is not correct, while MCR use ZDC rate to tune the beam. J.H Lee has called MCR to use BBC rate instead of ZDC rate to tune the beam. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63575)
- This ZDC problem came from a threshold change on a NIM crate after the power dip that depressed the values ~50% (can be seen by comparing to the BBC values).
- RICH scalars were out after the dip.
- The sTGC crate issue seems to have been solved.
III. Open issues/status
- sTGC shift crew manual
- Still talk of 2 beta squeezes
- Plans for low lumi runs next week (Tuesday?)
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006286.html)
01/14/22
I. RHIC Schedule
II. Recap
- Injection plan is tentatively for midnight. Until then zero-field cosmics.
- We're turning off the magnet now (ramping to zero current, no state change). Will be turned on ~midnight.
- Access can be done. Tim went in yesterday to replace a flash card. Today the plan is Tim will replace an sTGC raspberry pi, Christian will replace two failed DEP boards for the FCS, and Prashanth will go in for a quick inspection.
- Oleg will switch the ECal and work on the system after the DEP board replacement.
- Jeff fixed instability with Configuration speedup. There should be a significant speedup for run starts now.
- ^ We will change to 30 minute runs.
- Tonko made a change in the ITPC processing in case of auto-recovery failure:
Before) If the auto-recovery fails 4 times I force-stop the run with an appropriate message.
Now) If the auto-recovery fails 4 times I raise iTPC BUSY with an appropriate message but I DON'T force-stop the run. In this case the forward program continues and it gives the Shiftcrew some time to figure things out.
Crews are not expected to clear this busy.
- A brown out this morning seems to have taken out a few sectors in the TPC and nothing else.
- Carl has written detector requirements for the triggers in case, thus no detector -> disabled triggers. This email has been circulated to the trigger board and (will be) to the ops list. If you have expertise or opinions, review this.
- Elke wants detector experts to get ready for the low luminosity runs. This is just a reminder.
III. Open issues/status
- sTGC shift crew manual
- sTGC LV still has some issues
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006275.html)
01/13/22
I. RHIC Schedule
II. Recap
- The collider will switch from the Siemens to the Westinghouse injector, which will reduce the quality of our fills.
- Took out BBC from the triggers, as we always reached 200 BBC errors "Slow DMA error" (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63341) -- BBC not configured for the next 12 hours
- Multiple problems with the FCS and FST stopping runs
- We need some way to communicate to the shift leaders about what to do after repeated errors.
- Jeff will work on run control during CeC
- We should take advantage of the low luminosity fills from the Westinghouse generator to do some work which should be detailed later.
- Tim will go in to check the LED control box for HCAL after the beam dump
III. Open issues/status
- No injection into RHIC, no beam
- Daniel wants to do a few quick runs for the sTGC (one with and one without beam)
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006267.html)
01/12/22
I. RHIC Schedule
II. Recap
- Access right now (until 11am) -> 5 hours of CeC (4pm) -> Physics
- BCE had to be power cycled because the 0x1d000000 board failed to configure.
- The BBC hit 200 errors and was out of the trigger for the rest of the night.
- Hongwei reported that he thinks he fixed the issue of l4 hanging run stop requests.
- Jeff is working on making run start faster. Perhaps we'll switch to 30 min runs tomorrow.
- Carl reported that the forward trigger is reading out the MTD and TOF. TOF mult may be useful, but the MTD likely is not. That should be removed.
III. Open issues/status
- Still no sTGC critical plot instructions
- 30-minute runs soon
- We're currently at ~ 80% of requested luminosity from RHIC
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006258.html)
01/11/22
I. RHIC Schedule
II. Recap
III. Open issues/status
- sTGC LV is constantly disconnecting. The solution to this is still not known.
- We need sTGC critical shift plot instructions for the shift crew
- We need higher luminosity
- Daniel's 2 5-minute sTGC runs
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006245.html)
01/10/22
I. RHIC Schedule
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Communication with MCR about luminosity should probably be about ZDC and (without killer bit) rates. MCR has more rates to look at, but we want the communication to be clear. Angelika will make something like Jamie's singles-corrected value for them to look at too.
- Daniel wants to do a short sTGC test today - only 5 min run x2
- Tonko recoded the ETOF TCD so that it fires the ETOF whenever TOF is in the run and gets a trigger.
- MTD bias scan: 23009043 (13.1 kV), 23009044 (13.2 kV), 23009045 (13.3 kV), 23009046 (13.4 kV), 23009047 (13.5 kV), 23009048 (13.6 kV), 23009049 (12.8 kV).
- Jeff and Tonko are working on speeding up the DAQ. Jeff will spend a few hours during the CeC today about this.
- We've had a lot of L4 crashes at run stop. The run won't actually stop until L4 is rebooted. Some events might be too big for L4 and the buffer may need an increase. Jeff and Hongwei are working on this.
III. Open issues/status
- sTGC LV is constantly disconnecting. The solution to this is still not known.
- We need to cull some QA plots
- We need sTGC plots in the critical shift plots (this is underway)
- We need sTGC and FST critical shift plot instrucitons for the shift crew
- We need higher luminosity
- l4 crashing at run end
- Daniel's 2 5-minute sTGC runs
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006222.html)
01/09/22
I. RHIC Schedule
- Physics until 0800 Monday.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Tim replaced a control card for sTGC LV crate #2 during access at the end of a fill. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62992)
- MCR had some issues leading to low luminosity fills and they haven't really managed to hit the luminosity they had a few days ago.
- MCR might be able to give us two beta squeezes/fill.
- sTGC LV lost communication frequently. Every time when we need to change the status we will have to restart the sTGC LV IOC and re-build connection. But the communication would be lost again within 5-10 minutes. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=63042). This seems to be ongoing. It isn't stopping data taking, but it is an inconvenience for the shift crew. Once Tim thinks he understands it the fix will likely require an access.
- Bill has proposed a sort of triangle cut to remove high-rapidity clusters in the TPC to help the DAQ. The number of primary tracks looks reasonable, but the number of globals is huge. He will send out a document on this to be discussed during the triggerboard meeting (Monday 1pm). It may need some additional information from the TPC group and the spin PWG.
- The trigger group has implemented code to skip these problematic BBC issues. This should ease the issues it causes for STAR. They're still working on the source.
- We have a lot of outdated QA plots. If you are/were a subsystem expert please check (see https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006211.html)
- Tuesday will be cold. This might be a concern for the sTGC gas vent.
III. Open issues/status
- sTGC LV is constantly disconnecting
- We need to cull some QA plots
- We need higher luminosity
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006204.html)
01/08/22
I. RHIC Schedule
- Physics until 0800 Monday.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Yesterday MCR desteered the beams to check how much of our background was coming from single beams. Angelika: "The result from our quick separation test earlier today was that the BBC singles rates came down to 25-25 kHz with full separation (1.8mm). This corresponds to 0.24-0.32% of the singles rates at full overlap. This does not look like background (consistent with it not responding to collimator settings) and appears to be associated with collisions."
- ^ Thus we cannot collimate out the background. There is ongoing discussion why the BBC singles rates are higher than in 2017.
- Special FCS gain calibration runs taken yesterday afternoon (23007067, 68, 69, 75, and 76). We've requested production for these.
- We asked RHIC a few days ago to reduce the luminosity ~20% to ~250k ZDC and rates. After this meeting JH requested we go back up to ~300k. We plan on asking for ~20% higher again in the future.
- We would like 2 beta squeezes/fill. JH will ask about this.
- Jeff set approximate scalar ranges for the DAQ monitoring page. Bright red means the scalar rate for the trigger is too high. Black means the scalar rate for the trigger is about right. Brown means the scalar rate for the trigger is too low. These are only preliminary numbers. (https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62904)
- Geary would like to do an HV scan for the MTD. This would comprise 6 20-min runs. These are normal physics runs, they'll just have different MTD biases.
- It looks like any issues with the TPC automatic restart from Jeff and Tonko were ironed out. This will save the last DAQ restart from clusters of runs that cannot start because of a TPC RDO.
- Hongwei fixed some issues that looked like L4 was stopping the run. Jeff is still working on an issue where a window will pop up saying "configuration not sent" (I think the quote is right) after everything configures and will require a run restart.
III. Open issues/status
- FCS end-of-fill runs
- 2 beta squeezes/fill request
- We will start requesting more luminosity
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006174.html)
01/07/22
I. RHIC Schedule
- Physics until 0800 Monday.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- MCR will do some background tests by unsteering the beam at the end of the current fill
- We will do some FCS gain scans on either side of this desteering
- First fill of the night had about as high luminosity as we have seen this run
- Several problems occurred at once. At flat top the TPC cathode
wouldn't come up, the sTGC had connection errors, and the EEMC GUI went
white. It took a long time (1.5h?) to get the first two back. The EEMC
wasn't back in runs for ~5h.
- VPD got new TAC offsets. Jeff updated the trigger IDs (
https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62817 +
https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62820(5am) "
- Tonko called in and made us enable all the triggers, more than
half of them were not enabled since previous shifts." -- I suspect this is
from triggers being taken out for VPD TACs
- Daniel suggests we change the max TAC for the VPD. There will be an
email about this. There also seems to be an issue with ch14 on the west
VPD. This will be masked out.
- Shuai updated the MTD timing window after the VPD was changed. We
should change the trigger ID.
III. Open issues/status
- RHIC desteer beams to check background
- FCS end-of-fill runs
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006174.html)
01/06/22
I. RHIC Schedule
- CeC from now until they finish in the early afternoon. After that there are some beam tests and another hour-long access (~5 or 6pm). Then the focus is on physics until Monday morning.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- No beam since yesterday. Yellow cavity issue overnight -> CeC took over starting around midnight.
- Cosmic data taken around midnight was triggering at an extremely high rate due to TOF sector 3. Hank wonders if this is due to the BBC-MIX CPU swap.
- Jeff and Tonko implemented the TPC autorecovery
- Carl has a proposal for taking four 30-minute FCS runs at the end of a fill. Hopefully we can do this tomorrow.
- We will get a new configuration for Tonko to consistently work on the TPC FEEs during the day
III. Open issues/status
- No beam = no VPD timing, no test of BBC crate, no test of TPC autorecovery, etc.
- After the splitter was taken out and put back in the VPD pedestals look okay, but they continue to be very wide. This is true on both E/W. It's not known why.
- FCS end-of-fill runs + configuration
- no-TPC configuration for Tonko
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006154.html)
01/05/22
I. RHIC Schedule
- After access we go into physics until 0800 tomorrow morning. 0800-1600 tomorrow is CeC.
- Long access from 0700-1700 today.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- 11:15 yesterday access. Christian disconnected DSMI boards. Prashanth checked sTGC gas lines. I think we can do a bit better at people who come in for access writing shift entries.
- MCR called and said that they have not received the BBC delayed signals since Sunday. Tim will replace a PS on a nim crate tomorrow. This doesn't need access. I called MCR to let them know that they should get them back then.
- We had issues with the BBC crate not being configured after TRG reboot/reboot all.
- The CPUs for the BBC and MIX crates were swapped. 1M events were taken without error (w/out beam either). If the CPU is the culprit we'll start seeing the same issues with the MIX crate.
- Christian tested the VPD splitter. It was found to be fine. Hopefully just reconnecting solves the problem and we can redo the timing.
- Christian did not have time to look at the eTOF cable while the poletip was open, so this problem is not fixed.
- sTGC HV was not put on an NPS. If there is a parasitic access opportunity Tim can do this quickly, but we won't request an access.
- As a proof of principle Tim was able to reprogram 2 TPC RDOs, so the test discussed yesterday was a success.
- A cable for the TPC gating grid was found to be disconnected. This was reconnected and things can go back to normal in run control.
- The sTGC gas vent on the roof had partly frozen over. Prashanth is working on a solution, but we may see sTGC gas issues in cold weather.
III. Open issues/status
- We're watching the BBC and MIX crates for issues.
- The VPD will need new runs to reset the timing
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006140.html)
01/04/22
I. RHIC Schedule
- Schedule: physics beam until 0700 tomorrow. Access from 0700-1700 (01/05). Physics from 1700(01/05)-0800(01/06). CeC from 0800(01/06)-1600(01/06)
- Long (10h -- 0700-1700) access tomorrow. Check PDF attached to (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006107.html) for the list of tasks.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Beam energy change from 254.87 to 254.21 yesterday
- Long (10h) access planned tomorrow.
III. Open issues/status
- Today we will have a 15min access for Christian to disconnect the DSMIs from the backplane of the BBC crate and for Prashanth to inspect the sTGC gas lines.
- Jiangyong will cover day shift for the next few days followed (possibly) by Ben Schweid. Next week is unclear.
- Tomorrow's 10am meeting will, instead, be held at 5pm BNL time.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006107.html)
01/03/22
I. RHIC Schedule
- 0900-1700: CeC
- 1700 (Mon)-1600 (Tues): physics beam
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Beam energy change from 254.87 to 254.21 yesterday
- Long (10h) access planned on Wednesday. This includes opening the East poletip. If you would like to enter please respond to the star-ops message.
III. Open issues/status
- VPD max TAC not changed since yesterday
- The Trigger group needs a 1 hour access for Christian to swap
processors for the BBC and MIX crates. This should happen tomorrow after a
beam dump.
- VMEs lost connection (white in GUIs) -- This requires the main CANBUS
to be restarted in the IR. This needs a 10 minute access for Tim to go in
and turn it off and on. We'll wait until right after CeC (maybe 5:30?) or
any other available time today. In the future (Wednesday?) we need an NPS
on this.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006078.html)
01/02/22
I. RHIC Schedule
- 0800-1600: injector polarization work + physics
- 1600-0800: physics beam
- 0800-1600 (monday 1/3) CeC
- Beam energy change from 254.87 to 254.21 planned today
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Beam energy change from 254.87 to 254.21 planned today
III. Open issues/status
- Access yesterday at noon. sTGC was fixed by a crate powercycle (will be put on NPS in next major access) while DSM 0x17 was replaced
- Lost a lot of time from beam aborts last night
- Clusters of L2 issues happened again. Trigger group is looking into this again.
- TOF gas freon bottle changeover was neglected for several hours.
- Carl noted that the fast offline is just looking at the StPhysics stream (MinBias and HighTower) and not at jet events or the forward stream. I'll contact them.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006057.html)
01/01/22
I. RHIC Schedule
- 0800-1600: injector polarization work + physics
- 1600-0800: physics beam
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- After midnight (year change) the DAQ monitor messages, the JEVP plots, and all of the scalar algorithms went down. The DAQ monitor messages came back with Tonko ~6am and the servers for the plots weren't restarted until ~7:45am. Jamie messaged Jinlong about the scalars.
- BBC DSM board stopped many runs, even after yesterday's access to replace a DSM board.
- sTGC lost connection to a HV board, this needs access.
- Access at 12pm today for both systems
- Typically alarms are masked when detectors are ramping. Given the recent issues of HV going down for both the sTGC and TOF without alarms David has proposed removing these masks.
- Tonko needs a procedure every day to take time to bring back masked RDOs. He also needs accesses periodically to fix ones that cannot be brought back via software.
III. Open issues/status
- Access today at 12pm to replace BBC crate 0x17 board and check sTGC HV board
- Tonko's time to bring back RDOs. In the short term we'll just stop the run for him to do this. In the long term we'll get a special configuration for forward measurements not including TPC.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2022-January/006066.html)
12/31/21
I. RHIC Schedule
- 0800-1600 - injector polarization work, 1600-0800 collision for physics.
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- BBC DSM board issue was a really bad problem starting yesterday
afternoon. We'll plan on a 15-minute access when Akio/Christian are ready
to go in and replace the board. This can be done in the 0800-1600 block. It
looks like it will be 1:15pm.
- Many TPC errors hurt running
- L4 had a few issues that Hongwei fixed.
- We're planning a beam energy change (254 GeV beams) for better
polarization perhaps on Tuesday (01/04).
- Polarization: B~40% Y ~50% until some time in the overnight shift when
blue suddenly dropped to 11% (
https://online.star.bnl.gov/apps/shiftLog/logForEntry.jsp?ID=62215).
This is primarily due to them changing the fill pattern at the end of the
fill, and was not actually representative of the polarization. Elke has
mentioned that the offline analysis isn't nearly so bad (~40%?)
III. Open issues/status
- Access at 12:30 for the BBC VME crate
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006046.html)
12/30/21
I. RHIC Schedule
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- iTPC caused many issues last night
- BBC DSM board stopped one run, but not many like before.
- Several hours of sTGC data were lost from confusion on shift.
III. Open issues/status
- Does iTPC need some change so that it doesn't stop so many runs in preparation and doesn't hang the system?
- BBC DSM board issue is hard to understand, but trigger group may replace the board or even the CME CPU
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006028.html)
12/29/21
I. RHIC Schedule
- Work on polarimetry in the afternoon (~4pm).
- Physics at night (~midnight).
II. Recap
- production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
- zdcPolarimetry_2022: trg+daq
- CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
- Switched to 45 min runs
- Laser back to every 4 hours (from 2h cycle) since the TPC methane pressure back to (or slightly higher than) normal.
III. Tasks for commissioning
- sTGC voltage scan planned for tomorrow morning when Daniel and Tonko are both awake
IV. Open issues/status
- L2 (which is really the BBC DSM board) stopped many runs in evening and overnight. Hank + trigger group will look into it.
- sTGC caused many issues in the evening perhaps related to a loss of connection to a raspberry pi
- crate 7, board 4 of the BTOW complains in run control. This can be ignored, but Tonko will stop it to avoid confusion.
- L4 seems to be aborting VPD TACs (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006006.html). The rates don't make sense. This is being looked into.
- Polarizations looked to be about B ~ 30, Y ~ 50 last night.
More detailed discussion can be found on the ops list (https://lists.bnl.gov/mailman/private/star-ops-l/2021-December/006016.html)
12/28/21
I. RHIC Schedule
Work on blue injection during the day to prevent increase in emittance
Collisions later afternoon and overnight
Maintenance day is rescheduled to Jan 5th, no planned access tomorrow
II. Recap
production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
zdcPolarimetry_2022: trg+daq
CosmicLocalClock_FieldOn: trg + daq + itpc + tpx + btow + etow + esmd + tof + gmt + mtd + l4 + fcs + stgc + fst
III. Tasks for commissioning
a) Local polarimetry
b) sTGC noise and HV scan and FST HV scan finished yesterday
c) MTD HV scan, after avalanche/streamer analysis
d) VPD splitter board (Christian, maintenance day)
e) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
IV. Open issues
a) FCS Mpod slot-1 looks dead, no alarm for LV
b) Eemc-pwrs1 NPS has a network interface failure, spare is available with NEMA 5-20 plug (maintenance day, Wayne)
c) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th
V. Plan of the day/Outlook
a) Work on blue injection during the day
b) Collisions later afternoon and overnight
c) ETOF by expert operation
12/27/21
I. RHIC Schedule
Diagnostic for quench detector and ramp development during the day
Snake settings to compensate for partial snake
Collisions for physics with store-to-store change in emittance in the afternoon and collisions overnight
II. Recap
production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
zdcPolarimetry_2022: trg+daq
III. Tasks for commissioning
a) Local polarimetry
b) FCS gain calibration, full FastOffline for HCal
c) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (call Daniel + Prashanth)
d) FST HV scan, sw update without beam, call Xu when physics, together with sTGC, dedicated production configuration
e) MTD HV scan, after avalanche/streamer analysis
f) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
g) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
IV. Open issues
a) Leaking valve replaced for TPC gas, methane concentration from 9% to nominal 10% over these two days, more frequent laser runs (2 hours)
b) TPX automatic power-cycling, ongoing
c) Eemc-pwrs1 NPS has a network interface failure, spare is available with NEMA 5-20 plug (maintenance day 29th, Wayne)
d) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th
V. Plan of the day/Outlook
a) Ramp development during the day
b) Collisions with emittance changes store-to-store later afternoon and collisions overnight
12/26/21
I. RHIC Schedule
Slower ramp rate (x5) due to problem with quench detectors, work scheduled for tomorrow
Ramp development during the day
Collisions afternoon with intensity steps and overnight
II. Recap
production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
zdcPolarimetry_2022: trg+daq
III. TPC gas
a) Fluctuations in PI8 and CH4-M4 since yesterday afternoon, interlock overnight
IV. Tasks for commissioning
a) Local polarimetry
b) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (Daniel + Prashanth)
c) MTD HV scan, after avalanche/streamer analysis
d) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
e) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
V. Open issues
a) Temperature increase in WAH, yellow alarms for several VMEs
b) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th
VI. Plan of the day/Outlook
a) Ramp development during the day, also stores for physics, MCR will inform
b) Collisions with intensity steps afternoon and overnight
12/25/21
I. RHIC Schedule
Collisions for commissioning
Energy scan shall resume on 12/26
II. Recap
production_pp500_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
III. Updates
a) BEMC PMT trips
b) Set of triggers elevated to physics (entry for Run 22359013)
IV. Plan of the day/Outlook
a) Collisions for commissioning
b) Energy scan tomorrow 12/26
12/24/21
I. RHIC Schedule
Energy scan was interrupted by QLI in blue and power dip (2 out of 6 points done), access ongoing for recovery from the quench (~4 hours)
Collisions afternoon, intensity steps, and overnight
Energy scan shall resume on 12/26
II. Recap
zdcPolarimetry_2022: trg+daq for part of energy scan
III. Tasks for commissioning
a) Local polarimetry
b) FCS gain calibration, FastOffline finished, ECal ok (pi0), HCal
c) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (Daniel + Prashanth)
d) MTD HV scan, after avalanche/streamer analysis
e) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
f) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
IV. Open issues
a) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th
b) TPX automatic power-cycling
c) Readiness and detector states
V. Plan of the day/Outlook
a) Access ongoing
b) Collisions afternoon, intensity steps and overnight
c) Energy scan 12/26, call Ernst
12/23/21
I. RHIC Schedule
Energy scan, low intensity, afternoon: intensity steps, overnight: collisions
II. Recap
Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
zdcPolarimetry_2022: trg + daq
III. Tasks for commissioning
a) Local polarimetry, scan will start later because of a quench (11:30 EST)
b) Spin direction at STAR, longitudinal in blue is a part of systematic error
c) Scaler bits timing ok now
d) FCS gain calibration
e) sTGC noise thresholds, 2-3 hours without beam when possible, HV scan with beam (Daniel + Prashanth)
f) MTD HV scan, after avalanche/streamer analysis
g) FastOffline, new request for FCS finished
h) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
i) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
IV. Open issues
a) BTOW LV + FCS LV alarm, minor -> major for channel trip
b) sTGC LV
c) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V, January 4th and 5th
d) TPX automatic power-cycling
e) Mailing lists to inform about any changes + logbook
f) BTOW PMT recovery when opportunity for access, call Oleg (daytime/evening)
g) Readiness and detector states
h) ZDC-SMD pedestal for west horizontal #4
V. Plan of the day/Outlook
a) Energy scans, ZDC polarimetry, all detectors for machine commissioning
b) Collisions overnight
12/22/22
I. RHIC Schedule
Vernier scan, cross section compatible with run 17, energy and squeeze ramps
Local component in blue beam, possibilities include use of existing snakes or phenix rotator, orbit imperfection tuning and energy scan
II. Recap
Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
Polarimetry runs zdcPolarimetry_2022: trg + daq
III. Tasks for commissioning
a) Local polarimetry
b) Scaler bits timing
c) Trigger for Vernier scan
d) FCS gain calibration
e) sTGC data volume
f) sTGC noise thresholds
g) MTD gas, more SF6, HV scan, trigger config
h) FastOffline re-running to include EPD
i) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
j) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
IV. Open issues
a) Temperature in WAH
b) Gating grid sector 21 outer disconnected, RDOs masked out, relevant anodes at 0 V
c) Anode HV for sector 15, channel 3 at 1000 V as default
d) TPC Chaplin frozen (gui available also on sc3 or on alarm handler)
e) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)
f) BEMC critical plots checked by shift crews (holds in general)
g) Reference plots for critical plots
h) SL on star-ops list
V. Plan of the day/Outlook
a) Scans related to longitudinal component and intensity steps during the day
b) Collisions overnight
12/21/21
I. RHIC Schedule
9 MHz RF cavity adjusted, can go to full intensity, alignment for yellow abort kicker, IPMs configured
Snake current increased from 300 to 320 A, blue polarization improved to ~42%
Stores during the day with intensity steps and overnight
II. Recap
Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
Polarimetry runs zdcPolarimetry_2022: trg + daq
Run with 0 V at TPC 21 outer, 400 V after that
III. Tasks for commissioning
a) FCS rates, x10 - 20 higher, tests runs with change in gain/masks, beam position?
b) sTGC data volume, firmware update
c) Local polarimetry, spin angle
d) FastOffline re-running to include EPD
e) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
f) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
g) BEMC tolerable tripped boxes, 1 out till 29th, DOs follow procedure to recover, run flag as questionable, note in shift log (specific for crate)
h) Vernier scan, low number of bunches
IV. Open issues
a) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out, meeting today 3:30pm
b) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)
V. Plan of the day/Outlook
a) Stores during the day with intensity steps
b) Collisions overnight
12/20/21
I. RHIC Schedule
Blue snake re-wired for correct polarity (coil #3)
Timing alignment for abort kicker in yellow beam
Access at 10am for 9 MHz cavity
Ramp development after the access, then collisions after 5pm till tomorrow day
II. Recap
Collisions with 111x111 bunches, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
Polarimetry run zdcPolarimetry_2022: trg + daq
Blue polarization at 30%
III. Tasks for commissioning
a) FCS closing
b) ZDC-SMD hot channel, daughter card to be replaced (Christian)
c) Local polarimetry, scaler bits (Hank, Chris)
d) FastOffline completed for previous 3 stores
e) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
f) MTD dimuon trigger, prod id, trigger patch recovery at maintenance day
IV. Open issues
a) Increase in magnet current, east ptt, Monday morning
b) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out, meeting to determine the risks tomorrow
c) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)
V. BEMC operation
a) Shift crew should star looking at critical plots, they are the same for BTOW as last many years. 2d hit map is main indicator of HV status. There are four boxes tripped since probably Sat., this was not noticed.
b) Det. operator please don't hit wrong button in HV GUI, that can lead to a long recovery of HV, as it was today ~3 hours.
c) For operation instruction:
(a) during long downtime shift should run btow_ht configuration just to check HV was not tripped, looks like during Sat. evening shift no one exercise the system at all.
(b) given that recovering one PMT box may lead to trip and then long recovery of entire BEMC HV, we better not to do such thing during overnight shifts for example. Instead, may be, barrel jet triggers should be disabled, and live HT triggers only. Then recover HV between fills?
VI. Plan of the day/Outlook
a) Access 10am, beam development after
b) Collisions after 5pm
12/19/21
I.RHIC Schedule for today-tomorrow
Ramp-up intensity (up to 1.5*10^11) (limited by yellow RF)
(partial) blue snake ramp-up
Collisions with luminosity likely with blue+yellow snakes overnight (111 bunches)
II.Recap
Collisions 111 bunches since 2am, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
Polarimetry run zdcPolarimetry_2022: trg + daq
Abort gap at 2/8
Intensity ~1*10^11
BBC/VPD/ZDC : 0.9 / 0.4 / 0.07M
~55% polarization for yellow ~0% for blue
Access: ZDC scaler / TCMI (Zhangbu,Tim) – fixed
ZDC SMD E-V 2 hot channel (Aihong) - ongoing
III.Tasks for commissioning
a) Detector performance at higher luminosity / issues
b) Any issues with “Beam loss”? (6:43 am)
c) Trigger rates vs beam (ex: BHT3 rate lower ~x2 vs Run17)
d) ZDC SMD hot channel
e) Local polarimetry
f) FCS closing Monday morning?
IV.Open issues
a) Increase in magnet current, east ptt, Monday morning
b) NPS for BC1 for 208V, power cord over two racks (Tim)
c) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out
d) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)
V.Plan of the day/Outlook
a) Ramp development (intensity, snake) during the day
b) Collisions (run production) in owl shift
12/18/21
I. RHIC Schedule
Ramps with higher intensity, abort gaps to be aligned, work for UPS for blue RF 9 MHz cavity
Collisions with larger luminosity overnight (111 bunches)
Tomorrow: Snake ramp up, intensity recommissioning, polarized collisions overnight
II. Recap
Collisions 56x56 bunches since midnight, production_pp200_2022: trg + daq + tpx + itpc + btow + etow + esmd + tof + mtd + gmt + fcs + stgc + fst + l4
Polarimetry run zdcPolarimetry_2022: trg + daq
60% polarization for yellow from RHIC
III. Tasks for commissioning
a) sTGC mapping
b) FST status
c) FastOffline requested for st_fwd
d) ZDC east channel 2 on in QT, no coincidence in RICH scalers after TCIM reboot, incorrect discriminator level, access 2pm - 3pm, SMD to be checked also
e) Local polarimetry
f) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
g) FCS closing Monday if blue RF ok
IV. Open issues
a) BBC is ok (no trigger on previous xing on east) after power cycle to BBQ, bit check to be monitored (Akio)
b) Increase in magnet current, east ptt, Monday morning
c) NPS for BC1 for 208V, power cord over two racks (Tim)
d) Gating grid sector 21 outer disconnected, 12h min + risk of need to remove parts in front, maintenance 29th, RDOs masked out
e) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin, spare is available with NEMA 5-20 plug (Wayne)
V. Plan of the day/Outlook
a) Ramp development during the day, access for ZDC afternoon
b) Collisions in owl shift
12/17/21
I. RHIC Schedule
Potential controlled access till 1pm, ramp development after (squeeze ramp, blue tune kicker, intensity ramp up)
Collisions in owl shift
II. Recap
Collisions 12x12 bunches since 4am, sTGC and FST voltage scans with field ON, tuneVertex_2022: trg + daq + tpx + itpc + fcs + stgc + fst + l4
III. Tasks for commissioning
a) FST (nominal voltages as before) + sTGC voltage scan (sTGC done, 2900 V is default for now)
b) BBC lost earliest TAC on east, EPD was used for voltage scan instead
c) VPD splitter board (Christian, maintenance day 29th, Daniel to be notified)
d) Local polarimetry, results west ZDC only, code issue? (Jinlong), polarimetry runs tonight
e) FCS mapping to be checked after cable swap
IV. Open issues
a) Increase in magnet current, east ptt
b) BC1 fan tray swap, no alarm when ongoing, no on/off via slow controls, NPS? (Tim, David)
c) Gating grid sector 21 outer disconnected, anode at sector 21 outer at 800 V, RDOs are masked, capacitance consistent with cable alone, 12h min + risk of need to remove parts in front, maintenance 29th
d) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
e) sTGC has no data in first run after LV power up, under investigation
f) star-ops mailing list is slow in delivery, also other lists (stgc)
g) AC in control room
V. Plan of the day/Outlook
a) Potential controlled access till 1pm, ramp development after
b) Collisions in owl shift, production configuration (prod ids except mtd), ZDC polarimetry, FCS closing Sat/Sun
c) Forward detectors by experts only, sTGC mapping (Daniel)
d) Saturday: ramp development during the day, collisions in owl shift
12/16/21
I. RHIC Schedule
Blue snake reconfigured for coils #1 and #3, tests for abort kicker UPS
CeC till 8pm, beam development after
Collisions in owl shift
II. Recap
No collisions because of water flow problem at beamstop, caused by incorrect orifice
Cosmics, tune configuration
III. Tasks for commissioning
a) Magnet on/off? -> feedback from FST by 4pm EST
b) FST + sTGC voltage scan, procedure will be set by magnet on or off case
c) MTD, no dedicated commissioning run?
d) VPD slew parameters loaded, TAC windows set, investigation ongoing for splitter board
e) Scalers board, signals ok, more than 6 bunches needed
f) FCS status
g) ZDC status ok
IV. Open issues
a) BC1 multiple power-cycle on crate, SysReset, on/off in slow controls? Fan tray swap when possible (Tim)
b) Gating grid sector 21 outer disconnected, anode at sector 21 outer at 800 V for no gain, fix at maintenance day
c) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
d) sTGC has no data in first run after LV power up, under investigation
e) star-ops mailing list is slow in delivery, also other lists (stgc)
V. Resolved issues
a) EPD mapping at the splitter
b) Magnet monitoring ok after maintenance yesterday, alarm limits ok
VI. Plan of the day/Outlook
a) CeC till 8pm, beam development after
b) Collisions in owl shift
12/15/21
I. RHIC Schedule
Maintenance for CeC and blue snake re-wiring, ramp development after 4pm
Collisions late afternoon / overnight
II. Recap
Collisions with 12x12 bunches with forward detectors, production_pp500_2022, tuneVertex_2022
III. Open issues
a) sTGC voltage scan, another scans today (Prashanth + David to be called), in sync with FST
b) FST voltage scan, looks ok from last night, another scans today
c) tuneVertex_2022 for sTGC and FST voltage scans, runs for target number of events + add FCS, use BBC trigger
d) Lists of tasks for collisions from experts passed to SL
e) FCS status, trigger list
f) VPD one channel to be checked for max slew - mask out this one for now, TAC window, need feedback on pedestals while still in access, cabling check (Christian)
g) EPD calibrated now
h) Cal scan, ESMD PMT voltages updated, ETOW phase to be applied
i) ZDC towers check ok (Tomas), signal ok
j) One run with ZDC-SMD HV off, signal cables checked ok on side patch (Aihong)
k) Cabling check today (Christian)
l) Scalers board, SMD counts still at RHIC clock (Jinlong)
m) MTD commissioning (Shuai), VPD trigger and cal needed, instructions for SL by Shuai
n) BC1 power cycled on crate (Tim), booted ok, CAN address 73 will be set (Christian)
o) Gating grid status (Tim), sector 21 timing
p) Laser runs every 4 hours
q) TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), log affects, delay does not, new tier1 fixed it, readback added
r) Magnet alarm limits
s) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
t) sTGC has no data in first run after LV power up, under investigation
IV. Resolved issues
a) Commissioning done for: BBC, EPD, BTOW, ZDC
V. Updates
a) production_pp500_2022, BBC, BBCTAC, BHT3 and BHT3-L2W elevated (Jeff) + ETOW, VPD (almost)
b) Contact Jeff when a trigger can elevate to physics
VI. Plan of the day/Outlook
a) Restricted access now
b) Cosmics for gating grid, magnet up preferred
c) Beam development after 4pm, detectors in proper safe state
d) Collisions in the evening / overnight
e) SL tasks shift crew based on what we're running
12/14/21
I. RHIC Schedule
Damage in blue snake after power dip on Sunday evening, could use coils #1 and #3, access to rewire for these coils
UPS was disabled for abort kicker
Access now for kicker, snake and CeC, ramp development afternoon, collisions overnight
II. Recap
VPD, EPD and Cal scans
Magnet trip yesterday evening
Controlled access now (~4 hours)
III. Open issues
a) VPD commissioning (Isaac, Daniel), non-VPD trigger (Jeff), slew test with beam
b) EPD commissioning (Rosi)
c) ZDC SMD bits in scalers fire at RHIC clock (9.38 MHz), test with HV off, pedestal issue, cabling (Jinlong + Hank)
d) ZDC commissioning (Tomas, Zhangbu), signal seen, work for 1n peak
e) Cal scan (Oleg, Will J), BTOW 4ns shift, crate-by-crate scans
f) MTD commissioning (Shuai), VPD trigger and cal needed, instructions for SL by Shuai
g) Local polarimetry (Jinlong)
h) BC1 crate off? fails during boot, spot crash in startup file, power-cycle now (Tim)
i) Spike in 1st gating grid time bin (David), test now with cosmics
j) TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), test today, log affects, delay not
k) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
l) sTGC has no data in first run after LV power up, under investigation
IV. Resolved issues
a) Commissioning done for: BBC
b) Cards in EQ1, EQ2 and EQ3 replaced yesterday (Christian)
V. Updates
a) Separate trigger configurations for commissioning (Jeff)
b) File stream name for forward detectors: st_fwd
VI. Plan of the day/Outlook
a) Access now, beam development during afternoon, collisions overnight
b) Production configuration with final prescales, start with BBC, BTOW, production_pp500_2022
c) Forward commissioning with low intensity beam, Xu, Prashanth, David, VPD and EPD needed before
d) Magnet work tomorrow
e) Scalers need to run
12/13/21
I. RHIC Schedule
Polarization development and ramp development during the day, collisions with rebucketed beam late afternoon or overnight
Access at IP2
Low intensity now because of mistime abort in both rings at the power dip
Cogging depends on snake availability, needed for correct longitudinal position of vertex
Lossy blue injection
II. Recap
Collisions yesterday after 8pm, BBC HV scan, ended by power dip
Next collisions 5 am, ZDC polarimetry with singles at 2 kHz, VPD HV scan, EPD timing scan
III. Open issues
a) VPD HV 13.01 didn’t turn on, at lower voltage (1627 V) now ok, might need to swap the channel
b) Non-VPD trigger needed (BBC coincidence in L4) for VPD slewing correction, Jeff will make separate configuration file, instructions for SL by Daniel
c) Separate configuration for local polarimetry (Jeff)
d) EPD commissioning (Rosi)
e) ZDC commissioning (Tomas, Zhangbu)
f) Every trigger detector sends a message over star-ops when done with commissioning
g) Cal scan (Oleg)
h) MTD commissioning (Shuai), VPD trigger needed, instructions for SL by Shuai
i) Spike in 1st gating grid time bin, seen as perpendicular planes in event display, should fix after new pedestal, open/close test after beam dump, IOC restart (David)
j) TAC windows for BBC, EPD, ZDC, VPD in investigation (Eleanor, Jeff), monitoring to check the registers
k) L4 was not present because of incorrect R and z vertex cuts, ok now
l) Collision triggers in tune_22 for calibration and tune configuration
m) Dead QT32B daughter card for EPD (daughter A in EQ3 slot 10), also cards in EQ1 and EQ2, access needed to replace (Christian), controlled access (SL), SL calls Rosi after done to check
n) Local polarimetry in progress (Jinlong), not yet from scalers
o) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
p) sTGC has no data in first run after LV power up, under investigation
q) iTPC Sector 13, RB 3 masked out and powered off, keep like this
r) FCS LV was overheating, rack backside was opened (Tim), 1 deg drop, not critical now
s) No ETOF
IV. Resolved issues
a) BBC commissioning done for run 22, Akio not in call list for collisions
V. Updates
a) Call list for collisions, SL informs over star-ops
b) File stream name for forward detectors: st_fwd
VI. Plan of the day/Outlook
a) Potential access
b) Tune configuration with beam development, detectors in proper safe state
c) Could get collisions later afternoon or overnight
12/12/21
I. RHIC Schedule
Collisions later afternoon (4/5pm), likely 6 bunches rebucketed
Magnet quenches were caused by temperature problem at 1010A, not beam induced
Lossy blue injection, work needed on Y2A RF cavity
Rebucketing successful yesterday with 6 bunches
Scans and ramp development till 4pm, stores with collisions after
II. Recap
tune_22: trg + daq + btow + etow + esmd + fcs
III. Open issues
a) Global timing with collisions
b) TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), test with rebucketed collisions
c) Dead QT32B daughter card for EPD (daughter A in EQ3 slot 10), access needed to replace (Chris)
d) Local polarimetry (Jinlong)
e) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
f) First run after LV power up sTGC has no data, under investigation
g) iTPC Sector 13, RB 3 masked out and powered off, keep like this
h) FCS LV was overheating, rack backside was opened (Tim), 1 deg drop, not critical now
i) No ETOF
IV. Resolved issues
a) Phones were out yesterday night due to update, fixed early morning
V. Updates
a) New Readiness checklist, cosmics with 8+ hours without beam
VI. Plan of the day/Outlook
a) Tune configuration with beam development, detectors in proper safe state
b) Could get collisions later afternoon or overnight, call list for shift leaders
12/11/21
I. RHIC Schedule
Polarized scans and rebucketing tests till 8pm, then CeC until tomorrow morning
II. Recap
Collisions at 3am, 28 bunches, both snakes ramped, polarization 44% blue, 54% yellow, beam abort after 20 minutes
Next collisions 8am, ended by blue quench near the snake (but not the snake)
Cosmics, tune_22, trg + daq + btow + etow + esmd + fcs
III. Open issues
a) Global timing with collisions
b) Phones out at STAR and MCR due to update to phone system, fake (?) magnet trip in west trim at the same time, now back (9am)
c) Investigation in DSMs on TAC windows for BBC, EPD, ZDC, VPD (Eleanor, Jeff), affects triggers which use TAC, read from registers is different from write, access will be good to test the VME board-by-board (Jeff)
d) sTGC gas pressure increased after yellow alarm (Prashanth)
e) Timing for scaler board with beam (Chris), expect to be ok, needed for local polarimetry (Jinlong)
f) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
g) First run after LV power up sTGC has no data, under investigation
h) iTPC Sector 13, RB 3 masked out and powered off, keep like this
i) FCS LV was overheating, rack backside was opened (Tim), 1 deg drop, not critical now
j) No ETOF
IV. Resolved issues
a) L4 plots missing from Jevp, fixed (Jeff)
V. Updates
a) New Detector States, 12/10, sTGC for both HV & LV is OFF for PHYSICS and Vernier scan, FST HV is OFF for PHYSICS and vernier scan
b) Output from individual ZDC towers tested (Tomas)
VI. Plan of the day/Outlook
a) Tune configuration with beam development, detectors in proper safe state
b) No collisions overnight (CeC instead)
c) Cosmics only if there will be 8+ hours without beam
12/10/21
I. RHIC Schedule
Blue9 snake ramps today till 4pm, there was shorted diode against spikes from transient current
Recommissioning after that if blue snake is available, or rebucketing and ramp development if not
Stores with collisions during owl shift if ready by 10pm today
II. Recap
Collisions at 4am for short time, ended by multiple beam aborts, access ongoing now
tune_pp500_2022 with collisions, tune_22 or cosmics, field on
III. Open issues
a) Jpsi*HTTP at 1 kHz without beam, hot/warm tower ETOW/BTOW, leave out until calorimeters commissioned
b) Update in TAC min/max for ZDC, EPD, BBC (Jeff)
c) BBC HV adjusted to lower values (initial), need to finish HV scan (Akio)
d) FCS LV overheating, rack backside to be opened (Tim), 1 deg drop, not critical now
e) iTPC Sector 13, RB 3 masked out and powered off, keep like this
f) Timing for scaler board with beam (Chris)
g) Mask from L0 to L1 for a trigger patch
h) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
i) First run after LV power up sTGC has no data, under investigation
j) No ETOF
IV. Resolved issues
a) Remote access to scalers for polarimetry on cdev for Jinlong, was related to 64bit/32bit change, ok now
b) Fan tray for EEMC CANbus, crate #70 replaced (Tim), also reboot to main CANbus, gating grid restored
V. Updates
a) VPD voltages HV changed to 2013 values (Isaac)
VI. Plan of the day/Outlook
a) Schedule from rhic is largely uncertain, could get collisions in owl shift
b) tune_pp500_2022 with collisions, tune_22 or cosmics, field on, safe state when beam development
c) FST keep off until very nice beam, expert present for any operation (Xu)
d) sTGC by expert only (Prashanth)
e) Commissioning starts with collisions on, state of experimental setup now
12/09/21
I. RHIC Schedule
Possible collision setup in upcoming owl shift, progress on collimator, kicker alignment and timing, vertical injection matching
and yellow injection damper, safe state important for detectors during beam development.
blue9 snake: beam induced quench without substantial beam loss, question on magnet training or real problem,
access today for a p.s. related to the snake
Today after p.s. access: beam development without blue snake
II. Recap
Cosmic runs with field on, tune_22 with beams, trg + daq + btow + etow + esmd + fcs
III. Open issues
a) EEMC CANbus fan failure, crate #70, few minutes access to replace the tray (Tim)
b) sc5 reboot by mistake by DO when trying to reboot crate #70 caused by incomplete instructions
c) FCS LV overheating, rack backside to be opened (Tim), ½ hour to observe temperatures
d) Level for yellow alarm for sTGC pentane gas, done
e) Online database not visible yesterday ~2pm → ~5pm, Dmitry was called
f) sTGC HV IOC having multiple instances (red alarm), ok now
g) EEMC and EQ2, MXQ, and BBQ in alarm handler (David, input from experts on what to unmask in alarm handler)
h) iTPC Sector 13, RB 3 was asserting busy even masked out, was powered off, Tonko + Jeff will take a look
i) BCE DSM2, new algorithm uploaded, in test yesterday, in trigger, L0 to be checked by Chris
j) Instructions on recovery for BBC/ZDC/VPD HV system (LeCroy1440) communication after power failure, pwd to bermuda needed
k) Remote access to scalers for polarimetry on cdev for Jinlong
l) Add instructions to recover forward detectors after power dip (sTGC call experts), Oleg T will add inst for FSC, FST call experts
m) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
n) sTGC auto-recoveries (is a place-holder for final message), empty plots for a few runs → on hold for commissioning
o) Disk that stores TPC sector 8 pedestals needs to be replaced by Wayne (not urgent)
p) No ETOF
IV. Resolved issues
a) Scaler board replaced during access yesterday (Chris), SCLR48 in trigger since run 22342037
V. Updates
a) Update in sTGC HV and LV GUI (channel numbering), instructions are updated
b) Magnet current limit alarm, Flemming + David for default limits
VI. When collisions are delivered
a) Commissioning plan
b) Time scan for BEMC together with ETOW and ESMD
c) ETOW and ESMD basic QA with collisions to test if its configuration is ok
VII. Plan of the day/Outlook
a) beam development with detectors in correct safe states, tune_22 or tune_pp500_2022, cosmics when possible
b) p.s. access for blue snake, beam work till midnight, possible collisions setup during owl shift
c) ETOF may be turned over to SC for a few weeks during the run
12/08/21
I. RHIC Schedule
Test for blue9 snake ok (partial snake, ongoing work), beam work till 10:00, access 10:00 -> 12:00,
then injection, ramps and rebucketing till tomorrow (12/09 4pm)
II. Recap
Cosmic runs with field on, tune_22 with beams, trg + daq + btow + etow + esmd + fcs
III. Open issues
a) Restricted access 10am today, scaler board (Chris, finished), also for EPD (finished)
b) Add instructions to recover forward detectors after power dip
c) EEMC and EQ2, MXQ, and BBQ in alarm handler
d) iTPC Sector 13, RB 3 was asserting busy even masked out, was powered off, Tonko + Jeff will take a look
e) sTGC HV at 2900 V for now
f) sTGC auto-recoveries (is a place-holder for final message), empty plots for a few runs → on hold for commissioning
g) BCE DSM2, new algorithm uploaded, ready to test (during today), not in trigger now
h) Eemc-pwrs1 NPS which has a network interface failure and affects access to eemc-spin needs to be replaced (not urgent)
i) Disk that stores TPC sector 8 pedestals needs to be replaced by Wayne (not urgent)
j) No ETOF
k) Instructions on recovery for BBC/ZDC/VPD HV system (LeCroy1440) communication after power failure, pwd to bermuda needed
l) Access to scalers for polarimetry on cdev for Jinlong
IV. Resolved issues
a) BTOW crate Id 8 failed configuration fixed, (disconnected 0x08 board 1 and put it back)
b) Replaced the problematic DSM1 in BCE crate, hole in trigger patch 250-259 seems gone from btow_ht run, 22340037
V. Updates
a) Two screens for sc3 (VPD/BBC/ZDC HV)
b) 30 new mtd plots to the JevpPlots
c) evb01/evb07 added to the DAQ default
d) New firmware in BE004 DSM2
e) sTGC LV IOC to follow the procedure
f) To power cycle a EEMC follow the operation guide, power off and on not enough, follow manual strictly
g) TPC current calibration should be done once per day
h) Magnet current limit alarm, Flemming + David for default limits, sampling frequency?
VI. When collisions are delivered
a) Global timing, tune_pp500_2022 trigger definition
b) Time scan for BEMC together with ETOW and ESMD, files from DAQ by Tonko, min bias trigger, time interval and steps to be set
c) ETOW and ESMD basic QA with collisions to test if its configuration is ok, first reference plots will be available with collisions
VII. Plan of the day/Outlook
a) beam work till tomorrow afternoon, cosmics when possible
b) no collisions are expected till tomorrow 12/9 4pm at least
c) exercise for BBC/VPD/ZDC lecroy recovery after power failure (David)
d) ETOF may be turned over to SC for a few weeks during the run
11/17/21 to 12/07/21 Zilong Chang
11/16/21
RHIC schedule: no new info: “2-3 weeks” start-up delay (incomplete cryo controls upgrade).
11/15 Blue 4K cool-down, starting 1/2 (12-6), 11/29 < for Yellow
11/15: magnet polarity change RFF -> FF
11/15: Calibration sets taken: Long Laser run, polarity flip, long laser run + laser with resistors in the chain
Currently 1.5MOhm in the chain, How long?
will learn from the analysis (Gene) of the data set on the short in TPC
Magnet stable
All detectors are included and currently running (except ETOF)
gmt trigger is enabled
Issues and resolved:
MTD: issue with LV. RDO masked out (1 out of 2): running / Geary
BTOW: configuration fail. Fix by resetting board / Oleg
Plan for today
new shift crew + period coordinator (Zilong)
NO access 07am-12pm tomorrow (11/17) for access controls test
cosmic with all available detectors with Forward FF
run until Thursday morning with FFF
Flip the polarity back to RFF the polarity on Thursday morning (combined with BBC installation, MTD work)
let crew know the detector is not ready to be included
laser / 4 hours (separate run)
pedestal / shift
TOF,MTD noise run / day
11/15/21
RHIC schedule: the same: “2-3 weeks” start-up delay (incomplete cryo controls upgrade).
11/15 Blue 4K cool-down, 11/29 < for Yellow
Short term plan:
11/15: Flipping magnet polarity to RFF -> FF (BBC installation postponed)
11/15: Long Laser run (done), polarity flip (ongoing), long laser run + laser with resistors in the chain (to be done)
11/18: evaluate the short in TPC with data taken with two field settings, and decide on the need to open East pole-tip to fix the if necessary
magnet stable
a trip yesterday 4:30pm with “daily” power dip
TPC GG issue resolved: with correctly reloaded value
Issues, detectors not included:
FST: Error with HV ramping / 7am
STGC running, included but HV off
MTD: too many recoveries.LV control / 3am
Shift procedure
FST, STGC under shifter control?
pedestal after “warm up” time
Plan for today
cosmic with field on with all available detectors with RFF -> FF
long Laser runs
TPC, BTOW, ETOW, ESMD,TOF, FCS, sTGC, FST, MTD
let crew know the detector is not ready to be included
laser / 4 hours (separate run)
pedestal / shift
TOF,MTD noise run / day
11/13/21
RHIC schedule: the same: “2-3 weeks” start-up delay (incomplete cryo controls upgrade).
11/15 Blue 4K cool-down, 11/29 < for Yellow
Any beam activities with only Blue cold?
Short term plan:
11/12 - 11/15: continue with cosmic data taking at Reverse Full Field
11/15 Monday morning: Magnet polarity flip, BBC (West) installation
11/15 - cosmic (+laser) data taking at Forward Full Field.
11/18: evaluate the short in TPC with data taken with two field settings, and decide on the need to open East pole-tip to fix the if necessary
magnet stable
trip yesterday likely from power dip
alarm: set value, range to reduce false alarm from fluctuation
ETOW: cable fixed. DAQ error. trigger/hardware/DAQ issue?
FST overheating module 3-11. not resolved. masked-out. Still out of run?
STGC: DAQ 0. Still out
shift QA plots, online QA, event display: lagging
laser run: separate
Plan for today
cosmic with field on with all available detectors
TPC, BTOW, ETOW, ESMD,TOF, FCS, sTGC, FST, (MTD)
let crew know the detector is not ready to be included
laser / 4 hours (separate run)
pedestal / shift
TOF noise run / day