This blog is intended to maintain a log of issues that will be identified as the
run progresses. For initial start the process will be as for run20 and 19. See earlier QA blog page

The run is planned to be from Jan 26, 2021 to July14, 2021, and primarely low energy &.7 collider data

Status of iTPCs FEEs and dead channels

Run ended on July 8 (dAu). There was 24 hours of cosmic with Forward Field setting

Lifteed from Tonko's google doc indicating the issues, all in the iTPC

• iTPC

  • iRDOs

    • iS04-4

      • wasn’t masked but flaky trigger iface. Tonko needs to test first.

    • iS08-2 -- remove to lab

      • 2 FEEs failed with not-POK on ports #2 and #14

    • iS11-4  -- remove to lab

      • 1 FEE with not-POK on port #13

    • iS13-2 -- remove to lab

      • bad POK on port #13

    • iS18-1 -- remove to lab

      • pad POK on port #15

    • iS20-3

      • flaky trigger interface. TL needs to test first.

      • 13-Jul seems OK on TCD-clock. Continue checking...

    • iS21-4 -- remove to lab

      • RXbad+ERRs

      • FEEs are shorting? So probably the RDO is OK but I need
        to reprogram it in the lab.

    • iS24-4

      • flaky trigger interface? TL needs to test.

      • 13-Jul seems OK on TCD-clock. Continue checking…

      • set RHICx5 phase to 0 for entire Sector

      • also holds for super-fast pedestal run -- thus BUSY holds

  • iFEEs

    • iS3-1-3

      • flaky?

    • iS3-2-14

      • loads but doesn’t sync -- cable? Or FEE?

    • iS3-4-7

      • flaky?

    • iS4-3-10

      • flaky?

    • iS8-2-2

      • no POK -- fuse on RDO?

    • iS8-2-14

      • no POK -- fuse on RDO?

    • iS9-4-4

      • loads but doesn’t sync -- cable? Or FEE?

    • iS11-4-13

      • no POK -- fuse on RDO?

    • iS13-2-12

      • no POK -- fuse on RDO?

    • iS18-1-15

      • no POK  -- fuse on RDO?

July 8
Runs ended. The dAu was bad in the beginninh withbthe crossing angle

une 21 Update of 15-3 HV section
From Alexei in elog

I checked TPC anode bad channel 15-3. I disconnected cable from CAEN crate and able to run this channel up to 1390V(outer sector nominal voltage) without any trips. i connected cable again and this section tripped at 1070 in 5 min. Could be problem with cable or sector itself. We need to keep this channel at say 500 V.
Lasers also been checked and tuned.

The HV section causes some cluster to be created so # masked pad jumps around due to the 10 rows with reduced efficiency.

June 17
RDO masked out from 2216811 fixed for 2216820 by Tonko. Only one data run was affected.
Re-checked masked out RDO iS5-3. Dead FEE PROM prevented any of the FEEs to start. Re-flashed PROM, checked and all OK

June 15
starting run 166023. Sector 15 HV section 3 could not hold volatge lowered over several runs
from 1070 to100 and finally to 0 and then 500 V. Still not fixed.
Hopefully the tracking code knows about masking out dead HV sections??

for two runs 039 and 010 by op mistake RDO TPX 3 (2,3,4,5) was masked out as they though this corresponded to HV section 3.

June 12
sector 24-4 was masked out for the rest of the run. hardware error

June  11
There are many issues with the 3.85 GeV beam, many autorecovery
In particular Tonko found

"1) Please unmask (by selecting, clicking, enabling) the
iTPC Sector 3, RDO 4.

There's a bad FEE which I masked out.

2) iTPC Sector 21, RDO 4 will need to stay masked.
It has power problems, probably a component is shorted."

Some more information on the background causing the many autorecoveries
msg to ops list
Following the discussion this morning I reviewed the number of auto recoveries for 5 fills
last night starting ~20:00 yesterday and ending this morning.
The plot enclosed shows the BBCE singles rates v.s time with the largets background typical at the end of the stored. Most runs were around 1 hour. The number of auto-recoveries for each run are indicated by the numbers.

There is some correlation with recoveries being smaller at beginning of store and larger later, but there are expectations
- at store ~3:30 there was 10 in the first run in the store.
- the store starting ~6 has many recoveries but only for s24:4 so this is likely the hardware issue we see on occasions

Shortening the store the store length should reduce the number of autorecoveries, but will not eliminate them.`

June 3
from e-log:
"Issues on starting first run with iTPC 21:4 that weren't solved by power cycling so masked it out. Tonko called in and found the power cables flipped between 21:3 and 21:4, suggested power cycling whole sector if there are persistent problems in the future. iTPC 21:4 masked out in runs 22154004 - 21154010, back starting with 22154011. 

Had "ITPC[2] [0xBF82] died/rebooted -- try restarting the Run" error which stopped run 22154009 but this may be to do with the other iTPC problem? Four restarts in a row from iTPC 24:4 in first run, restarted between fills and no problems since. TPX 14:4, 22:5, then 17:6 stopped three consecutive run starts, power cycling fixed each time."

May 24
There had by many autorecovery for S24-RDO4.in the last week.
Today Tonko reported
Re-flashed Sector 24, RDO 4 PROM and also re-flashed the corresponding FEE PROM. Checked and it works OK. So let's see if this RDO continues to have issues.