Run 22 EPD issues

Executive Summary

EPD QT crates (EQ1, EQ2, EQ3) have their clocks fed by the BBC RCC2 and not the L1/L0 Master RCC --> If the BBC crate is out of the run for any reason, the EQ crates will just go bonkers.

During cosmics there were 2 hot channels that were maxed out: EQ1 Slot=14 EQ00b bd=b QTc ew=1 ch0  and
EQ2 Slot=13 EQ01a bd=a QTc ew=0 ch0 (West PP5 TT2 and East  PP1 TT2)

There was no data seen in: QTB daughter card A in EQ3 Slot 10 (1x1A) when we started running, this will be replaced.

Pedestals are very problematic in:

EQ1 QT32C Daughter card B in the 13th slot (0x1D, QTC EQ00A)
EQ2 QT32C Daughter card B in the 12th slot (0x1C, QTC EQ01A)
These will need to be replaced.

There is a mismatched tile-channel combination. EQ3 Bd 0x13 Ch 21 --> West PP2 TT0 shows signal, so it is obviously not a zero tile.

Details

Two channels were hot during the cosmic running early in the run.  They were masked out via:
EQ1_QT11_DA_Chnl_Mask 0 -> 1
EQ2_QT10_DA_Chnl_Mask 0 -> 1

These correspond to: 
EQ1 S=14 EQ00b bd=b QTc ew=1 ch0  ADC=37 TAC=1916
EQ2 S=13 EQ01a bd=a QTc ew=0 ch0  ADC=38 TAC= 653

Which are:
West PP5 TT2
East  PP1 TT2

The pedestals for every channel for all runs separated by crate are at:
EQ1: drupal.star.bnl.gov/STAR/system/files/pedHisteq1_12112021_0.pdf
EQ2: drupal.star.bnl.gov/STAR/system/files/pedHisteq2_12112021_0.pdf
EQ3: drupal.star.bnl.gov/STAR/system/files/pedHisteq3_12112021_0.pdf
Run 22342044 - As of this run, EPD vped values have been updated.
The zero values right before this run on all the plots are from those runs where I zeroed out the pedestal for various calculations (vped determination, mapping) so this is nothing to worry about.

For EQ1 and EQ2, boards 0 - 5 are QT32B, boards 6-10 are QT32C.
For EQ3 boards 0-2, 5 - 7 are QT32B, boards 3-4 and 8-9 are QT32C.
For the QT32C boards, ch 0 - 3, 8 - 11, 16 - 19, 24 - 27 are ADC, the other channels are TAC.

Hank says "There are 0's for EQ1 Bd 13 DC B, and for EQ2 Bd 7 DCA"

EQ1:

The zeros in the plots that are common across all channels from right before 22342 come from the runs were I specifically zeroed out the pedestals when I was doing vped/mapping runs, so no need to worry.
 
EQ1 Board 6 (following the number scheme from the mean pedestal files on L2 which starts at 0), in slot 0x19 labeled QTC EQ007 had 16 channel drift to a zero pedestal sometime near the end of last run and had been pegged at zero until after I did the vped settings.  Let's give this one some time to see if it is "fixed" (one can ask why the pedestal drifted - is it the FEE that is changing?  The QT?  The QT daughter?)
 
EQ1 Board 7 (Slot 0x1A QTC EQ008) - channel 1 had a really high pedestal, that also seems to have dropped with my vped setting.  In fact, it looks like all the channels on the A daughter card were drifting over time, and have now been set at normal values.  For some of the TAC 7 (corresponds to ch 3 ADC) the sigma was also all over the place, but seems to have improved with my vped setting.
 
I think for this daughter card we should keep an eye on it.  If the pedestals continue to drift we should replace it.  (I'll note here I need a fairly large vped from the FEE to compensate, at least compared to the other boards.)
 
Hank flagged the B daughter card in what is called Board9 in my plots.  For that, I see the sigma for channel 9 and channel 11 are both quite large.  These were also doing a lot of drifting in the last run, I think we should replace this daughter card.  It's on the way out.

EQ2:

A couple of the QTB channels were wandering around and then pegged at a "high" value for the pedestal. I would say to keep an eye on these for now.
0x10 QTB EQ011 ch 0, 2, 21
0x11 QTB EQ012 ch 16
0x13 QTB EQ013 ch 4
0x14  QTB EQ014 ch 0, 5, 6
0x16 QTB EQ015 ch 0,
0x17 QTB EQ016 ch 3, 7, 8
 
Sigmas on the first QTC board in that crate, daughter card A seem large for channels 1 and 3.  (Board in slot 9, 0x17 QTC EQ017).  Perhaps Hank should check whether these have 0 events in their pedestal runs?  Channel 2 in this daughter card is also wandering.
 
For the next board, the first DC is also wandering around - (Slot 10, 0x18 QTC EQ018) ch 0, 1, 2, 3 - another one to keep an eye on.
 
Board in 0x1C Slot 12 QTC EQ018 daughter card B is showing some problematic behavior - drifting values, and large sigmas, this should also be checked by Hank. Ch 8, 9, 10, 11, 12 (TAC channel)
 

The A daughter card on the next board really shows problematic behavior. 0x1D Slot 13 QTC EQ019.  Ch 0, 1, 4 (TAC channel)

EQ3:

0x10 QTB EQ021 - ch 0, 2, 4, 25, 26
0x11 QTB EQ022 - ch 22, 23, 24, 25, 26, 27
0x12 QTB EQ023 - ch 22, 24, 25, 30
0x17 QTB EQ026 - ch 0, 4, 25, 26,
0x19 QTB EQ027 - ch 18, 21, 22, 24, 27
 
 
QTC boards with the same issues:
0x13 QTC EQ024 - DC B
0x1C QTC EQ029 - DC B

A run to check out is 22345020.

There is a mismatched tile-channel combination. EQ3 Bd 0x13 Ch 21 --> West PP2 TT0 shows signal, so it is obviously not a zero tile.  Two blank channels are:
EQ3 Bs 0x1D Ch1 --> West PP2 TT1
EQ3 Bd 0x1D Ch24 --> West PP2 TT4

This did show up on the mapping test that we didn't fully follow up on.