TPC issues February

Status  2/10/19
Checked last nights run.
Some are very noisy e.g. 41001, 41002- 41005
most others quit clean but with brief ~1 min  spikes 

Alexei took a picture of the cable and mapping to drivers
It is attached here

Went back thru several days (Midnite Sunday). I was looking for any specific event before no such clear correlation was observed.

on 2/3 Sunday a single run with the failure 34019  ; ~1 min spike and sector 4,6,7,8,9 bad clusters/adc
on Monday a single instance (weak) 35058
on Tuesday 36031,36048
on Wed 37007,(12,13) 29,48,50,51,63
on Thursday numerous runs  
Clearly an increasing tendency for failures.

First case found from last Sunday 34091
------
On or about 39010 during a laser run lots of noise was observed in L3 display during a laser run.
It was thought that this was due to bad pedestals. It has now been recognized that this is most likely due to 
a problem in the GG driver.

in 39041 I saw that for a short while ~ 5 min TPX and iTPC data volume increased a factor of two.
It was observed by Tonko after some discssion and investigations
"Appears  only in sectors 4,5,6,7,8,9 & 10. Inner and outer sector

And it's at the same timebin but not the same timebin for all
sectors. Hm, gating grid is suspect. OTOH, there are too many
different boards in play so it must be the main driver, so-called "TDBm"
board. Perhaps some driver chips come in quads which is why
it affects sectors in quads." (Tonko) 

This happens in many runs. 
A typical pattern in DAQ rates is below. For a few minutes the rate is higher

For any run where this appears the lots of clusters at same time bin in a given sector, but not same for all sectors are seen

Here are some sample plots from online monitoring that illustrates this.