Executive summary:  There is a significant correlation between higher SiPM dark currents in groups of tiles, and the temperature readings on the corresponding FEEs.  The temperature differences between FEEs is almost certainly an airflow issue within the FEE box.  Differential temperatures of the FEE cards was noticed in previous years, but the dark currents were low, so we didn't give much attention to it.  We can live with the current situation, but are discussing ways of reducing temperatures and dark currents.

Update 16 April - things have gotten a lot hotter since 5 April.

---

Dark currents in the SiPMs have increased over the years, presumably due to radiation damage.  The increase was slight, until the 2022 run of pp510.  This was not unexpected.  The dark current in some tiles increased to as high as ~4 uA when biased to physics voltages (~60V).  MIP peaks were still fine, so we decided not to swap the SiPMs.  The currents did not increase notably between the start and the end of the 2023 Au200Au run.  The currents as we install for the 2024 run are identical to what they were when we uninstalled at the end of the 2023 run, as expected.

We noticed that there is a grouping of high-current tiles along certain sectors. (A supersector, or "position" is composed of two sectors.  The 16 even-numbered tiles for a position form one sector, and the 15 odd-numbered tiles form the other sector.)  Each sector is serviced by one FEE card.  The FEE cards servicing the high-current tiles have higher temperatures than the rest.

Importantly, the temperatures are high even if voltage is set to zero (hence, it is not the high dark current that produces the high temperature).  They are also high even if the FEEs are set to "OFF".  It is unclear whether this was to be expected.  (These observations were made after significant (hours) time was allowed for temperature changes to stabilize.)


Fig 1: Temperatures of the FEE cards, seen on the EPD GUI on 5 April.


Fig 2: The dark currents.  To maximize contrast, slightly different temperature scales are used for East and West, as the East side has slightly higher currents overall.

The correlation between temperature and dark current is I believe, present (we will quantify later), but is certainly not perfect.  In some cases it is completely violated: e.g. West PP08 even tiles has high temperature but low dark current.  I also do not know where precisely the temperature sensors are placed.  It is not impossible that, for example, the reading for a sensor on a given card is most relevant for the SiPMs attached to the neighboring card.

Figure 1 shows a clear symmetry: The hot FEEs (9 odd+even, 4 even, 6 odd+even) are the same on East and West.  As can be seen on this page, there is a symmetry in the FEE-box layout, for East/West and North/South.  A bit of the figure from that page is shown below:


Fig 3: Detailed mapping of Positions/Supersectors/Fibers/FEEs for the West wheel.

Figure 3 shows that the FEEs are kind of "off-center" inside the FEE boxes.  The FEEs for PP04 and PP09 are roughly in the middle of the boxes.  You can see more detail in the figure below.


Fig 4:  The Northwest EPD FEE box.

---

Update 16 April.

Things have gotten a lot hotter!


Figure 5:  Temperatures measured with the GUI on 16 April, using same scale as in figure 1.  The hottest FEEs are again for SS09 and SS04, reaching 30C.


Conclusion:

• The temperatures on the FEE cards "between" the fans are significantly higher than those to the sides (28.5C versus 24C for the cooler ones).
• This is almost certainly an airflow issue, and is not new for this year.
• These higher temperatures may be leading to typically higher dark currents in the SiPMs with these FEEs.

We are discussing with Prashanth the possibility to put a portable cooling unit just below the SouthEast FEE box fans, to see if temperatures and dark currents drop.  This is in principle very easy, but our request comes late in the game, so we shall see.  In any event, this is not an emergency.