fms meeting 10.12.17
Report to fms group of problem and clues

FORENSICS
-----------------------------------------

David found this email: a mouse caused a power loss on 7/24 in the evening. The 7/25 coldstart of the PDUs likely occurred as soon as power to STAR WAH was recovered.
http://www.star.bnl.gov/HyperNews-star/protected/get/rts/844/1/1.html

Subsequent emails in this forum detail effects of the power loss: http://www.star.bnl.gov/HyperNews-star/protected/get/rts.html?maxm=200

After FMS Meeting: Akio restored South PDU communication; log files recovered

Log Files:
North PDU Log
South PDU Log

Some visual inspection was performed along with spot checking uvLED power supplies;
see http://www.star.bnl.gov/HyperNews-star/protected/get/starfms/1248/3.html

DAMAGE ASSESSMENT
-----------------------------------------

- PMTs look ok -- all cells marked as 'working' in run 17 are still working, according to an FMS-LED run 18311028

- LED peaks are sharp and do not drift over time

- Scanned through flasher LED amplitudes; looks normal
--- 18311032  @  amp=120
--- 18311033  @  amp=150
--- 18311034  @  amp=180
--- 18311035  @  amp=200
--- 18311036  @  amp=225
--- 18311037  @  amp=255 (usual amplitude during production running)

- Scanned through PMT HV settings (with flasher amplitude at 180 so we don't saturate);
   This is to check gain curves with no B field at all
    --> gain curves look very good and we conclude the PMTs are responding normally
           

- Opened FMS and inspected glass, no problems found

- Small cells look very clear, compared to radiation-hard glass (according to a snake camera, whose colors may not be completely accurate)

- Flasher LED system looks like it is intact (no LEDs fell off; should check more carefully before run 18)

- Thread seal tape for pipe fittings (for chilled water) has melted; this is only true for the fittings at the cold plates, whereas the fittings at the manifolds look ok
   (actually, Bob Soja says these are seal 'indicators')

             
   
    David has more pictures at https://drupal.star.bnl.gov/STAR/blog/dkap7827/fms-pictures-after-2-month-uv-curing

- Some of the blue and red hoses which carry water into / out from cold plates are darkened, and appear to 'fade' to regular color as one proceeds away from the cold plates
   The blue hose shown below is almost grey at the top (near the cold plate) and fades to blue as one proceeds downward toward the manifold

- any point at which a wire was touching a cold plate or copper tube also melted, some conductor was exposed, especially if the wire was touching a corner of a cold plate
   --> this likely caused a major short of the array, and was probably the reason the PDUs claimed a >20A bank current draw ~10 min after the system powered on in the middle of the night
          (normal bank currents should be around 13-15 A)
    --> these wires were carrying ~50 V at ~3 A each
    --> likely had some current flowing through cold plates and frame, possibly even through the unistrut / FMS frame itself and mu-metal shielding
    --> we will need to very carefully inspect any other nearby wires, especially ribbon cables which connect the flasher LEDs;
            fortunately, this ribbon cable is (mostly) not in contact with cold plates

- about 1/3 of the fuses are blown

- The uvLEDs themselves were not tested, given all the aforementioned damage;

- Could uninstall the uvLED array prior to run 18; need to assess how this will affect the flasher LED system, but that probably will be ok
   --> this will allow for a more-complete inspection of the damage caused by this incident
   -->  radiation damage in run 18's BES is not expected to be too bad anyway, so no need for UV curing

- Good news is that the FMS is in a working state and can be used for run 18