TOF Error Handling

Last updated: March 23nd 2010 

TOF Error Handling

 TOF on-call expert for this week: Bill Llope (Apartment: 631-344-1042, Cell: 713-256-4671)

 

The Anaconda window should always be running on the TOF work station. It keeps a log of important operating conditions and automatically clears most type of electronics errors at the beginning of each run.

The most important indication of the TOF system "health" are the online plots. They should be checked after the beginning of each run.

Shift crews should check the Warning and Error sections of the Anaconda window to see that Warnings and Errors are being cleared automatically at the beginning of each run.

Please make a note of all TOF alarms or any error not automatically cleared by Anaconda in the shift log. Please include enough information so the experts can understand what happened and what you actually did.

Never click the red 3 button in the Anaconda window or power cycle a tray during a run, only when the run is stopped.

 

Additional actions: 

  1. Electronics errors and/or bunchid errors
    Stop the run and determine which tray has the issue (there likely will be a yellow LV current alarm). Clear all (other) errors in Anaconda by pressing the red (3) button. The affected tray will most likely turn gray in Anaconda. In Anaconda, turn off "Auto-Refresh" (in "Command" menu). Go to the main voltage control GUI, turn the LV for that tray off, wait 10 seconds, and then turn the LV for that tray on again. Wait 60 seconds. Then click the red (3) button in the Anaconda window. The tray status should turn green. Turn on "Auto-Refresh" again. Start a new run. Make a note in the shift log. If the problem isn't fixed, then call the expert. 

     

  2. Yellow alarm at LV power supply associated with a tray.
    First, check TOF QA plots and look for electronic errors or bunchid with electronic errors. If there are errors, then follow step 1. Otherwise don't call an expert, just make a note in the shift log.

     

  3. Missing tray or half-tray in TOF online plots (specifically Multiplicity plots), follow procedure 1.

     

  4. In the TOF_MON GUI (Anaconda II):
    1. If you see red errors and/or the online plots indicate problems, stop and restart run. If error doesn't get cleared by this, make sure that all of the small LV circles in the main voltage GUI are green. If not, follow procedure 1 above. Otherwise, call the expert.
    2. If you see yellow alarms, no need to call experts, these clear automatically at the beginning of the next run.
    3. If it is grey alarm, click the "Refresh" button (left of the "1" button) to see if this clears it, otherwise follow procedure 1.

     

  5. TOF readout 100% DAQ errors and one of the RDO 1-4 lights on the TOF DAQ receiver is not responding (LED is black instead of blue or purple). The cure is to stop the run, mark it bad, and start a new run. Nothing else is required. This has worked 100% of the time so far. Make a note in the shift log.

     

  6. Gas alarm on FM2, which is for iso-butane gas.
    When it's cold, the pressure on FM2 will decrease and may cause alarms. In this case, no need to call expert. 

     

  7. If you see the LV off, i.e. dots are blue. (In the normal situation, the dots are green, indicating the LVs are on), please call expert.

     

  8. TOF HV slow control lost communication, please leave a note in the shift log, and follow the procedure in the trouble shooting of HV manual to fix it. Call expert for help in case you do not fix the problem.