BTOF Operations (incl. slow controls)

 Barrel Time-of-Flight Operations

  1. TOF Gas: Bottle Switchover Procedure
  2. Slow Controls
  3. On-call Experts
  4. TOF Error Handling




On-call expert

Last updated: March 24th 2010 

How To Clear TOF Errors

 TOF on-call expert for this week:  Yi Zhou (cell: 631-965-6132)

                                                        Frank Geurts (Apartment: 631-344-1042)

 

Important Notes: 

The Anaconda window should always running at the TOF Station!  Do Not Close this window.

Shift crews must check the Warning and Error sections of the Anaconda window after every new run is begun. Warnings will automatically cleared when runs are begun. If an Error is not cleared automatically when a new run is begun, see sections 4 and 1 below. 

For all TOF alarms or errors, leave a note in the shift log.  Be detailed so the experts will understand what happened and you actually did.

Never click the red 3 button in the Anaconda window or power cycle a tray during a run, only when the run is stopped.

 

Additional actions: 

  1. Electronics errors and/or bunchid errors
    Stop the run and determine which tray has the issue (there likely will be a yellow LV current alarm). Clear all (other) errors in Anaconda by pressing the red (3) button. The affected tray will most likely turn gray in Anaconda. In Anaconda, turn off "Auto-Refresh" (in "Command" menu). Go to the main voltage control GUI, turn the LV for that tray off, wait 10 seconds, and then turn the LV for that tray on again. Wait 60 seconds. Then click the red (3) button in the Anaconda window. The tray status should turn green. Turn on "Auto-Refresh" again. Start a new run. Make a note in the shift log. If the problem isn't fixed, then call the expert. 

     

  2. Yellow alarm at LV power supply associated with a tray.
    First, check TOF QA plots and look for electronic errors or bunchid with electronic errors. If there are errors, then follow step 1. Otherwise don't call an expert, just make a note in the shift log.

     

  3. Missing tray or half-tray in TOF online plots (specifically Multiplicity plots), follow procedure 1.

     

  4. In the TOF_MON GUI (Anaconda II):
    1. If you see red errors and/or the online plots indicate problems, stop and restart run. If error doesn't get cleared by this, make sure that all of the small LV circles in the main voltage GUI are green. If not, follow procedure 1 above. Otherwise, call the expert.
    2. If you see yellow alarms, no need to call experts, these clear automatically at the beginning of the next run.
    3. If it is grey alarm, click the "Refresh" button (left of the "1" button) to see if this clears it, otherwise follow procedure 1.

     

  5. TOF readout 100% DAQ errors and one of the RDO 1-4 lights on the TOF DAQ receiver is not responding (LED is black instead of blue or purple). The cure is to stop the run, mark it bad, and start a new run. Nothing else is required. This has worked 100% of the time so far. Make a note in the shift log.

     

  6. Gas alarm on FM2, which is for iso-butane gas.
    When it's cold, the pressure on FM2 will decrease and may cause alarms. In this case, no need to call expert. 

     

  7. If you see the LV off, i.e. dots are blue. (In the normal situation, the dots are green, indicating the LVs are on), please call expert.

     

  8. TOF HV slow control lost communication, please leave a note in the shift log, and follow the procedure in the trouble shooting of HV manual to fix it. Call expert for help in case you do not fix the problem.

 

Slow Controls

Slow Controls Detector Operator Manuals

 by Betrand J.H. Biritz (linked version: Jan.5, 2010)
 

 EPICs interface

 

HV manual

 

LV manual


 

 

BTOF-MTD-ETOF operation instruction documents (Run21 and After)

 

  • The instructions for the control room can be found at STAR operation page:
    • STAR Operation (BTOF/MTD/ETOF) both .doc and .pdf file are attached 
  • Expert Knowledge Manual (Only for Oncall Expert)
  • Note that, Anaconda was not in use since Run2021, all the procedures related to Anaconda can be ignored at this moment unless we include Anaconda back to the operations.

===============================================================

Run 15 TOF+MTD Operations Documents

Run 15 TOF + MTD Operations Documents

 

This page is dedicated to hosting documents  to aide in the operation of the TOF, MTD, and VPD subsystems.  Listed below are a list of the documents attached to this page and a brief description.  The .docx files are listed to update the documents as needed.

Canbus_Restart_Procedure.pdf
The first part of the document describes how to restart the canbus interface and the processes dependent on it (recoveryd and anaconda).
The second part adds a powercycle of the interface to the procedure.

Expert_Knowledge_Base.pdf (document formerly known as Etc.pdf)
A collection of tidbits for TOF+MTD folk.  Not quite intended for det. operators.

MTD_HVIOC_Restart.pdf
This guides the detector operator on how to restart an HVIOC for the MTD.
  If you want to get super fancy with restarting IOCs, you might want to check their saved states in the ~/HV/HVCAEN*/iocBoot/save_restore_module/*.sav* and see what the IOC is initializing with.  The same thing applies for the LV (MTD/LV/LVWiener*/iocBoot/save_restore_module/) and TOF.

MTD_Power_Cycle_Procedure.pdf
Instructions for the detector operator on performing a LV powercycle.

Recoveryd_Checking_Restarting.pdf
 This document describes how to check and restart recoveryd.

Responding_To_DAQ_Powercycle_Requests_Monitoring_And_Clearing_Errors.pdf (Formerly known as General_Instructions)
  This document is used as a guide to respond to DAQ powercycle requests and monitoring and clearing errors given by DAQ.

TOF_Freon_Bottle_Switchover_Procedure.pdf
 The document decribes how to monitor the freon and what to do when it runs out.

TOF_HVIOC_Restart.pdf 
To guide the detector operator through a TOF HVIOC restart.

TOF_Power_Cycle_Procedure.pdf
Instructions for the detector operator on performing a LV powercycle.

 

Run 16 TOF+MTD Operations Documents

This page holds TOF and MTD operations documents used during Run-16. Previous versions and possibly current versions may be found on Run-15's equivalent page. [ie: If you do not see the document attached to the bottom of this page, it has not been updated since last run and may be found on the Run-15 page.]

Run 16 TOF + MTD Operations Documents

 

This page is dedicated to hosting documents  to aide in the operation of the TOF, MTD, and VPD subsystems.  Listed below are a list of the documents attached to this page and a brief description.  The .docx files are listed to update the documents as needed.

Canbus_Restart_Procedure.pdf
The first part of the document describes how to restart the canbus interface and the processes dependent on it (recoveryd and anaconda). (This document has been updated since Run-15)
The second part adds a powercycle of the interface to the procedure.

Expert_Knowledge_Base.pdf (This document has been updated since Run-15)
A collection of tidbits for TOF+MTD folk.  Not quite intended for det. operators.

MTD_HVIOC_Restart.pdf
This guides the detector operator on how to restart an HVIOC for the MTD.
  If you want to get super fancy with restarting IOCs, you might want to check their saved states in the ~/HV/HVCAEN*/iocBoot/save_restore_module/*.sav* and see what the IOC is initializing with.  The same thing applies for the LV (MTD/LV/LVWiener*/iocBoot/save_restore_module/) and TOF.

MTD_Power_Cycle_Procedure.pdf
Instructions for the detector operator on performing a LV powercycle.

MTD_LVIOC_Restart.pdf  (This document is new since Run-15)
To guide the detector operator through a MTD LVIOC restart.

Recoveryd_Checking_Restarting.pdf
 This document describes how to check and restart recoveryd.

General_Instructions.pdf (This document has been updated since Run-15) (The document formerly known as "The document Formerly known as General_Instructions" or Responding_To_DAQ_Powercycle_Requests_Monitoring_And_Clearing)
  This document is used as a guide to respond to DAQ powercycle requests and monitoring and clearing errors given by DAQ.

TOF_Freon_Bottle_Switchover_Procedure.pdf
 The document decribes how to monitor the freon and what to do when it runs out.

TOF_HVIOC_Restart.pdf  (This document has been updated since Run-15)
To guide the detector operator through a TOF HVIOC restart.

TOF_LVIOC_Restart.pdf  (This document is new since Run-15)
To guide the detector operator through a TOF LVIOC restart.

TOF_Power_Cycle_Procedure.pdf
Instructions for the detector operator on performing a LV powercycle.

 

Run19 TOF and MTD operation instruction documents

 This page summarizes all the documents for the BTOF and MTD operation.

Run20 TOF and MTD operation instruction documents

This page summarizes all the documents for the BTOF and MTD operation during Run 20.

Attached files:

  • version 1/6/20 by Zaochen Ye

Know issues:

TOF Error Handling

Last updated: March 23nd 2010 

TOF Error Handling

 TOF on-call expert for this week: Bill Llope (Apartment: 631-344-1042, Cell: 713-256-4671)

 

The Anaconda window should always be running on the TOF work station. It keeps a log of important operating conditions and automatically clears most type of electronics errors at the beginning of each run.

The most important indication of the TOF system "health" are the online plots. They should be checked after the beginning of each run.

Shift crews should check the Warning and Error sections of the Anaconda window to see that Warnings and Errors are being cleared automatically at the beginning of each run.

Please make a note of all TOF alarms or any error not automatically cleared by Anaconda in the shift log. Please include enough information so the experts can understand what happened and what you actually did.

Never click the red 3 button in the Anaconda window or power cycle a tray during a run, only when the run is stopped.

 

Additional actions: 

  1. Electronics errors and/or bunchid errors
    Stop the run and determine which tray has the issue (there likely will be a yellow LV current alarm). Clear all (other) errors in Anaconda by pressing the red (3) button. The affected tray will most likely turn gray in Anaconda. In Anaconda, turn off "Auto-Refresh" (in "Command" menu). Go to the main voltage control GUI, turn the LV for that tray off, wait 10 seconds, and then turn the LV for that tray on again. Wait 60 seconds. Then click the red (3) button in the Anaconda window. The tray status should turn green. Turn on "Auto-Refresh" again. Start a new run. Make a note in the shift log. If the problem isn't fixed, then call the expert. 

     

  2. Yellow alarm at LV power supply associated with a tray.
    First, check TOF QA plots and look for electronic errors or bunchid with electronic errors. If there are errors, then follow step 1. Otherwise don't call an expert, just make a note in the shift log.

     

  3. Missing tray or half-tray in TOF online plots (specifically Multiplicity plots), follow procedure 1.

     

  4. In the TOF_MON GUI (Anaconda II):
    1. If you see red errors and/or the online plots indicate problems, stop and restart run. If error doesn't get cleared by this, make sure that all of the small LV circles in the main voltage GUI are green. If not, follow procedure 1 above. Otherwise, call the expert.
    2. If you see yellow alarms, no need to call experts, these clear automatically at the beginning of the next run.
    3. If it is grey alarm, click the "Refresh" button (left of the "1" button) to see if this clears it, otherwise follow procedure 1.

     

  5. TOF readout 100% DAQ errors and one of the RDO 1-4 lights on the TOF DAQ receiver is not responding (LED is black instead of blue or purple). The cure is to stop the run, mark it bad, and start a new run. Nothing else is required. This has worked 100% of the time so far. Make a note in the shift log.

     

  6. Gas alarm on FM2, which is for iso-butane gas.
    When it's cold, the pressure on FM2 will decrease and may cause alarms. In this case, no need to call expert. 

     

  7. If you see the LV off, i.e. dots are blue. (In the normal situation, the dots are green, indicating the LVs are on), please call expert.

     

  8. TOF HV slow control lost communication, please leave a note in the shift log, and follow the procedure in the trouble shooting of HV manual to fix it. Call expert for help in case you do not fix the problem.

 

TOF On-call expert

Last updated: February 8th 2010

 

TOF on-call expert for this week:Lijuan Ruan, Yi Zhou, Geary Eppley and Jo Schambach

 

Expert information is also below:

Today through Thursday (Feb 09-Feb 11th):
Feb 12th 0:30-7:30 shift is included.
Lijuan Ruan: 510-637-8691 (cell)
I will leave for APS meeting on Friday morning.

Friday-Saturday (Feb 12th - 13th):
Feb 14th 0:30-7:30 shift is included.
Yi Zhou: 631-344-1014 (apt #), 631-344-7635 (office).

Sunday-Monday (Feb 14th-15th):
Feb 16th 0:30-7:30 shift is included.
Geary Eppley: 713-628-2738 (cell)

Jo will arrive on the night of Feb 15 th, so for Feb 16th shift, please call
Jo Schambach: 631-344-1042 (Apt #) for help.

 

For all TOF alarms, leave a note in the shift log. Additional actions:

 

  1. Gas alarm on FM2, which is for iso-butane gas.
    When it's cold, the pressure on FM2 will be changed, this will lead to the flow of isobutene dropping, thus give us alarms. In this case, no need to call expert. We expect isobutene to run out sometime in Feb.

     

  2. Temperature alarms on the LV powersupply, no need to call expert.

     

  3. Electronic errors or bunchid errors with electronic errors
    Stop the run and determine which tray has the issue (there might be a yellow LV current alarm). Turn the LV off, wait 10 seconds and then turn the LV on again. After waiting 10 seconds click the red 3 button in the Anaconda window, the tray status should turn green. Start a new run - if the problem isn't fixed then call the expert.

     

  4. Yellow alarm at LV poweruspply associated with a tray.
    First, check TOF QA plots and look for electronic errors or bunchid with electronic errors. If there are then follow step 3, otherwise don't call but make a note in the shift log.

     

  5. Missing tray on online plots, call expert.

     

  6. If you see the LV off, the dots are blue. (In the normal situation, the dots are green, indicating the LVs are on), please call expert.

     

  7. In the TOF_MON GUI:
    1. If you see red errors, stop and restart run. If error doesn't get cleared by this, call expert.
    2. If you see yellow alarms, no need to call experts.
    3. If it is grey alarm, call the expert.

     

  8. TOF HV slow control lost communication, please leave a note in the shift log, and follow the procedure in the trouble shooting of HV manual to fix it. Call expert for help in case you do not fix the problem.

TOF noise rate pattern

BTOF and MTD noise patterns are determined from dedicated TOF noise runs. Such runs are typically executed once per day -time permitting- during each RHIC Run at times when there is no beam stored in the rings. With a dedicated trigger 4M events are collected with BTOF and MTD systems both included with HV switched on.

The results are archived at this location: https://www4.rcf.bnl.gov/~geurts/noise/

Input data:

  • TRG Setup: pedAsPhys_tcd_only
  • Number of events: 4M
  • Detectors: tof {mtd, etof} dan trg
  • RHIC status: no beam
  • Detector status: HV fully ramped up

recent example in STAR Runlog: Run 24151017

Ideally, shift crew note in the Shift Log that these runs are successfully executed TOF noise runs. Make sure to check the Shift Log and/or SlowControls to verify.

Potential error modes:

  • HV was not ramped up (check slow controls status)
  • Beam was still in RHIC or beam injection happened (check shift log or RHIC beam current info in slow controls)
  • Not all relevant detectors have been included (check list of detectors for this run in Run Log)