This blog is dedicated to the work performed by the FST group in January 2023 for the re-installation of the FST detector. The FST detector was removed from it's inserted position in the TPC cone to bake out the beam pipe. Our work begins after they rejoined the two FST halves together and placed them within the installation tooling.

Operation Log for Run 23
https://drupal.star.bnl.gov/STAR/blog/zyzhang/FST-Operation-Log-Mid-April-2023

01/25/2023 (Wednesday)

No drop in cooling system reservoir level.

Vacuum group came at 10:30 am to add another layer of Aluminized Mylar wrap around the beam pipe. All went well.

The FST returned to STAR at 2pm to help Bill finish putting up the black partitions. Then we taped down any area where light could get in with black electrical tape. The purple cables and cooling lines were dressed and then we took a pedestal run with all the lights off.

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24025004.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24025004.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24025004.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24025004.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2] 

01/24/2023 (Tuesday)

No drop in cooling system reservoir level. 

CAD Survey arrived at STAR around 9 am to measure the FST with respect to the STAR IP. They measured the centroid of the third FST plane, the survey holes on each side of this planes metal structure, and the four rails connected to the rest of the FST system for installation.

The survey team finished up around 10:30 am. We will need to contact Matthew Ilardo (milardo@bnl.gov) for the results.

The FST group returned to STAR at 2pm to meet with the vacuum group and Rahul. It was agreed that another layer of Aluminized Mylar foil would be put over the beam pipe at 10:30am on Wednesday (01/25).

The FST group returned to STAR at 5pm to put up black plastic and test the FST. Voltage of sensors was ramped to Physics setting and pedestal run was taken.

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24024013(1).pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24024013.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24024013.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24024013.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2] 

01/23/2023 (Monday)

Cooling system reservoir level dropped to ~60.1% at 5:40am. Current leak rate ~0.62%/day, but that seems biased by the first stable point after the system was filled on 01/15. 

Summary slides for FST work since 01/14/2023 from 8:30am FST meeting: STAR/system/files/userfiles/6551/20230123_WorkSummaryUpdate(2).pdf

The FST group returned to STAR at 12pm to meet with the survey group. We met with Matt from the survey group and they inspected what needed to be done for the survey. Matt will find the old file to see what they did last time for the survey. Survey group will return tomorrow morning at 9am to perform the survey, which will take roughly 3-4 hours. The FST group will arrive at this time as well to see if any help is needed. 

The FST group arrived at STAR again at 5pm to try swapping the purple cable for the module corresponding to MPOD03 u106 & u107. The corresponding module has a strange noise pattern compared to other modules.

Black plastic was put up to block light on the FST sensors again.

We swapped purple cable on PPB A8 (MPOD03 u106 & u107) with cable on PPB A7 (MPOD03 u104 & u105). The noise pattern follows the cable and the module.

The plots above show the pedestal, total noise, and random noise (left to right) for the two modules with purple cables swapped. It can be seen that the noise pattern follows the purple cable + module combination.

We swap the cables on PPBs A8 and A7 back to their original position, ramp to physics voltage, and take a pedestal run. Pedestal and noise match December 2022 values. 

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24023002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24023002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24023002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24023002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/22/2023 (Sunday)

No drop in cooling system reservoir level.

FST group arrived at STAR at 1pm. 

Blue tarp was no longer in proper position, adhesive of tape not strong enough. Remove blue tarp for today's test. Black plastic is still in place. 

Turn off as many lights as possible and then ramp up FST HV to Physics mode with shift crew GUI. All channels ramp to proper voltage with no errors or trips.

Wait 20 mins for electronics to warm up.

Take pedestal run with 2000 events. Everything looks good.

Turn all lights back on and remove black plastic cover in preparation of survey team coming tomorrow (01/23).

The plan for tomorrow (Monday, 01/23) is to head to STAR after the morning 8:30am FST meeting and see if survey is working / when they will arrive. The hope is that they will finish sometime in the afternoon, after which, Bill can install the black FST partitions. We will then adjust the cables and cooling lines to move them away from the projected active FST area. We will run pedestal runs to ensure everything is working properly and then continue to monitor the FST over Tuesday (01/24) and Wednesday (01/25). 

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24022003.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24022003.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24022003.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24022003.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/21/2023 (Saturday)

No drop in cooling system reservoir level.

FST group arrived at STAR at 1pm.

Turn off as many lights as possible and then ramp up FST HV to Physics mode with shift crew GUI. All channels ramp to proper voltage with no errors or trips.

Wait 20 mins for electronics to warm up.
- MPOD01 u1:     160V with 17 μA current.
- MPOD01 u101:   70V with 27 μA current.
- MPOD01 u207: 120V with 36 μA current.
- MPOD03 u0:     120V with 36 μA current.

Take pedestal run with 5000 events. Everything looks good.

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24021002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24021002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24021002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24021002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/20/2023 (Friday)

The cooling system reservoir level dropped from ~65.4% to ~63.2% at about 10:33pm on 01/19/2023. We will continue to run the cooling system until the end of our work and compute the leakage rate after another drop in the level.

We arrived at STAR at 10am to help remove the large tables used for the FST during the beam pipe bake out. Bill and Prashanth led the work and we were finished by 10:30am. The small table below the FST remains in case the survey group needs it for something. The blue tarp was removed during the process and the FST group placed it back over the FST afterwords. 

 

The FST group arrived at STAR at 5pm.

The voltage limit for MPOD03 HV channel u0 was increased to 120V from 100V and the current limit to 50μΑ from 30μA.

1st restart of MPOD IOCs and ramp up to Physics:
- MPOD03 u0, u107, u207 turn on but do not ramp.
- MPOD03 u100 turns on but reports -nan.
- MPOD01 u1 trips after current limit changed for the resting Physics on setting (not ramping). It reached 33μA with the ramping current limit of 50μA, then the current limit is set to 30μA after ramping. After a minute or so, the current drops back down to 20μA. We decided to increase the current limit to 50μA after ramping for the time being.

2nd restart of all MPOD IOCs and ramp up to Physics:
- MPOD03 u105 turned on but did not ramp up.

Restart just MPOD03 IOC and ramp up to Physics:
- Everything works just fine.

Wait for electronics to warm up to take a pedestal run.
Pedestal, noise, and random noise values on each module looks good.

The plan for tomorrow is to arrive at 1pm to take a pedestal run.

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24020002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24020002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24020002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24020002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/19/2023 (Thursday)

No drop in cooling system reservoir level at 10:26am since filling on the morning of 01/15/2023.

Plan: Arrive at STAR around 4pm, when most people have concluded work in the STAR experimental area. This will allow us to turn off all of the lights for FST testing for an extended period of time. The MPOD IOCs need to be restarted following the expert GUI manual and the shift crew GUI will be tested for ramping up the voltage on all sensors. The goal is to have no trips and all sensors at their Physics mode value as there was an issue on 01/18/2023 where some channels did not ramp. Then a pedestal run will be taken and should be consistent with the December 2022 results. Cooling will also be monitored. 

The FST group arrived at STAR at roughly 4pm. Oleg Tsai needed the lights on for another hour so we delayed our testing.

We spoke to Rahul Sharma about the plan for the next week. He submitted a survey request for the FST as of today. The request is for Monday, 01/23. Tomorrow, Friday 01/20, the survey group has a meeting about how to allocate resources and they will decide when they can do the survey. 

The FST group will arrive at STAR tomorrow (01/20) at 10am to help remove the installation tooling on the TPC face. The metal table used for installing the FST will be removed after the survey group is done.

Testing of the FST began at roughly 5pm and the main goal was to get the shift crew GUI to ramp all channels to their designated values with no trips. We also wanted to make sure it worked consistently each time the FST was turned off and on. To do this, we needed to repeatedly restart the MPOD IOCs. I will outline each IOC restart and ramp up of the FST voltage below. All lights were turned off in STAR hall except the overhead lights like before.

1st restart of MPOD IOCs and ramp to Physics:
- MPOD01 u1 tripped (NEW), this was fixed by reducing the voltage ramp per step for that channel to 10V/step from 20V/step.
- MPOD02 u0-u7 only ramped to 20V.
- MPOD02 u101 did not ramp but turned on.

We manually ramped the channels to their proper operating voltage. 
- MPOD01 u1 at 160V had a current of 18μA with a max current of 30μA allowed. This current slowly dropped to this value after ramping manually. 

The group took a pedestal run and values were consistent with December 2022.

Run I-V scan for MPOD03 u0 again on request from Zhenyu. We had this HV channel set to 100V, but this time the HV scan goes up to 120V. 

The max current on this channel is 40μA currently, but we may be able to increase to 50μA to run at 120V.

2nd restart of MPOD IOCs and ramp to Physics:
- MPOD02 u204 did not turn on or ramp
- MPOD03 u107, u206 turned on but reported -nan for voltage

Ramp down voltage and ramp back up to physics:
- same exact issues (need to restart IOCs)

3rd restart of MPOD IOCs and ramp to Physics:
- MPOD01 u101 only ramped to 65V for some reason

4th restart of MPOD IOCs and ramp to Physics:
- MPOD02 u104 and u207 did not ramp

5th restart of MPOD IOCs and ramp to Physics:
- MPOD02 u204 did not ramp

Just restart MPOD02 IOC to address u204 not ramping.
- Issue is fixed, meaning we do not need to restart all IOCs each time an issue occurs after changing the alarm handler values and restarting all IOCs at least once.

Work concluded around 6:30pm. 

The plan for tomorrow, Friday (01/20), is to help with the FST installation tooling removal at 10am and talk to Rahul about any updates to the schedule. We will return at 5pm to address any tasks or changes we would like to make to the FST. A pedestal run will be collected and the cooling will be monitored. We are returning late since people have been concluding work slightly before 5pm each day and it is an optimal time to turn off all of the lights for testing the FST.

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24019002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24019002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24019002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24019002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/18/2023 (Wednesday)

The FST group arrived at STAR around 11am to arrange the insertion of FST into TPC cone. 

Bob, Bill, and Rahul were present and it was decided the FST was able to be pushed in that morning. They removed the black plastic and tarp, and then lowered the FST metal platform slightly to avoid any sliding conflict. Bill and Bob pushed in the FST and removed the installation tooling.

The work concluded near 11:45am.

The FST group returned at 1pm. Bill helped put up the black plastic and blue tarp over the TPC cone opening the shield the FST from light during testing. All lights were turned off again with the exception of the four "emergency" lights. 

Prior to ramping to the previous HV values from the end of Run 22, the group decided to modify the default config file for the FST to the values used in other pedestal tests over the past few days. They also updated the alarm handler values for the IOC. This was performed as instructed by the expert GUI manual.

Changes:
MPOD01 u101: 160V --> 70V, current limit 30μA --> 40μA
MPOD01 u207: 140V --> 120V
MPOD03     u0: 120V --> 100V 
Each of these channels had their ramp step changed from 20V to 5V.

The changes did take effect for the shit crew and expert FST GUIs. When using the shift crew GUI to ramp up all channels simultaneously there were a few issues with channels not ramping, but these were not trips.
These channels were:
MPOD02 u101
MPOD03 u1, u207

These channels were then ramped up to their proper operating voltage manually. To fix this issue, the group plans to restart the MPOD IOCs on 01/19/2023. This is listed as the fix in the expert manual if something goes wrong.

A high voltage scan was conducted for the 3 channels with lower operating voltage than the end of Run 22 from 0V to their current operating voltage. When conducting the scan, MPOD 01 u101 was ramped down and MPOD01 u102 tripped directly after, which is not currently understood. u102 does have an operating voltage of 70V compared to the standard 140V on inner sensors.

The work was concluded around 3pm as others needed to work in the STAR experimental area before the end of the day. 

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24018002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24018002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24018002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24018002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/17/2023 (Tuesday)

The cooling system reservoir level remained the same.

Dmitry helped get the online monitoring for the FST MPOD crates working.

Bill Strubel placed black plastic on both ends of the TPC openings to block light from hitting the FST. He also placed a blue tarp material over the black plastic on the FST side, which was used to block light in the original installation of the FST in Summer 2021. 

  

All of the lights were off in the STAR experimental area except for the four large lights near the ceiling. These lights are essentially emergency lights are require you to use a circuit breaker to turn them off. 

MPOD01 u101 still trips. This channel is set to 70V again with 40μΑ current limit.
MPOD01 u207 set to 120V with a measured current of 39μA. 
MPOD03 u0 did not trip on ramp up to 120V (37μA), but was very close to the current limit (40μA) and therefore was ran at 100V for safety and stability. 

Pedestal, noise, and random noise look identical to the Winter 2022 results for all disks and modules. 

The FST group returned later that night to attempt swapping the purple cable corresponding to MPOD01 u100, u101 on PPB C2 with the cable for MPOD01 u102, u103 on PPB C1. This was to isolate the issue with this HV channel. After ramping up all the channels, it was found that the high current followed the HV channels for the cable originally on PPB C2 (u103 was now tripping before reaching close to 100V). Thus, the purple cable + module is the issue and this cannot be fixed with the FST in its current position. 

Summary:
Original setup:
MPOD01 u100, u101 (PPB C2)           MPOD01 u102, u103 (PPB C1)
 - trip on u101 with HV < 100V
 - u101 at 70V with 31μA
Modified setup (swap):
MPOD01 u100, u101 (PPB C2) <--> MPOD01 u102, u103 (PPB C1): 
 - trip on u103 with HV < 100V
 - u103 at 70V with 31μA

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24017002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24017002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24017002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24017002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/16/2023 (Monday)

The cooling system reservoir level remained the same.

The FST group arrived at STAR at 8:30am to assess the tripping HV channels. They disconnected and reconnected the purple cables from their corresponding PPBs for modules on MPOD01 channels u101 and u207. This did not fix any issues with tripping. 

The group was able to manually increase the trip limit of MPOD01 u101 from 30μΑ το 40μA and then ramp this sensor to 70V with a current of 33μA. MPOD01 u207 was manually ramped to 140V with a 60μA current, but eventually tripped as it was not stable (set back to 120V). MPOD03 u0 was set to 100V with a 28μA current.

The settings for the pedestal test for the troublesome channels were as below:
MPOD01 u101: 70V
MPOD01 u207: 120V
MPOD03     u0: 100V 

It was noticed that FST error code 2 persists on the shift crew GUI even after the channel is tripped.

When taking pedestals, Disk 1 module 11 was found to have higher noise than December 2022 when taking a pedestal run with 1000 events. Using 3000 events removed this difference.

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24016002.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24016002.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24016002.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24016002.pdf
- In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

01/15/2023 (Sunday)

The FST group went to STAR around 8am to fill the cooling unit again. It was filled to 75.9% and then the level reached  65.3% after turning on the unit.

Then the group began connecting the purple cables to their corresponding PPBs on the TPC face. The crates were turned on and they attempted to use the shift crew GUI to ramp up the voltage. 

There were 6 tripping HV channels:
MPOD01: u101, u207
MPOD03: u0, u1, u3, u4

It was decided to perform a power cycle of the FST crates manually and the screws on the FST purple cables were all tightened down for a better connection.

There were 3 tripping HV channels after this:
MPOD01: u101, u207
MPOD03: u0

We found that sometimes clearing the trip on MPOD03 u0 would cause all other HV channels on that same MPOD and Bias to trip.

MPOD03 u0 was changed from 120V (End of Run 22) to 100V with a current of 28μA.

MPOD01 u101 was changed from 160V (End of Run 22) to 50V.
MPOD01 u207 was changed from 140V (End of Run 22) to 120V.

We took a pedestal run and the levels for the pedestal, noise, and random noise for disks 1 and 2 were consistent with results from December 2022 before the FST was removed from the TPC cone. Disk 3 had higher values which we can now attribute to excess light present on that FST plane. 

The day's final pedestal results and comparison with December 2022.
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_24015004.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_24015004.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_24015004.pdf
Comparison: STAR/system/files/userfiles/6551/fst_23361007_24015004.pdf 
 - In the plot titles, the first number is the module number [0,35] and the second is the time bin [0,2]

Zhenyu Ye went back to UIC on this evening.

01/14/2023 (Saturday)

The FST group (Zhenyu Ye, Ziyue Zhang, Te-Chuan Huang, and Gavin Wilks) arrived at BNL and began work at 2pm. The goal of this day was to assess what needs to be done for the FST re-installation. It was decided that the cooling could be reconnected and the system could be turned on the check for leaks. 

The group reconnected all cooling lines and filled the cooling crate reservoir to roughly 80% from an initial 11%, and after the NOVEC fluid filled all of the lines, the level was at 39%. It took several attempts to get the pump started without triggering the flow interlock. To fix this problem, some FST cooling loops were closed (at most half of them at once) to allow the fluid to push the air out of the lines. Then these were opened and the other half were closed.

The final flow values were:
Return: 5.95 L/min
Supply: 7.39 L/min

These values match what was recorded in December 2022 prior to the removal of the detector.

The team will continue to monitor the cooling until their departure from BNL.

December 2022

Blog for work conducted by the FST group at the end of December 2022: https://drupal.star.bnl.gov/STAR/blog/zyzhang/2022_2023Winter-FST_Operations_log

Final Pedestal Files before pulling the FST out of TPC cone:
Pedestal: STAR/system/files/userfiles/6551/fst_pedestals_23361007.pdf
Total Noise: STAR/system/files/userfiles/6551/fst_totStdDev_23361007.pdf
Random Noise: STAR/system/files/userfiles/6551/fst_ranStdDev_23361007.pdf