April 2016

STAR Newsletter

April 2016 edition

Contents:

Event Plane Detector Activites
From the Spokesperson
News from Software & Computing
Operations Activities

A note from the editor: as a collaboration-wide communication tool, this newsletter is set up to allow comments (subject to moderation against abuse), and all STAR Collaborators are welcome to do so! Please keep in mind that some content (including all comments) may be considered internal to the Collaboration and only accessible when logged into Drupal. Documentation is available here.

Event Plane Detector Activities
(Sierra Garrett - Research Assistant, LBNL)

Figure 1. Rosi Reed and Jinlong Zhang working on the EPD prototype at BNL before its installation at STAR.

Figure 2. Sierra Garrett and Jinlong Zhang assembling the EPD prototype at LBNL. The scintillator and embedded WLS fibers can be seen in the lower right hand corner.

The research and development stage for the Event Plane Detector (EPD) Upgrade for BES-II is coming to a close and construction on the final detector is set to begin in the coming months. The EPD is designed to replace the BBC and provide improved event-plane resolution and centrality measurements outside of the TPC acceptance. It will also be important for the STAR Fixed Target Program by providing full coverage to beam rapidity and extending STAR’s physics reach to higher baryochemical potential. For more information on the physics purpose and EPD requirements please refer to Mike Lisa’s presentation at the STAR collaboration meeting in January 2016.

The EPD will consist of two disks made from 1.0 cm plastic scintillator installed 3.75 m away from the center of STAR on the east and west sides. The disks will extend from an inner radius of 4.5 cm to 90.0 cm and will be separated into 24 azimuthal sectors, each with 16 optically separated radial tiles. Light signals in the scintillator will be collected by 1.0 mm wavelength shifting (WLS) fibers embedded into each tile and connected to 1.2 mm clear optical fibers for transporting the signal to be read out by silicon photomultipliers (SiPMs) a short distance away. Currently, a prototype representing one sector of the full detector is in place at STAR and is successfully collecting data. Details about the prototype and the construction requirements of the EPD can also be found in Mike’s presentation.

At this time, the details of many aspects of the EPD - including the designs of the sectors and the components of the detector - are being finalized by various institutions. While the layout of the larger tiles is complete, tests to determine the optimal pattern for embedding the WLS fibers into the smallest tiles are being administered at Lawrence Berkeley National Lab. Research into an efficient and effective way of polishing the optical fibers is being conducted at Lehigh University in addition to assembling the fiber connections.

Radiation hardness tests on the optical epoxy used in the detector’s construction will be performed in early April at LBNL’s 88-Inch Cyclotron. Kent State University has concluded the physics simulations but calculations for the lead converter and the expected radiation level of BES-II will be completed by the University of Tsukuba.

The electronics and mechanical support structure fabrications are also in progress. Indiana University is designing the readout electronics which will be similar to the existing FPS and will include QT boards, TUFF boxes, etc. provided by the STAR DAQ, electronics and trigger groups. The boxes for protecting the electronics will be built by Ohio State University. The construction of the frame and mountings for the detector and its various components is spearheaded by Brookhaven National Lab.

Once these various tests and designs are completed construction on the final detector will begin with participation from OSU. At this time we expect to have 1/8^th of the full detector, or 6 sectors, installed at STAR by the end of 2016. This will allow us to commission the detector during Run 17.

Figure 3. The EPD prototype before installation with the clear optical fibers attached to the WLS fibers via custom 3D printed fiber connectors.

Figure 4. Mike Lisa, Jinlong Zhang, Rosi Reed, Isaac Upsal and Alex Schmah at BNL preparing to install the EPD prototype.

From the Spokesperson
(Zhangbu Xu - Spokesperson)

Dear STAR Collaborators:

This spring, the collaboration has been busy. We have the ongoing RHIC Run 16 with Au+Au collisions at 200 GeV, the last and biggest dataset with HFT, and double statistics with MTD for the heavy-flavor program. The collaboration has made many improvements in the detector, trigger, DAQ, and firmwares in striving to achieve our goals with the constraining 10 weeks of data-taking instead of the requested 13 weeks.

We have discovered an issue with HFT firmware, which scrambled the PXL readout in the first 3.5 weeks of Run 16 data-taking and Run 15 p+p and p+Au datasets. The root cause of this is still under investigation and the physics impact is being evaluated. The effect is a reduced single-track efficiency with PXL hits included, and unknown bias on two-particle decays. To be conservative, we have assumed that the lost efficiency cannot be recovered, and calculated the good events accordingly for this Run. We have assessed the physics impact this issue has on the Run15 programs. The preliminary conclusion is that over the last 3 years, we have developed a few analysis techniques (BEMC high-tower trigger on charged hadrons for high-p_T D⁰, and combinatorial method for low-p_T D⁰) to mitigate the intrinsic difficulties of operating and analyzing HFT data in small systems. These developments have been successful and we will not lose any major physics goals despite the PXL firmware issues in Run 15. A full report is expected to be available very soon. I would like to thank the experts who have taken this issue very seriously and have worked hard to investigate and correct the firmware for the normal data-taking immediately after the issue was discovered.

We have conducted reviews for two proposals:

Endcap TOF in collaboration with CBM,
the report from the committee chaired by Bill Llope is available here
Event Plane Detector for BES-II,
the report from the committee chaired by Akio Ogawa is available here

I would like to take this opportunity to congratulate the proponents for their excellent work and for presenting compelling cases for the upgrades. Additionally, I would like thank the committees for their dedication and expertise on evaluating the proposals. We are in the process of reviewing another proposal of post-shower for the FMS. We anticipate installing the detector (similar to the FMS pre-shower) in time for the Run 17 pp500 Spin program.

One of the exciting and unexpected events from recent weeks is the addition of Run 18 in order to accommodate all the must-do programs at RHIC. The main program for Run 17 is the sign-change in Spin measurements between p+p and DIS collisions (HP13), and for Run 18 is likely to be the isobar collisions (⁹⁶Zr+⁹⁶Zr and ⁹⁶Ru+⁹⁶Ru) to constrain the chiral magnetic effect (CME). However, we should explore other programs as well and articulate their importance. One of the possibilities is the global polarization measurement of Λ(Λ) with an order of magnitude statistics increase at 27 GeV. This brings us to the ongoing effort on Beam Use Request (BUR). The information and weekly updates are available here. We are looking forward to input from all collaborators to the writing committee. The schedule and membership of NPP Program Advisory Committee (PAC) are available at: https://www.bnl.gov/npp/pac.php.

Congratulations go to Dr. Amilkar Quintero from Kent State University for successfully defending his thesis, titled: "Measurement of Charmed Meson Production in Au+Au Collisions at √s_NN = 200 GeV". I would like to take this opportunity to thank his supervisors, Prof. Spiros Margetis and Dr. Flemming Videbaek, for their effort in educating the next generation of young scientists and contributions to the STAR scientific program.

News from Software & Computing
(Jérôme Lauret - S&C Leader)

National Lab Day / Event Display
Data-Taking, Data Samples
Production News (p+p launched)
Recent Embedding Reshape

National Laboratory Day - BNL

National Laboratory Day, a yearly event called upon by the U.S. Department of Energy (DOE), was held this year on Capitol Hill on Wednesday, April 20^th. BNL's showcase was located in the "universe" theme area of the exhibition, and as you may know, included a live STAR event display (which continues to be available here), a noticeable addition to BNL's booth as seen in the picture on the right.

STAR collaborator Paul Sorensen drew passers by in with the attention-attracting display of RHIC collisions and then conversed with them about the goals and benefits of nuclear physics.

"The exhibit that included the STAR event display was a representation of not just STAR, not just RHIC, and not just Heavy Ions, but all of nuclear science. The fact that the live STAR event display was chosen out of many possible displays to represent nuclear science reflects very well on the STAR Collaboration and on the people within the Collaboration who made it possible. Fellow STAR collaborator Bob Tribble, BNL's Deputy Director for Science & Technology, was also in attendance." -Paul Sorenson

A note that our display showed near-real-time events as we delayed the maintenance day to Thursday (many thanks to the cooperation from the operation group and CAD for this re-scheduling). Last but not least, many thanks go to Dmitry Arkhipkin (S&C) for creating this new event display, superseding the display we had under Valeri Fine's time (but no longer operational). Revamped leveraging newer technologies and allowing visualization with the sole requirement to have a recent Web browser (with WebGL support), the new event display allows a broader outreach.

Data-Taking and Samples

From an offline perspective, Run 16 is moving along very nicely. After the machine problem [see the Operations Activities contribution], we are now back to a steady 93 TB of data per day (~38 tapes and 22015 files per day) as shown in the below monitoring graphs.


Data size / day (TB)	# of files / day

Another way to look at the data is to see how much used tape space we have accumulated so far. The graph below represents exactly that:

So far, we have accumulated ~5 PB of data and we are still far from the end of the Run. To give an idea, we were at about the same data-set size by July in Run 14 (as illustrated in Data set projections for Run14), and we accumulated 11.8 PB total in Run 15. As a side note, our storage space in HPSS continues to be sustained by operation funding. However, due to a freeze of RACF equipment funds, our central and distributed storage spaces have essentially reached saturation.

We recently acquired 425 TB of central storage which was all allocated to production buffering. Our needed storage space (projected) for 2016 was 5.8 PB of live storage (distributed). The currently added storage will barely serve as a temporary storage, not a permanent nor a sustainable storage for all data productions (in fact, if we reach a point where we have no storage left, data production will need to be stopped).
The Xrootd space is at 95% full as shown by the below graphs - despite far-too-small captions, the illustration is dramatic: our Data Management system has evenly spread data around on all nodes, however there is no space left anywhere (apart from a few recently deceased disks/nodes). The black bars represent a break of the node number sequence; blue is the occupancy. The bottom line being that we will have to either (a) acquire more disk space [chances are slim], (b) replace some datasets with more compact format, (c) rotate datasets (Analysis Train and the like), or ... (d) stop data production.

CAS	CRS

Data Production News

The data production priorities have been set in collaboration with the PWGC. Balancing those priorities with practical & optimal utilization of resources, the current order of productions is:

Restart Run 15 pp with PPV/W-mode
- Status: The first samples are already available on NFS as noted by Lidia Didenko (our Production Manager). We are eager to see the subsequent QA. We are also facing a farm saturation issue unrelated to the data sample (Condor put many jobs in the HELD state and they are never released); however, this may be resolved (TBC).
At the same time significantly scale down the resources for the Run 14 HT production.
At the same time prepare and roll out the Run 13 pp test productions which involve PPV/W-mode+StiCA on a few fills w/ and w/o HFT.
- Status: several test samples were already produced as noted in starsoft. An evaluation summary was presented by Devika Gunarathne (Temple) and Salvatore Fazio (BNL) as shown in You do not have access to view this node. The work was thorough and confirms past findings. Code integration issues are being sorted in.
Prepare and roll out (at a similarly lower scale of priority) a Run 14 reproduction of 150M events with the HFT bug fixed (and VF parameters fixed for a typical Au+Au 200 GeV).

A graph of running efficiency (thanks to Lidia) is shown on the right. The bulk of all jobs have a ratio close to 1.004 (i.e. >99% efficiency within one sigma) - we will try to understand the tail at a later time (but have already identified slow access to the database which may affect this behavior).

In other news, we have been trying to leverage resources available at our institute in Dubna using a Grid-based workflow similar to the one we used at the KISTI/Tier-1 center in 2014. While the results are preliminary and a statement would be premature, let us say that the testing is encouraging and moving to a test at scale. If this effort confirms satisfactory usability, the promise of thousands of additional CPUs, in addition to other supplemental resources also being investigated at NERSC, could very well put our global computing resources inline with the requirements projections (storage will however, and as already hinted above, remain a deep challenge).

Embedding Team Consolidation

During the You do not have access to view this node for S&C at the last STAR Collaboration Meeting, the overall constituency of the embedding team was presented. I reminded all that it is rather crucial to keep the team properly staffed as several (worrisome) vacancies appeared.

I am pleased to note that the embedding team was joined by Kunso Oh as Embedding Deputy (ED). I am also grateful to the Physics Working Coordinators who helped identify new Embedding Helpers, and namely Zachariah Miller (UIC) for the HF PWG, Leszek Adamczyk (AGH) for UPC, Zillay Khan (UIC) for the Jet-Corr PWG, and Usman Ashraf (Tsinghua) for the LFS PWG. I am even more so to the Helpers who accepted to take upon this important task for the year(s) to come - without their help and the relentless work from the Embedding Team, our science deliverables would be from delayed to impossible to achieve. Note that our Organization page is constantly updated and up-to-date, reflecting the current status of all S&C activities. We should now be properly staffed (we are still missing one ED) and while we have a slow-down in embedding requests at the moment, it was important to re-staff the team right away. Indeed, we are currently ramping up Run14 and Run 15 ongoing analyses. More importantly, with several major incoming conferences such as SQM 2016, Hard Probes 2016, and QM 2017, we expect to see a large number of requests in the next few months. A further reminder: a list of incoming conferences is kept by the STAR PAC (Frank Geurts) @ Conferences.

Operations Activities
(Oleg Eyser - Period Coordinator,
Bill Christie - Operations Leader)

The failed diode (the vertically oriented cylinder) in situ [photo from Gary McIntyre's presentation].

RHIC Run 16 is nearing the completion of its third month since physics was declared on February 7^th. At this point we had expected to be well into the energy scan of d+Au collisions already. However, on the morning of March 18^th a problem developed in one sector of the RHIC blue beam that was identified as a short in the quench protection diode of one dipole magnet. The replacement of this diode required a whole section of the blue ring to be warmed up and the end volume of the magnet to be cut open. This kind of repair work has only been done once before at RHIC about 15 years ago and it required a tremendous effort from many groups at BNL. A detailed presentation of the different steps of the diode replacement can be found in Gary McIntyre's slides (including numerous photos, such as the one on the left showing the cylindrical diode in situ).

The diode failure was diagnosed to be a result of chronic and acute radiation exposure in sector 10 from the protection bumps in both yellow and blue beams. The protection and compensation bumps in blue have now been moved to other sectors, and all bumps that have previously pointed outwards of the ring are now pointing inwards, thereby avoiding any direct irradiation of the diodes.

Altogether, owing to the tremendous and successful efforts of a number of groups and people inside and outside of C-AD the entire interruption to beam operations for RHIC was kept down to just 19.5 days, including warm up and cool down of RHIC sector 10 and additional machine setup for Au+Au collisions. During the extended downtime, we collected several days of additional cosmics data with inverse polarity of the magnetic field ("A", or "forward field" polarity; regular running has been with "B", or "reverse field" polarity for a few years). This will allow for detailed studies of field distortions and help improve the TPC tracking and alignment of the MTD.

With RHIC back online, the discussions between the RHIC experiments and the Directorate management resumed at the weekly RHIC Coordination meeting held on April 12^th. How to adapt the Physics running plan for this year's RHIC Run to account for the days that the physics program was offline, and hence the plan for the rest of the run was the topic for this meeting. STAR requested a two week extension (beyond the original ten weeks) so that the HFT data set goal could be met, accounting for the significantly reduced PXL efficiency in the first three and a half weeks of the Run resulting from a firmware issue [see the From the Spokesperson contribution]. PHENIX asked that all of the remainder of the Run be used to deliver four short runs with d+Au collisions (200, 62, 39, and 19.6 GeV). What came out of the meeting was a decision that the current Au+Au running be extended to make up for the 19.5 days the program was offline. This pushed the earliest changeover date from Au+Au to d+Au running until the vicinity of May 7^th.

There was also discussion of whether the length of the current RHIC Run, which was scheduled to cease beam operations on June 3^rd, with the Cryo warm up complete by June 7^th, could be extended further into June. Due to the reduced power cost for the period when RHIC was offline, and the low cost of power through the month of February, it was determined that the Run could be extended by at least two weeks. Once the March power cost would be known (usually about mid-month the following month) a final determination/decision would be made whether the Run can be extended by three weeks.

The experiments were asked to provide answers to specific questions to the Directorate Assistant Lab Director (Berndt Mueller), for further discussions on exactly how the remaining run time between May 7^th and the end of the Run will be allocated.

For a heads up as far as planning goes, with a two week extension the beam operations would cease on Friday, June 17^th, with the Cryo warm up complete by June 21^st. If the three week extension would occur, beam operations would cease on June 24^th, with the Cryo warm up complete by June 28^th.

At the RHIC Coordination meeting on April 26^th, the experiments gave their answers to the questions, and STAR again asked for an extension of the Au+Au running by 1.5 weeks, so that we could reach the HFT data set goals. The decision was made to extend the Run by a total of three weeks, and Berndt decided that the Au+Au running would cease on Monday morning, May 9^th, an extension of two days. The remainder of the Run will be used to run the d+Au energy scan.

On another note, at the time of the last Newsletter we were still working on the "Plan B" implementation of improving the precision of the vertex cut used as the base trigger for the HFT program. This effort is important to get as close as possible to the original goals of the program within the available beam time. It was commissioned and online about March 17^th.

The bunch intensity in RHIC is nearly maxed out at around 2.3e9 ions. With the displaced beams at STAR, we are seeing a ZDC coincidence rate of 70 kHz for six or more hours per fill with stable rates for the HFT program. STAR is currently running extremely well. We are pushing the detector to its limits, taking data at event rates a bit over 1800 Hz, which corresponds to about 1.8 GB/s. And the shift crews are doing a very good job of getting STAR online quickly after the declaration of "Physics ON".