Information, Requirements and Instructions for Fast Offline Quality Assurance Shifts for Run 24

Introduction and Requirements:

Welcome to the STAR Fast Offline Quality Assurance Shift service.  There are no new shift crew requirements or browser features for Run 24 compared to Run 23. Run 24 will be dedicated top+p 200 GeV collisions for 12 weeks with about 6 weeks of Au+Au collisions at 200 GeV until the end of the run (Oct. 7). The new detectors for the forward physics program will be included. Most will likely be part of the fast-offline QA shift work. The forward detectors are the FST, FTT, ECAL and HCAL and together constitute the Forward Tracking System (FTS) and the Forward Calorimetry  System (FCS) respectively. Please familiarize yourself with the QA shift plots before starting your shift work.

The purposes of the Fast Offline QA shifts are to monitor the validity of the data, calibrations and event reconstruction from the full experiment and to provide near, real-time feed-back to the experiment as needed. An equally important purpose is to compile a record, in the form of reports, of QA issues associated with each run for later data filtering and diagnostics prior to physics analysis. From a practical standpoint it is only possible for Offline QA to detect fairly large problems, and in so doing we continue to strive to examine all of the data collected.

• No programming skills are required. All the tasks are web based "point-and-click" activities.

• Please use RACF's Mattermost Offline QA chat mechanism for recent QA information and updates and for communicating with the QA experts and with the detector experts during your shift. You may also send email directly to the QA experts or the detector experts. The STAR Mattermost Offline QA link is here: Mattermost.
  

• General knowledge of the STAR detector, reconstruction methods and calibration issues are necessary because the purpose of this work is to spot problems with the hardware and event reconstruction, or with the calibrations. Expert level knowledge is not required however.

• In the past, QA shift crew without previous experience were required to do their first QA shift week at BNL. However, due to travel limitations and on-site access restrictions the QA team is offering an online training video for collaborators who have not done QA shifts previously. Please notify Gene van Buren as soon as possible if you need this training. Good internet connection and access to the STAR web and protected areas are essential.

• Subsequent Offline QA shifts may be done from non-BNL sites provided adequate web access to the Auto QA system can be demonstrated from the remote site.

• The Offline QA shift may be done any time throughout the day but it is expected that a serious effort will require at least 8 hours a day.

• Please refer to this and the other web based documentation as you prepare for the shift work and during your shift. If you have further questions please send email to Lanny Ray, ray@physics.utexas.edu and/or Gene van Buren.

Welcome aboard!

Responsibilities of the Fast Offline QA shift crew:

• Using the Automated QA browser review the QA Shift histograms for Fast Offline Data Production (highest priority) for all available runs which have not been examined for each trigger stream (st_physics and possibly others are expected in Run 24) and for each trigger group (e.g. general, minbias, high-tower, central etc.). You may combine the "general" QA results and the "Minbias" QA results into one report for each run number, trigger stream and trigger group.
• Complete a useful and informative Offline QA Shift report using a web-based form noting especially any and all suspected problems with the detectors, calibrations, and reconstruction. The report will be archived and the summary sent to 'starqa-hn' hypernews automatically. Please use the "play" mode if you are a first-time user to practice filling out the report.
• Review the Online Run Log and Electronic Shift Log book information and comments for each data run examined and summarize the Run/Data Quality status by marking the job as "Good" or "Bad." This will also indicate that the data have been examined by Offline QA. Jobs will normally be considered "Good" even when there are hardware outages or calibration/reconstruction issues. Please check with the QA experts before marking jobs as "Bad."
• Notify the appropriate experts and/or the QA contacts for any and all suspected problems with the detectors, calibrations, or fast-offline reconstruction.

Instructions for the Fast Offline QA Shift:

Getting Started:

• Go to the STAR Computing Offline QA home page on drupal (i.e. from the STAR Home Page select "Computing" in the left panel, then select "Offline QA" in the table row labelled "Production" or go directly to the Offline QA home page and open the Auto QA browser by clicking on the "Automated Offline QA Browser" button in the upper portion of the page. You will have to enter the STAR protected area username and password. Contact your local STAR council representative, Lanny Ray or Gene Van Buren if you do not know it.
• If the browser fails to open contact the QA Experts ASAP.
• Enter your RCF username.
• Select button 2.1 (shift work) and hit OK which takes you to the page where you may select data runs to examine. Note that for Run 24 the Offline QA shift crew are only responsible for the fast-offline production (button 2.1).

Selecting Data to examine:

• For the Fast Offline Production QA (Button 2.1) on the next page (Page 22) select the data grouping method using buttons (A) - (D). Generally the Auto-combined grouping is preferable as this combines all available files for the run to achieve the best possible statistics. Grouping method (C) allows the user to arbitrarily combine data from any of the available runs. Note that the ZDC coincidence rates are listed for each run/file sequence. Ideally multiple jobs should only be combined for similar ZDC rates as some QA histograms depend strongly on the amount of background and pileup which are affected by luminosity.
• Then select the job listing order (applies to data grouping options B and C only), the date and run number ranges and click OK. 
• On the next page select the run (or the combination of runs for grouping option C) to be examined. Priority should be given to the most recent data that have not been examined yet. Click OK.

Examining the QA Histograms:

• The next page provides access to several new features available since 2012.  You should examine all histograms visually, including all listed trigger groups, and file a report for each trigger group.
• There are numerous "Help" buttons, generally located in the upper-right of any given panel, which present instructional information in the context of what is being viewed at that moment.
• Automated QA testing is available and can be used, provided a suitable set of reference histograms are ready.  These generally take about a week to load once stable physics data production and reconstruction have been achieved and are updated throughout the run period. If you wish to use the automated QA feature please select a reference data set which best matches the data conditions using the left arrow buttons to move from field to field.
• The default "QA Shift" histogram group is sufficient for shift work.  However, the entire set of QA histograms can be selected with "All."
• Links to the Run Log and Electronic Shift Log book for the selected run are at the bottom of this panel.
• Select "Plots only" to view the data only, or "Analyze" to view both the data and reference and to get the results of the automated comparisons with the reference. This option can be used to easily compare the histograms to a reference and enables a convenient way to attach example histograms to QA issues (see instructions: www.star.bnl.gov/devcgi/qa/QAShiftReport/refHelp.php). Note that despite the use of automated examination tools the QA shift crew's visual evaluation of the data remains essential.
• After a hopefully brief wait the list of histograms appears. They may be viewed individually by selecting the "Examine" buttons on the right which will then display the plot with reference (if selected) and a written description.
• Selecting the "All+Plots" button on the left lists all the plots and references (if selected).
• For the "Analyze" option, failed histogram auto-comparisons are listed by default, but all histogram results may be selected in the left-hand panel; the results are color coded.  If the auto-comparison option is used you must still examine the plots visually before completing the examination of the run. 
• After examining the data, mark the run as examined by selecting the Good or Bad buttons on the left. Generally the data will be marked as Good but in extraordinary circumstances can be marked as Bad. Please consult with the QA team before marking any data as Bad.
• To return to the QA run selection page use the "Back to data selections" or "Back to QA options" buttons in the upper panel.

Special issues to watch for in Run 24; the following list may be updated throughout the run:

• General Histograms:  Report dead TPC RDOs, dead/faulty anode wire grids, and RDO sections with large (~50% or more) outages. In general, do not report problems with individual FEEs or pads. However, if the number of bad FEE cards in the inner sectors (padrows 1-40) changes suddenly, or dramatically, then notify Richard Witt, Flemming Videbaek, and Irakli Chakaberia and include the incident in your QA shift report.  The anode and RDO boundaries are marked on the plots. For dead RDOs there is no signal indicated by a blank white space. For dead/faulty anodes there is only noise or the color coded amplitude is substantially different from neighboring anode grids. Be sure to watch for anode voltage sags or outages that may have happened during the run but did not cause the run to be aborted.  This issue is indicated by an unexpected, uniform drop (but not to zero) in the number of hits within the boundaries of an anode grid.  The FMS histograms are for experts only - do not examine or report these. 
• All 24 new inner TPC sectors (iTPC project) have been installed and are included in the QA. You should expect a higher numbers of pads, hits and perhaps increases in track number. Check the performance of these new sectors carefully and report problems to Richard Witt, Flemming Videbaek, and Irakli Chakaberia.
• Trigger Group histograms -- trigger dependence: A few of the histograms are sensitive to the trigger(s) used to collect the events. StE**QaNullPrimVtxMult displays the number of good, missed and questionable primary vertices. 
• Trigger Group histograms -- Luminosity dependence: High instantaneous luminosity increases pileup and the overall number of tracks in the TPC. The number of space points and global tracks will necessarily increase but their distributions in the detector should not change much.  More subtle effects to watch for include: (1) signed DCA for global tracks, StE**QaGtrkSImpactT, which may be affected by the increased distortion in space point position caused by increased space charge accumulation. (2) global track slope versus position relative to primary vertex, StE**QaGtrkTanlzf, where tracks associated with the primary vertex lie along the main diagonal and pileup tracks fill up the rest of the plot  (3) the ratio of primary to global tracks, StE**QaPtrkGlob, where this ratio will be distributed to smaller values when pileup increases. The average luminosity for each run will be provided by the QA browser and should be consulted when examining the histograms.  Do not report the changes in the histograms described here if the run specific luminosity is high. When using the "Combine several jobs" option (C) you should avoid combining histograms from runs with widely varying luminosity as this will distort the distributions and make the QA examination more difficult.
• Trigger Group histograms -- For the distributions of energy clusters in the BEMC (BTOW and BSMD) do not report individual spikes or minor outages (few channels), but do report large sections of obviously excessive or reduced yields.  The latter anomalies often indicate incorrect pedestal values which need to be updated by the experts.
• The MTD QA-Shift histograms display hits in the MTD. There are hit frequency and 2D plots for all hits and for hits matched to global TPC tracks. Report new outages in coverage, unexpected drops in the number of matched hits, or abnormal frequency distributions.  Note that there are no MTD trays underneath the STAR magnet near both ends causing there to be reduced coverage for backlegs 12 through 20. The MTD will be included in Run 23.
•  Watch for ETOF (end-cap time of flight), FTS and FCS histograms in Run 24. Information for QA evaluation of these additional histograms will be provided as these become available throughout the run.

 

Reporting the Results:

• Generally it is best to have the QA Shift Report web form open in a different window so you can fill it out as you check each set of histograms, job-by-job. Please follow the instructions on the QA shift web forms and supply all requested information about yourself and the jobs you have examined.
• If you have both the QA Browser and the QA Shift Report forms open in separate web browser windows, you may take advantage of the "New report entry" to populate a new entry in your Shift Report based on the data being viewed.
• After completing all the listed jobs add whatever comments you think are useful and appropriate to the QA Shift Report. Be sure to include a useful summary for Fast Offline Data that will be helpful to the shift crew, i.e. report any changes from the previous day including new problems or resolution of old problems. Note that the QA Issues mechanism of the web based QA shift report form automatically monitors day-to-day changes in these issues and lists them in the QA shift report summary that is mailed to starqa-hn.
• When new problems appear in the data please review the list of existing QA issues and use those if appropriate, before creating a new issue. Note that there is a key-word search tool to help you find previous, relevant issues. Please follow the naming convention established for the existing Run 24 issues.  You are encouraged to document the issues with histograms using the browse/upload tool in the QA issues editor web page. The browser provides an easy way to grab and upload individual histogram plots (svg file type). Refer to the Help buttons on the new page and click "full topic list", then select "Grabbing a histogram image and attaching to an issue" for instructions - i.e. right click on the image, save to your computer, then in the QA issues page select "Image attachments" and upload your saved file.
• MOST IMPORTANT!!! If you suspect any problem with the detector(s), calibrations, reconstruction or production you must contact the appropriate expert(s). This is the primary reason for having the Fast Offline QA system and these dedicated shifts. The experts may be contacted via either the QA Experts or Other Experts web pages. For Run 24 the various QA and detector experts are: (update pending as of April 17, 2024)

• BBC - Akio Ogawa

• BTOF - Zaochen Ye

• BEMC - Raghav Kunnawalkam Elayavalli

• EPD  - Rosi Reed

• eTOF - Florian Seck

• GMT -
  

• TPC- Richard Witt, Irakli Chakaberia, Flemming Videbaek

• HLT - Hongwei Ke

• VPD  -  Daniel Brandenburg

• Offline-QA - Lanny Ray  + the current week's Offline-QA shift taker

• LFSUPC conveners: Shuai Yang, Yue-Hang Leung, Zaochen Ye

  • delegate: Ben Kimelman

• CF and FCV conveners: Hanna Zbroszczyk, Nu Xu and Prithwish Tribedy, Subhash Singha, Zhenyu Chen, respectively. 

  • delegate: Takafumi Niida (BulkCorr)

• PAC - Sooraj Radhakrishnan 

• TriggerBoard

• S&C - Gene van Buren

• Complete your QA Shift Report and submit it. The ASCII text version will be emailed to 'starqa-hn'.

• Links to QA documentation, contacts, the Rcas/LSF monitor, Online Run Log, and the QA shift report web form are available from Page 2.

• Finally, you are done for the day; go get some rest!