Information, Requirements and Instructions for Fast Offline Quality Assurance Shifts for Run 20

Under:

Introduction and Requirements:

Welcome to the STAR Fast Offline Quality Assurance Shift service.  There are no new shift crew requirements or browser features for Run 20 compared to Run 19. Run 20 is the second run of the STAR Beam Energy Scan II (BES-II) program in which we will take lower energy Au+Au collision data in both collider and fixed target modes.   The plan calls for about 2 months of 11.5 GeV collisions with LEReC commissioning interspersed during the run and a few days of fixed target running. LEReC commissioning at 7.7 GeV is also planned during this run period. Then there will be about 3 months of 9.2 GeV collisions with LEReC and fixed target interspersed. At the end of the run there will be a series of fixed target runs at various energies. The TPC hit plots include the new inner TPC (iTPC) sectors with accompanying new formatted sector plots as in Run 19. The end cap TOF (eTOF) plots are again included and are to be examined.  Please familiarize yourself with these before starting your shift work.

The purposes of the Fast Offline QA shifts are to monitor the validity of the data, calibrations and event reconstruction from the full experiment and to provide near, real-time feed-back to the experiment as needed. An equally important purpose is to compile a record, in the form of reports, of QA issues associated with each run for later data filtering and diagnostics prior to physics analysis. From a practical standpoint it is only possible for Offline QA to detect fairly large problems, and in so doing we continue to strive to examine all of the data collected.

  • No programming skills are required. All the tasks are web based "point-and-click" activities.

  • You will need to subscribe to the 'starqa-hn' and 'starprod-hn' hypernews forums in order communicate with the QA and production experts. See the HyperNews home page or select "hypernews" from the left panel on the STAR Home page. You may unsubscribe from these forums when you are through with taking Offline QA shifts if you like.

  • General knowledge of the STAR detector, reconstruction methods and calibration issues are necessary because the purpose of this work is to spot problems with the hardware and event reconstruction, or with the calibrations. Expert level knowledge is not required however.

  • All persons are required to be at BNL for their first week of Offline QA shift service. This is motivated by the fact that many unforeseen problems will very likely arise during these shifts and quick, real time help, which is more likely to be available at BNL, is essential to ensure daily shift productivity. If this presents an undue hardship please contact the STAR Shift coordinator, Lanny Ray, ray@physics.utexas.edu and Gene van Buren as soon as possible.

  • Subsequent Offline QA shifts may be done from non-BNL sites provided adequate web access to the Auto QA system can be demonstrated from the remote site.

  • The Offline QA shift may be done any time throughout the day but it is expected that a serious effort will require at least 8 hours a day.

  • There are no special Offline QA training schools; the web based documentation is intended to fulfill such needs. But if you have further questions please send email to Lanny Ray, ray@physics.utexas.edu and/or Gene van Buren.

Welcome aboard!

Responsibilities of the Fast Offline QA shift crew:

  • Using the Automated QA browser review the QA Shift histograms for Fast Offline Data Production (highest priority) for all available runs which have not been examined for each trigger stream (e.g. st_physics, st_mtd, st_upsilon, etc.) and for each trigger group (e.g. general, minbias, central, high tower, jet patch, other, etc.).
  • Complete a useful and informative Offline QA Shift report using a web-based form noting especially any and all suspected problems with the detectors, calibrations, and reconstruction. The report will be archived and the summary sent to 'starqa-hn' hypernews automatically. Please use the "play" mode if you are a first-time user to practice filling out the report.
  • Review the Online Run Log and Electronic Shift Log book information and comments for each data run examined and summarize the Run/Data Quality status by marking the job as "Good" or "Bad." This will also indicate that the data have been examined by Offline QA. Jobs will normally be considered "Good" even when there are hardware outages or calibration/reconstruction issues. Please check with the QA experts before marking jobs as "Bad."
  • Notify the appropriate experts and/or the QA contacts for any and all suspected problems with the detectors, calibrations, or fast-offline reconstruction.

Instructions for the Fast Offline QA Shift:

Getting Started:

  • Go to the STAR Computing Offline QA home page on drupal (i.e. from the STAR Home Page select "Computing" in the left panel, then select "Offline QA" in the table row labelled "Production" or go directly to the Offline QA home page and open the Auto QA browser by clicking on the "Automated Offline QA Browser" button in the upper portion of the page. You may have to enter the STAR protected area username and password. Contact your local STAR council representative, Lanny Ray or Gene Van Buren if you do not know it.
  • If the browser fails to open contact the QA Experts ASAP. If you cannot get to the Auto QA browser then you are S.O.L.
  • Enter your RCF username.
  • Select button 2.1 (shift work) and hit OK which takes you to the page where you may select data runs to examine. Note that for Run 20 the Offline QA shift crew are only responsible for the fast-offline production (button 2.1).

Selecting Data:

  • For the Fast Offline Production QA (Button 2.1) on the next page (Page 22) select the data grouping method using buttons (A) - (D). Generally the Auto-combined grouping is preferable as this combines all available files for the run to achieve the best possible statistics. Grouping (C) allows the user to arbitrarily combine data from any of the available runs. Note that the ZDC coincidence rates are listed for each run/file sequence. Ideally multiple jobs should only be combined for similar ZDC rates as some QA histograms depend strongly on the amount of background and pileup.
  • Then select the job listing order (applies to data grouping options B and C only), the date and run number ranges and click OK. 
  • On the next page select the run (or the combination of runs for grouping option C) to be examined. Priority should be given to the most recent data that have not been examined yet. Click OK.

Examining the QA Histograms:

  • The next page provides access to several new features available since 2012.  You should examine all histograms visually, including all listed trigger groups, and file reports for each trigger group.
  • There are numerous "Help" buttons, generally located in the upper-right of any given panel, which present instructional information in the context of what is being viewed at that moment.
  • Automated QA testing is available and can be used provided a suitable set of reference histograms are ready.  These generally take about a week to load once stable physics data production and reconstruction have been achieved and are updated throughout the run period. If you wish to use the automated QA feature please select a reference data set which best matches the data conditions using the left arrow buttons to move from field to field.
  • The default "QA Shift" histogram group is sufficient for shift work.  However, the entire set of QA histograms can be selected with "All."
  • Links to the Run Log and Electronic Shift Log book for the selected run are at the bottom of this panel.
  • Select "Plots only" to view the data only, or "Analyze" to view both the data and reference and to get the results of the automated comparisons with the reference. This option can be used to easily compare the histograms to a reference and enables a convenient way to attach example histograms to QA issues (see instructions: www.star.bnl.gov/devcgi/qa/QAShiftReport/refHelp.php). Note that despite the use of automated examination tools the QA shift crew's visual evaluation of the data remains essential.
  • After a hopefully brief wait the list of histograms appears. They may be viewed individually by selecting the "Examine" buttons on the right which will then display the plot with reference (if selected) and a written description.
  • Selecting the "All+Plots" button on the left lists all the plots and references (if selected).
  • For the "Analyze" option, failed histogram auto-comparisons are listed by default, but all histogram results may be selected in the left-hand panel; the results are color coded.  If the auto-comparison option is used you must still examine the plots visually before completing the examination of the run. 
  • To return to the QA run selection page use the "Back to data selections" or "Back to QA options" buttons in the upper panel.
  • After examining the data mark the run as examined by selecting the Good or Bad buttons on the left. Generally the data will be marked as Good but in extraordinary circumstances can be marked as Bad. Please consult with the QA team before marking any data as Bad.

Special issues to watch for in Run 20; the following list may be updated throughout the run:

  • General Histograms:  Report dead TPC RDOs, dead/faulty anode wire grids, and RDO sections with large (~50% or more) outages. In general, do not report problems with individual FEEs or pads. However, if the number of bad FEE cards in the inner sectors (padrows 1-40) changes suddenly, or dramatically, then notify Flemming Videbaek, Irakli Chakaberia and Chi Yang and include the incident in your QA shift report.  The anode and RDO boundaries are marked on the plots. For dead RDOs there is no signal indicated by a blank white space. For dead/faulty anodes there is only noise or the color coded amplitude is substantially different from neighboring anode grids. Be sure to watch for anode voltage sags or outages that may have happened during the run but did not cause the run to be aborted.  This issue is indicated by an unexpected, uniform drop (but not to zero) in the number of hits within the boundaries of an anode grid.  The FMS histograms are for experts only - do not examine or report these. 
  • All 24 new inner TPC sectors (iTPC project) have been installed and are included in the QA. You should expect a higher numbers of pads, hits and perhaps increases in track number. Check the performance of these new sectors carefully and report problems to Flemming Videbaek, Irakli Chakaberia and Chi Yang.
  • Trigger Group histograms -- trigger dependence: A few of the histograms are sensitive to the trigger(s) used to collect the events. StE**QaNullPrimVtxMult displays the number of good, missed and questionable primary vertices. Typically the relative fraction of good vertices is larger for central trigger events (in A+A).  StE**EmcCat4_Point_Energy shows the frequency distribution of Cat4 energy clusters in the BTOW+BSMD. For central, high tower and jet patch triggers this distribution will extend to larger values and may have a second peak. StE**_Point_Flag shows the number of BEMC Category 1 - 4 clusters. Usually the number of Cat4 clusters increases for the high tower triggers.  Please check for these expected trigger dependencies in the histograms. Note that in Run 20 the BSMD is not used and there are no Cat4 histograms.
  • Trigger Group histograms -- Luminosity dependence: High instantaneous luminosity increases pileup and the overall number of tracks in the TPC. The number of space points and global tracks will necessarily increase but their distributions in the detector should not change much.  More subtle effects to watch for include: (1) signed DCA for global tracks, StE**QaGtrkSImpactT, which may be affected by the increased distortion in space point position caused by increased space charge accumulation. (2) global track slope versus position relative to primary vertex, StE**QaGtrkTanlzf, where tracks associated with the primary vertex lie along the main diagonal and pileup tracks fill up the rest of the plot (less useful for p+p). (3) the ratio of primary to global tracks, StE**QaPtrkGlob, where this ratio will be distributed to smaller values when pileup increases. The average luminosity for each run will be provided by the QA browser and should be consulted when examining the histograms.  Do not report the changes in the histograms described here if the run specific luminosity is high. When using the "Combine several jobs" option (C) you should avoid combining histograms from runs with widely varying luminosity as this will distort the distributions and make the QA examination more difficult.
  • Trigger Group histograms -- For the distributions of energy clusters in the BEMC (BTOW and BSMD) do not report individual spikes or minor outages (few channels), but do report large sections of obviously excessive or reduced yields.  The latter anomalies often indicate erroneous pedestal values which need to be updated by the experts.
  • Please note the relatively new MTD QA-Shift histograms. These display hits in the MTD. There are hit frequency and 2D plots for all hits and for hits matched to global TPC tracks. Report new outages in coverage, unexpected drops in the number of matched hits, or abnormal frequency distributions.  Note that there are no MTD trays underneath the STAR magnet near both ends causing there to be reduced coverage for backlegs 12 through 20.

 

Reporting the Results:

  • Generally it is best to have the QA Shift Report web form open in a different window so you can fill it out as you check each set of histograms, job-by-job. Please follow the instructions on the QA shift web forms and supply all requested information about yourself and the jobs you have examined.
  • If you have both the QA Browser and the QA Shift Report forms open in separate web browser windows, you may take advantage of the "New report entry" to populate a new entry in your Shift Report based on the data being viewed.
  • After completing all the listed jobs add whatever comments you think are useful and appropriate to the QA Shift Report. Be sure to include a useful summary for Fast Offline Data that will be helpful to the shift crew, i.e. report any changes from the previous day including new problems or resolution of old problems. Note that the QA Issues mechanism of the web based QA shift report form automatically monitors day-to-day changes in these issues and lists them in the QA shift report summary that is mailed to starqa-hn.
  • When new problems appear in the data please review the list of existing QA issues and use those, if appropriate, before creating a new issue. Note that there is a key-word search tool to help you find previous, relevant issues. Please follow the naming convention established for the existing Run 20 issues.  You are encouraged to document the issues with histograms using the browse/upload tool in the QA issues editor web page. The browser provides an easy way to grab and upload individual histogram plots (svg file type). Refer to the Help buttons on the new page and click "full topic list", then select "Grabbing a histogram image and attaching to an issue" for instructions - i.e. right click on the image, save to your computer, then in the QA issues page select "Image attachments" and upload your saved file.
  • MOST IMPORTANT!!! If you suspect any problem with the detector(s), calibrations, reconstruction or production you must contact the appropriate expert(s). This is the primary reason for having the Fast Offline QA system and these dedicated shifts. The experts may be contacted via either the QA Experts or Other Experts web pages. For Run 20 the various QA and detector experts are:
  • BBC - Akio Ogawa

  • BTOF - Zaochen Ye

  • BEMC - Raghav Kunnawalkam Elayavalli

  • EPD  - Rosi Reed

  • eTOF - Florian Seck

  • GMT - Dick Majka

  • TPC- Irakli Chakaberia, Flemming Videbaek

  • HLT - Hongwei Ke

  • VPD  -  Daniel Brandenburg

  • Offline-QA - Lanny Ray  + this week's Offline-QA shift taker

  • LFSUPC conveners: David Tlusty, Chi Yang, and Wangmei Zha 

    • delegate: Ben Kimelman
  • BulkCorr conveners: SinIchi Esumi,  Jiangyong Jia, and Xiaofeng Luo 

    • delegate: Takafumi Niida (BulkCorr)
  • PWGC - Zhenyu Ye

  • TriggerBoard (and BES focus group) - Daniel Cebra

  • S&C - Gene van Buren

  • Complete your QA Shift Report and submit it. The ASCII text version will be emailed to 'starqa-hn'.

  • Links to QA documentation, contacts, the Rcas/LSF monitor, Online Run Log, and the QA shift report web form are available from Page 2.

  • Finally, you are done for the day; go get some rest!