HLT

Those pages are provided for the High Level Trigger activities.

HLT Linux Cluster User's Guide

HLT Linux Cluster User's Guide
- Howe to get an account on HLT Farm

  1. Get an account to the STAR Protected Network. See instructions at https://drupal.star.bnl.gov/STAR/comp/onl/accessing-star-protected-network
  2. Once you can access STAR online gate way stargw.starp.bnl.gov, send an email to starsupport@bnl.gov (cc to me: kehw@bnl.gov) to request your access to xeon-phi-dev.starp.bnl.gov, which is the entry point to the HLT cluster.

- How to login HLT cluster
The STAR HLT cluster sites in a local network, which has an entry point to the STAR online network.
To reach the cluster, one need to jump a few time via ssh.
1. (if you are outside of BNL) Connect to the RCF gateway rssh.rhic.bnl.gov or cssh.rhic.bnl.gov.
The RCF NX service is recommended for better graphics performance.
https://www.racf.bnl.gov/docs/services/nx

2. From the RCF gateway or the NX node, connect to the STAR gateway: stargw.starp.bnl.gov
You must have a STAR online enclave account to be able to do this.
https://drupal.star.bnl.gov/STAR/comp/onl/accessing-star-protected-network

3. From the STAR gateway, connect to the HLT cluster entry point: xeon-phi-dev.starp.bnl.gov (aka L409)
Now you are on STAR HLT cluster.

The ssh ProxyJump or ProxyCommand is recommended to jump through multiple hosts
https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#Passing_Through_One_or_More_Gateways_Using_ProxyJump

- Computing Environment

   * SL 7.3
   * No STAR environment pre-configured.
   * If you need the STAR environment, add the following lines in you ~/.cshrc
--------------------------------------------------------------------------

source /net/l402/data/fisyak/STAR/packages/.DEV2/setupDEV2.csh
setup NODEBUG
setup 64b
starver TFG18m
--------------------------------------------------------------------------

- NFS mounted home dir: /star/u/[user name]
Size: 1.3TB, RAID 5, no quota yet, but please use your home dir wisely

 - NFS mounted Storage
/net/l401/data/scratch[1,2]
/net/l402/data
/net/l403/data
/net/l404/data
* 10+TB each, available on all HLT nodes.

- Ceph storate
On all nodes
/hlt/cephfs

 - Condor Queue: Intel Xeon E5-2670 @ 2.50GHz
condor submitter: xeon-phi-dev.starp.bnl.gov (aka l409), l404

 - Intel Xeon Phi coprocessor 7120P
l405-l408, l410-l417, two cards per node
l409(xeon-phi-dev) one card
In order to use the coprocessor, Intel C/C++ compiler is needed, which can be found at
/opt/intel_17
Documentation for Intel compiler and libraries can be found at
https://software.intel.com/en-us/intel-parallel-studio-xe-support/documentation
More information about Xeon Phi KNC (1st generation, x100 family) can be found at
https://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-codename-knights-corner

- PGI Compiler
The PGI compiler community edition has been installed in
/software/pgi

One need the following settings to use the compilers

% setenv PGI /software/pgi
% set path=(/software/pgi/linux86-64/18.10/bin $path)
% setenv MANPATH "$MANPATH":/software/pgi/linux86-64/18.10/man
% setenv LM_LICENSE_FILE /software/pgi/license.dat:"$LM_LICENSE_FILE" 

---------------------------------------------------------------------------------
Contact Person:
Hongwei Ke
kehw@bnl.gov
631-344-4575

HLT QA

 

KFParticle Tutorial

KF Particle Finder is the package for reconstruction and selection of the short-lived particles using the Kalman filter based mathematics. Properties of the package:

  • the vector of parameters include physical parameters: position of the particle, momentum and energy;
  • all parameters are provided together with their errors estimated by the Kalman filter;
  • short-lived particles and their decay products are treated exactly in the same way;
  • as a result, complicated decay topology can be easily reconstructed;
  • because of the general parametrisation, the package does not depend on the detector geometry;
  • the package is easy portable between experiments;
  • the package is optimised and fully vectorised;
 

KF Particle Finder is being developed be the FIAS group. Documentation on the package:

The example macro how to run KF Particle Finder is located in "/star/u/mzyzak/KFParticleFinderExample/". The folder contains:

  • setDEV2.csh - a script to configure the environment to TFG18d release;
  • StRoot - the latest version of the KF Particle software together with its interfaces;
  • data - a folder with one MuDst and one PicoDst file of the run 2016;
  • Analysis.C - the main macro to run the example;
  • lMuDst.C - macro to load needed libraries, called by.

In order to run the test:

  • copy the folder;
  • configure the environment running "source setDEV2.csh";
  • compile the code by running "cons";
  • to run PicoDst: root -l Analysis.C;
  • to run MuDst: root -l 'Analysis.C(100000, 1)'.

The output is a table with efficiencies every 100 events and "pico.root" or "mu.root" with the histograms.

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}

STAR HLT User's Manual

STAR HLT Weekly Discussion

 

STAR HLT Weekly Discussion

---------------------------------------------------------------------- 

Bluejeans Meeting Info.

https://bluejeans.com/934506067

To join via Phone:

1) Dial:

+1.408.740.7256

+1.888.240.2560(US Toll Free)

+1.408.317.9253(Alternate number)

2) Enter Conference ID: 934506067

---------------------------------------------------------------------- 
Jan 23, 2018
     - Maksym, Reconstruction of open charm in STAR with KF Particle Finder
     - Petr, KF Overview

Oct 3, 2017
     - Biao Tu, updates on dNdx fit with derivative

Sep 26, 2017
     - Biao Tu, improve dNdx fit by providing derivative of fitting function 

Aug 22, 2017

     - Hongwei. Produce K and p Embedding samples with HFT tracking in order to check the efficiency, DCA distribution and etc. due to real alignment. These samples will also be used to check the PID, such as dEdx and TOF.
    - Maksym and Iouri. Check the reconstruction efficiency for simulation with real alignment for Lambda_c, D0, D+ and Ds. These task is to check the alignment and analysis mechanism are working. After these, check the efficiency and purity from embedding data.
    - Michal. For idea geometry, check the KFParticle reconstruction decays. Results are expected to be presented in a table of purity and efficiency.
    - Yuri. Fixed the library version TFG17g. This version should be used for the above tasks instead of .DEV2 for better stability.

Aug 15, 2017
      - Michal Kocan, Efficiency of reconstruction D0->K0(PiPi)PiPi

Apr 11, 2017
      - Hongwei Ke, HLT Job Management System

Feb 13, 2017
      - Yongjin Ye, SC calibration with Run12 pp 510GeV data

Jan 31, 2017
       - Yongjin Ye, SC study in Run12 pp 510GeV

Jan 24, 2017
       - Yongjin Ye, SC calibration for Run12 pp 510GeV

Jan 10, 2017
       - Irakli Chakaberia, TPC new cluster finder comparison

Jan 3, 2017
       - Run 17 schedule, Hongwei Ke

Oct 18, 2016
        - HLT code refactoring for Xeon Phi, Jeff Landgraf, Hongwei Ke

Sep 6, 2016
        - Hongwei Ke, resources estimation for BES II
        - Yuri Fisyak, compiler tests

Jun 21, 2016

        - Hongwei, HLT has switch back all the calibration tables and settings for AuAu200GeV. QA plot look ok. If there is nothing change dramatically, we may not have any new calibration before the end of this run.

        - Grigory is working on the merger, which merges the HFT tracks and TPC tracks.

        - Zhengqiao updated HLT QA plot for Vz because the requirement was changed during the low energy dAu runs. Zhengqiao will also resume the development of SC auto calibration code for HLT.


Feb 15, 2016

        - Grigory and Zhengqiao are working on the data preparation for the merger test. There is some coordinates converting need to be done between STAR golbal coordinates and detector local coordinates. This is well undersood knowledge and HLT already has the code to do that for TPC.
        - Zaochen Ye is working on the new QA code MTD-hlt triggers.
        - Maksy Zyzak fix the problem of sending data to Phi via SCIF and is improving the sender.

Feb 9, 2016

        - Zhaochen Ye. Finished the first t0 calibration for Run16. Working on additional QA plots for the new diElectron algorithm.

        - Zhengqiao Zhang. Finished the first space charge calibration for Run16.

        - Grigory Kozlov. Working on the merger now, but need simulation data. Zhengqiao will help to preparing the data.


Jan. 25, 2016
       - Grigory Kozlov, The CA-HFT tracker is now running at ~90ms per event.  Speed is expected to improve when we run the code on AVX available CPU.
       - Maksym Zyzak, working on using SCIF APIs to transfer data between CPU and Phi.
       - Hongwei Ke, HLT is almost ready for the run, except a few calibration tables needed to generated during the first a few days after collisions are available.
       - Zaochen Ye, Working on HLT online QA code to add plots for MTD quarkonium.

Jan. 19, 2016
        - Hongwei Ke, HLT is prepared for the coming run. Given the new machines are not arrived here, we prepared the current machines to be used for now. Will do a test later today.
        - Maksym Zyzak, Keep working on the KFParticle server on the Phi card side.
        - Grigory Kozlov, The CA-HFT tracker is now running at ~200ms per event. One more optimization is hopefully finished by the end of this week.

Jan. 13, 2016
        - Maksym Zyzak, Status of the KF Particle Finder package for the Xeon Phi. Discuss the KFParticle secondary vertices reconstruction and task division of FIAS group.

Dec. 22, 2015
        - Mike Lisa, 4Sd: the "other" di-baryon state searching in STAR. Discuss the idea of search 4Sd in HLT. Grigory Nigmatkulov will use KFParticle to estimate the selection rate with Run14 data.
        - Maksym Zyzak, Investigation of decays reconstruction in HLT. Discuss the KFParticle secondary vertices reconstruction and task division of FIAS group.
        - Ivan Kisel, Status of the primary vertex estimation. Present the recent development on primary vertex estimation algothms, which is intended to be used to start CA-HFT tracking.

Dec. 15, 2015
        - Aihong, Thoughts on Exotic Searches at STAR.


Nov. 17, 2015
        - Grigory, Status of HFT CA track finder Grid. Triplete calculation has been vectorized and gives x7.7 speedup.

Nov. 10, 2015
        - Zhengqiao, Simulation events with pileup. Simulated ~1.5k AuAu 200GeV events with pileup for CA-HFT tracker turning.
        - Grigory, Status of HFT CA track finder Grid. Grid of HFT hits has been implimented and shows very good speed up, especially for  doublets calculation.
        - Rongrong, HLT performance using embedded muons. We invited Rongrong come to talk about the study on the HLT-diMuon trigger efficiency. New matching algorithm shows significant improvement.


Nov. 3, 2015
        - Valentina. PV reconstruction with KF Particle. Discussed prelimary results of using KFParticle to reconstruct PV in AuAu collisions at 200GeV.
        - Grigory. Implimented grid for the CA-HFT tracker which will significantly increased the speed of CA-HFT tracker.
        - Maksym and Hongwei. Vc CPU version and MIC version both works, but we still have problem of building offload version.


Oct. 20, 2015

        Hongwei Ke, Some experience with Vc

Hongwei - This problem was triggered by the speed optimization of the local z calculation if HFT hits, StThinPlateSpline::z(). It turns out that a simple implementation with raw C++ arrays works the best with icc auto-verization. Explicit vectorization with Vc library works the second best with gcc. Surpassingly icc + Vc works five time slower than gcc + Vc and 10 times slower than icc auto-verization. Maksym thinks it maybe something to do with how icc treats inline functions, which are heavily used by Vc library. Need more study on this.

Grigory - Need more simulation data with TPC and HFT without pile-up. Zhengqiao will provide that. Grigory is going to try tracking TPC and HFT together, i.e. extrapolate TPC tracks to HFT. 
 
STAR offline software group has obtained a lot of experiences on this method, which should have very good speed because they only need a few percent more time to tracking HFT hits. The difficulty here is that the pointing resolution of TPC tracks is large comparing to HFT hits errors and pile-up. Usually one will see multiple HFT hits in a projected area on PXL. Requiring good long TPC tracks to do the extrapolation is helpful. Statistically, good TPC tracks point to the right HFT hits. The other thing one need to pay attention is that Kalman filter needs to treat the coulomb scattering in HFT differently to what have been done with TPC.

Oct. 13, 2015
         Gregory Kozlov, Status of HFT CA track finder

        - Grgory talked about the new update of CA tracker for HFT. By calculate the doublets and triplets from outer station instead of calculating all of them, we see very good speed up, espically with low multiplicity events. There are still many optimization opportunities to imporve the speed.
        - Maksym contacted  Matthias, the author of Vc library, and it seems that there should not being problem of running Vc in offload. However, a real example is required by Mattias to work on. Maksym will contact Mattias again next week.
           What we do not understand now is how to call the same Vc code in CPU version and MIC offload version in the same program. It maybe helpful to know how Mattias thinks of this problem.
        - Hongwei has some news from supermicro tech. support aobut the low frenquency problem of phi card on HLT servers, model SYS-1027GR-TRFT+. The problem is that the phi card can only run at the lowest allowed frenquency when it is plugin in the right slot (see from front). After discuss and test for about three weeke, supermicro tech. support seems sure that a new riser card on the right side is needed, reversion from 1.00 to 1.01. Will send our card to supermicor will get a new one back.
           Hongwei will also talk with Zhengqiao about how to dump HFT hits for both MC and real events for Grgory. We may be able to joint some effort with Yuri, because he is producing MC events including HFT and pile-up for StiCA evaluation.

Oct. 6, 2015
        For the interests of this group, Hongwei, Sti and StiCA Performance Comparison

Sep. 29, 2015
         Gregory Kozlov, Status of HFT CA track finder

Jul. 8, 2014
         We disscussed the HLT tracking efficiency again. There are two approches we can use. First choice is Yuri's HLTCAInterface, which can run HLT TPC maps can CA tracker in the save time. We used this method before, but found asymmetry between east and west TPC. We suspect that there is somethink wrong with the TPC maps. However, we have not identified them yet. The second choice is to write our own association tools. We can use common hits schem used by Xiangming before. We conclude that we will need these tools anyway. So will write it in this summary. 

Nov. 12, 2013
         Hongwei Ke, PV Reconstruction at 19.6GeV

Oct. 29, 2013
         Hongwei Ke, HLT TPC Maps' Problem

​

Sep. 10, 2013
         Hongwei Ke, HLT+KFParticle PV finder

Aug. 13, 2013
         Hongwei Ke, HLT+CA

Jul. 30, 2013
         Hongwei Ke, HLT Status

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Helvetica; -webkit-text-stroke: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Helvetica; color: #0069d9; -webkit-text-stroke: #0069d9}
span.s1 {font-kerning: none}
span.s2 {text-decoration: underline ; font-kerning: none}

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Helvetica; -webkit-text-stroke: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Helvetica; color: #0069d9; -webkit-text-stroke: #0069d9}
p.p3 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Helvetica; -webkit-text-stroke: #000000; min-height: 17.0px}
span.s1 {font-kerning: none}
span.s2 {font-kerning: none; color: #000000; -webkit-text-stroke: 0px #000000}
span.s3 {text-decoration: underline ; font-kerning: none; -webkit-text-stroke: 0px #0069d9}

SpaceCharge and GridLeak Calibration for offline

calibration of SpaceCharge(SC) and GridLeak(GL)
 
cd /star/u/yjye/yjye1/HLT/SC_GL/run2018/AuAu27
 
operation step:
 
1. get the inital value of StarDb6.8_9/StarDb6.8_10/StarDb6.8_11, chain option of bfc.f and star version from Gene.
   a. time relationship: (early)beginTime on the website < filename of the StarDb* < daqfile(late). 
   
2. download the daqfile of this year which need to do calibration.
  a. open star RunLog and choose th daqfiles
  b. (20-30)daqfiles are needed and more than 5000 events for each daqfie.
  c. contain a wide lumilosity for those events in the daqfiles
  d. hpss_user.pl -f trans4
  d. we can confirm the download result on websit: http://www.star.bnl.gov/devcgi/display_accnt.cgi
 
3. cd submit
 
4. Use the StSpaceChargeEbyEMaker to create the root file from daq file for the following calibration.
   ./submit_all.bash
 
5. Use the function of doEvent_SCGL_Calib.C to calibrate the SC and GL from the input root file and create some histogram for the   next fitting.
   ./run_step2.bash
 
6. ./moveHist.bash
 
7. Fit the histogram and get the SC and GL.
   cd ../ and ./run_all.csh

SpaceCharge and GridLeak Calibration for online

 calibaration of SpaceCharge(SC) and GridLeak(GL) with online

 
cd /star/u/yjye/yjye1/HLT/SpaceCharge/SpaceCharge/online/RTS/src/L4_HLT
 
operation step:
 
1. we need to files, "HLTConfig" and "conf_18171003.xml" in the run_l4Emulator.csh, which is supported by Hongwei. 
 
2. set the value of "spaceCharge" in online_tracking_TpcHitMap.cxx. The value is decided by the sc which is calculated on the
   offline. The step of the value change is 0.001 or 0.002.
 
3. ./run_ye18171003.csh