Home » Blogs » yipkin's blog

Processing 2015 pp data (long/trans/long2/long3) to make picoDST (ofr Jarda's format)

Updated on Fri, 2019-08-16 15:21. Originally created by yipkin on 2019-08-16 09:24.
Due to some bug fixing (slide 3), Jerome's group voluntarily reprocessed the 2015 pp data as mentioned :

https://www.star.bnl.gov/HyperNews-star/protected/get/starprod/786/1.html

The query to find the muDst files on the NFS is :

 trgsetupname=production_pp200${period}_2015,production=P18ih,storage=NFS,filetype=daq_reco_muDst

where ${period} can be "long", "trans", "long2" or "long3".
  1. Some files (71 for "long", 477 for "trans", 323 for "long2" and 4 for "long3") are missing in the NFS disks (compared to what are available in HPSS) and I've taken the trouble to download them from HPSS.
     
  2.  Some files might have more than 1 version in the NFS and I used the one with the same file size as it is in the HPSS.  ( The production dates for these two seemed to differ by about 1 week or so.  I think it's the later one that I've picked up. ）
     
  3. I used Jarda's software : https://github.com/adamjaro/star-upcDst to produce the picoDst in Jarda's non-STAR-standard format.  Tomas Truhlar has provided the initial helpful experience/instructions.
     
  4. The production information of the production tag "P18ih" may be found from : https://www.star.bnl.gov/public/comp/prod/DataSummary.html for

    production_pp200long_2015 ,
    production_pp200trans_2015 ,
    production_pp200long2_2015 and
    production_pp200long3_2015 .

    Clicking each of the above links/pages, one may see the "triggerID" link near the bottom of the webpage (    on the right).  Clicking the "triggerID" link would give one the list of trigger ID's used in this production.  I need to put in all these trigger ID's when using Jarda's software.
     
  5. The output of the data are in /star/data01/pwg_tasks/upc01/output, under the directories of "long", "trans", "long2" and "long3".  I group files/data in terms of a STAR run and each run has one picoDst root file.

    To do this, I've used :

    get_file_list.pl -distinct -keys path,filename -cond trgsetupname=production_pp200${period}_2015,production=P18ih,storage=NFS,filetype=daq_reco_muDst,runnumber=$runno -limit 0

    to gather files for a run.  And at the end of processing them, I've used Jarda's script "RunMerge.py" to merge all files of that run into one root file such as :

    RunMerge.py .16071053_trans.in

    where ".16071053_trans.in" is an input file needed earlier in using Jarda's software (for example) for the run 16071053.
     
  6. I've done a few checks including:

    (i) The first one is "checkmergedfilesize.csh"  which checks the sum of the sizes of the individual root files (each run may have 1 to >10 such individual files before merging) is ~about the same as  the size of that merged file in /star/data01/pwg_tasks/upc01/output/${period} .

    Because it seems that when I was lucky or unlucky, the last batch job of the run just finished before my "cron" script "checkmerge_cron.csh" looked for the root file  and  somehow, the transfer took a long time and though the name of the file is there, not the entire file was actually tranferred.   So, if checkmergedfilesize.csh finds anything, I'd just re-run checkmerge.csh
    on that particular run.

    (ii) check_miss_included.csh checks whether the missing file for a run is indeed included in Jarda's filelist in the sub-directories of processing (eg. "list.16071053/sched/*_*.list").

     And checkmissing.csh  checks whether all missing files on disk have been downloaded in the "hpss" area .

    (iii) checkerr.csh   checks  whether those .err file ( eg. "list.16071053/logs/*.err" ) have sizes not equal to 213 bytes.

    It's actually here that I had first realized that there are repeated files.   But now the function or functions of this check is kind of given to checkmergedfilesize.csh.  So, ~90% of the time, if the err file size is > 213 bytes, it is most likely still OK.  But it's still possible that one
    intermediate batch/root file was incomplete.  So, whenever I saw any err file not equal to 213 bytes, I'd just re-process that relevant run.

  7. The STAR scheduler in principle knows to eliminate duplicated files but you have to use the query/dataset criteria (like the above after get_file_list.pl script).  Since I use explicit file lists, the STAR scheduler doesn't know to remove the duplicated files and I have to do so manually.
»