Embedding Documentation
Overview
The purpose of embedding is to provide physicists with a known; Monte Carlo tracks, where the kinematics and particle type is known exactly, are introduced into the realistic environment seen in the analysis - a real data event. How reconstruction preforms on these "standard candle" tracks provides a baseline which can be used to correct for acceptance and effciency effects in various analyses.
In STAR, embedding is achieved with a set of software tools - Monte Carlo simulation software, as well as the tools for real data event reconstruction - accessed through a suite of scripts. These documents attempt to provide an introduction to the scripts and their use, but the STAR collaborator is referred to the separate web pages maintained for Simulations and Reconstruction for deeper questions of the processes involved in those respective tasks.
Please note, the general procedure for embedding is:
- The Physics Working Group Convenor submits an embedding request. The request will be assigned a request number and priority by the Analysis Coordinator, see this page for more information.
- The Embedding Coordinator will coordinate the distribution of embedding requests. Users who wish to run their own embedding are encouraged to do so - but it must be verified with the Embedding Coordinator.
- The Embedding Coordinator is responsible for the QA of all embedding requests. When the Embedding Coordinator has verified that the test sample produced is acceptable, full production is authorized.
For EC and ED(s) (OBSOLETE)
Embedding instructions for EC and ED(s)
Last updated on Sept/06/2011 by Christopher Powell
Current Embedding Coordinator (EC): Xianglei Zhu (zhux@rcf.rhic.bnl.gov)
Current Embedding Deputy (ED):
Current PDSF Point Of Contact (POC): Jeff Porter (RJPorter@lbl.gov)
Revised history
-
Sept/06/2011: Update EC and ED
-
Nov/17/2010: Added some tickets for the records
-
Aug/02/2010: Update directory structure for LOG and Generator files
-
Jun/11/2010: Added approved chains by Lidia
-
May/29/2010: Modify procedures of 'Production chain options'
-
May/27/2010: Several minor fixes
-
May/25/2010: Added 'Production chain options', 'Locations of outputs etc at HPSS'
- May/24/2010: Update trigger id options
- May/21/2010: Update instructions for xml file, useful discussions
Contents
Please send me (CBPowell@lbl.gov) an e-mail if you have any questions/suggestions.
The typical xml file for the embedding job submission looks
<!-- Generated by StRoot/macros/embedding/get_embedding_xml.pl on Mon Aug 2 15:26:13 PDT 2010 --> <?xml version="1.0" encoding="utf-8"?> <job maxFilesPerProcess="1" fileListSyntax="paths"> <command> <!-- Load library --> starver SL07e <!-- Set tags file directory --> setenv EMBEDTAGDIR /eliza3/starprod/tags/ppProductionJPsi/P06id <!-- Set year and day from filename --> setenv EMYEAR `StRoot/macros/embedding/getYearDayFromFile.pl -y ${FILEBASENAME}` setenv EMDAY `StRoot/macros/embedding/getYearDayFromFile.pl -d ${FILEBASENAME}` <!-- Set log files area --> setenv EMLOGS /project/projectdirs/star/embedding <!-- Set HPSS outputs/LOG path --> setenv EMHPSS /nersc/projects/starofl/embedding/ppProductionJPsi/JPsi_&FSET;_20100601/P06id.SL07e/${EMYEAR}/${EMDAY} <!-- Print out EMYEAR and EMDAY and EMLOGS --> echo EMYEAR : $EMYEAR echo EMDAY : $EMDAY echo EMLOGS : $EMLOGS echo EMHPSS : $EMHPSS <!-- Start job --> echo 'Executing bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); ...' root4star -b <<EOF std::vector<Int_t> triggers; triggers.push_back(117705); triggers.push_back(137705); triggers.push_back(117701); .L StRoot/macros/embedding/bfcMixer_TpcSvtSsd.C bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); .q EOF ls -la . cp $EMLOGS/P06id/LOG/$JOBID.log ${FILEBASENAME}.$JOBID.log cp $EMLOGS/P06id/LOG/$JOBID.elog ${FILEBASENAME}.$JOBID.elog <!-- New command to organize log files --> mkdir -p $EMLOGS/P06id/JPsi_20100601/LOG/&FSET; mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/ <!-- Archive in HPSS --> hsi "mkdir -p $EMHPSS; prompt; cd $EMHPSS; mput *.root; mput ${FILEBASENAME}.$JOBID.log; mput ${FILEBASENAME}.$JOBID.elog" </command> <!-- Define locations of log/elog files --> <stdout URL="file:/project/projectdirs/star/embedding/P06id/LOG/$JOBID.log"/> <stderr URL="file:/project/projectdirs/star/embedding/P06id/LOG/$JOBID.elog"/> <!-- Input daq files --> <input URL="file:/eliza3/starprod/daq/2006/st*"/> <!-- csh/list files --> <Generator> <Location>/project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST</Location> </Generator> <!-- Put any locally-compiled stuffs into a sand-box --> <SandBox installer="ZIP"> <Package name="Localmakerlibs"> <File>file:./.sl44_gcc346/</File> <File>file:./StRoot/</File> <File>file:./pams/</File> </Package> </SandBox> </job>
Below is step by step instructions how to set it up for each request.
You can grab some xml files from other requests and modify manually
or can also create it by yourself with "StRoot/macros/embedding/get_embedding_xml.pl".
The option "-h or --help" will show all available options in "get_embedding_xml.pl".
1. Set up daq/tags files
Please contact POC at PDSF to locate daq/tags files if you don't find any daq/tags files
in the eliza disks at PDSF. The relevant descriptions in the xml file are
<!-- Input daq files --> <input URL="file:/eliza3/starprod/daq/2006/st*"/>
for daq files and
<!-- Set tags file directory --> setenv EMBEDTAGDIR /eliza3/starprod/tags/ppProductionJPsi/P06id
for tags files.
You can change the locations of daq/tags files in xml by "-daq [daq file path]" and "-tag [tags file path]" options like
> StRoot/macros/embedding/get_embedding_xml.pl -daq /eliza3/starprod/daq/2006 -tag /eliza3/starprod/tags/ppProductionJPsi/P06id
Note1-1: You can put the options in any order, so the command below gives the same result as above
> StRoot/macros/embedding/get_embedding_xml.pl -tag /eliza3/starprod/tags/ppProductionJPsi/P06id -daq /eliza3/starprod/daq/2006
Note1-2: If you have already created an xml file in your current directory,
"get_embedding_xml.pl" won't overwrite the previous xml file. If you want to overwrite it, put "-f" option.
2. Running job, archive outputs into HPSS
Below is the descriptions to run the job (bfcMixer), save log files, put outputs/logs into HPSS.
<!-- Start job --> echo 'Executing bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); ...' root4star -b <<EOF std::vector<Int_t> triggers; triggers.push_back(117705); triggers.push_back(137705); triggers.push_back(117701); .L StRoot/macros/embedding/bfcMixer_TpcSvtSsd.C bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); .q EOF ls -la . cp $EMLOGS/P06id/LOG/$JOBID.log ${FILEBASENAME}.$JOBID.log cp $EMLOGS/P06id/LOG/$JOBID.elog ${FILEBASENAME}.$JOBID.elog <!-- New command to organize log files --> mkdir -p $EMLOGS/P06id/JPsi_20100601/LOG/&FSET; mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/ <!-- Archive in HPSS --> hsi "mkdir -p $EMHPSS; prompt; cd $EMHPSS; mput *.root; mput ${FILEBASENAME}.$JOBID.log; mput ${FILEBASENAME}.$JOBID.elog"
The default bfcMixer is "StRoot/macros/embedding/bfcMixer_TpcSvtSsd.C".
The bfcMixer can be set by "-mixer [bfcMixer file]"
> StRoot/macros/embedding/get_embedding_xml.pl -mixer StRoot/macros/embedding/bfcMixer_Tpx.C
NOTE2-1: There are three different bfcMixer macros, bfcMixer_Tpx.C, bfcMixer_TpcSvtSsd.C and bfcMixer_TpcOnly.C.
You need to choose the proper bfcMixer macro depending on the Run;
<= Run4 : bfcMixer_TpcOnly.C
Run5 - Run7 : bfcMixer_TpcSvtSsd.C
>= Run8 : bfcMixer_Tpx.C
The library, production tag, trigger setup name and request number can be changed by using the following options;
"-production [production tag]", "-lib [library]", "-r [request number]" and "-trg [trigger setup name]"
> StRoot/macros/embedding/get_embedding_xml.pl -production P06id -lib SL07e -r 20100601 -trg ppProductionJPsi
The default library, production tag, and trigger setup name are SL08c, P08ic, 2007ProductionMinBias respectively
unless otherwise specified. These will be used for the locations of log files, scripts as well as the path in HPSS like
<!-- Load library --> starver SL07e ... ... ls -la . cp $EMLOGS/P06id/LOG/$JOBID.log ${FILEBASENAME}.$JOBID.log cp $EMLOGS/P06id/LOG/$JOBID.elog ${FILEBASENAME}.$JOBID.elog <!-- New command to organize log files --> mkdir -p $EMLOGS/P06id/JPsi_20100601/LOG/&FSET; mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/ <!-- Archive in HPSS --> hsi "mkdir -p $EMHPSS; prompt; cd $EMHPSS; mput *.root; mput ${FILEBASENAME}.$JOBID.log; mput ${FILEBASENAME}.$JOBID.elog" ... ... <!-- csh/list files --> <Generator> <Location>/project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST</Location> </Generator>
NOTE2-2: If the directories "/project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST"
doesn't exist, "get_embedding_xml.pl" will complain and doesn't generate xml like
Error: No /project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST exists. Stop. Make sure you've put the correct path for generator file.
Currently, I didn't implement to automatically create "LOG" and "LIST" directories
in get_embedding_xml.pl. So you have to make those directories manually.
Please don't forget to make those directories group read/writable. Please contact me
if you are not clear what you should do (Hiroshi).
NOTE2-3: LOG directory structure has been changed to make them search easier,
and final log files will be moved to
mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/
where we have introduced three additional directory "P06id" (production), "JPsi_20100601" (particle name and request id)
and "&FSET" (FSET number, which is determined during the job submission). This directory will be dynamically created
during the job submission, so you don't need to create it.
3. Arguments in the bfcMixer
You can find relevant informations for these numbers in the
STAR simulation request page, http://drupal.star.bnl.gov/STAR/starsimrequest.
Below is an example for the 2006 J/Psi in p + p 200 GeV request
Below is the details how to set each argument in the bfcMixer according to
the informations in the simrequest page
3-1. Particle id and particle name
The particle geantid can be found in "pams/sim/gstar/gstar_part.g".
You can see the J/Psi -> e+e- decay (100% B.R.)
Particle Jpsi code=160 TrkTyp=4 mass=3.096 charge=0 tlife=7.48e-21,pdg=443 bratio= { 1, } mode= { 203, }
"code=160" is the geantid for J/Psi so you can set the geantid and particle name by
"-geantid [GEANT3 id]" and "-particle [particle name]"
> StRoot/macros/embedding/get_embedding_xml.pl -geantid 160 -particle JPsi
Please don't put "/" in the particle name. The particle name is also used for the directory
in HPSS as well as in eliza disk to store the outputs. So "/" would be recognized as
directory.
3-2. Particle settings (how to simulate pt distribution)
Particle settings is usually "Flat in pt", generating the flat pt distribution of input MC particles,
which corresponds to the default option "FlatPt" in the last argument of bfcMixer.
There are two other options depending on how to simulate the input MC particles
"Strange" : Smear primary vertices in all (x,y,z) directions with vertex errors stored in tags files
"Spectrum" : Generate transverse momentum by pt*exp(-pt/T) shape, where T is inverse slope parameter
Default T is 0.3 GeV
These options can be changed by "-mode [option]"
> StRoot/macros/embedding/get_embedding_xml.pl -mode Strange
NOTE3-1: The mode option string is case insensitive, so "Strange", "strange", "STRANGE"
or any combination of letters can be accepted as long as it matches with "strange" string.
NOTE3-2: "Strange" option cannot be run in "VFMCE" chain option.
You should disable VFMCE by "-VFMCE" and put the appropriate vertex finder option back to the chain options.
NOTE3-3: "Spectrum" option can only be run after SL08f. We don't have any solutions at this moment except for
doing single MC simulation and embedding, separately.
3-3. Multiplicity
The default multiplicity is 1, i.e. throw 1 input MC particle per event. You can change it by "-mult [multiplicity]"
> StRoot/macros/embedding/get_embedding_xml.pl -mult 0.05
if the multiplicity is less than 1, the input MC particle will be generated
by the fraction of multiplicity from real data.
The 0.05 gives 5% of multiplicity per event, so the number of particles
will be varied event-by-event depending on the multiplicity.
In the current example, we don't need to modify the multiplicity.
3-4. Primary z-vertex range
The minimum and maximum z-vertex cuts can be set by
"-zmin [Minimum z-vertex value]" and "-zmax [Maximum z-vertex value]"
> StRoot/macros/embedding/get_embedding_xml.pl -zmin -30.0 -zmax 30.0
if PA requests some specific z-vertex cut. The default z-vertex cut is |vz| < 200 cm.
In the current J/Psi case, we don't need to put any z-vertex cut.
3-5. Rapidity and transverse momentum range
The rapidity and transverse momentum cuts can be set by
"-ymin [Minimum y value]", "-ymax [Maximum y value]",
"-ptmin [Minimum pt value]" and "-ptmax [Maximum pt value]"
> StRoot/macros/embedding/get_embedding_xml.pl -ymin -1.0 -ymax 1.0 -ptmin 0.0 -ptmax 6.0
NOTE3-4: The simulation request shows the eta range, but we'll throw in rapidity instead of eta.
The default rapidity and pt cuts are |y| < 1.5 and 0 < pt < 10 GeV/c, respectively.
For the current example, we only need to modify maximum pt cut off to 6 GeV/c
3-6. Trigger id's
If PA request trigger id(s), you can put them by using "-trigger [trigger id]"
> StRoot/macros/embedding/get_embedding_xml.pl -trigger 117705
you can add multiple triggers by adding more "-trigger [trigger id]" like
> StRoot/macros/embedding/get_embedding_xml.pl -trggier 117705 -trigger 137705 -trigger 117001
minor NOTE3-5: (This doesn't work, please don't try) I had try to implement the "-trigger" argument like
> StRoot/macros/embedding/get_embedding_xml.pl -trigger 117705 137705 ...
to accept multiple numbers with single "-trigger" option, but couldn't succeed with the current
perl version at PDSF/RCF, though it worked with newer perl in my Macbook (Hiroshi).
3-7. Production chain name
In order to use the correct chain option in the bfcMixer, you should set the production chain name like
StRoot/macros/embedding/get_embedding_xml.pl -prodname P06idpp
NOTE3-6: It might not be trivial what kind of chain options are available in the bfxMixer's.
Please take a look at the bfcMixer to make sure that the relevant chain option has been implemented.
Including the all relevant options in get_embedding_xml.pl, the command to produce the J/Psi xml file is
> StRoot/macros/embedding/get_embedding_xml.pl -f -daq /eliza3/starprod/daq/2006 -tag /eliza3/starprod/tags/ppProductionJPsi/P06id \ -production P06id -lib SL07e -r 20100601 -trg ppProductionJPsi -geantid 160 -particle JPsi -ptmax 6.0 -trigger 117705 -trigger 137705 -trigger 117701 \ -prodname P06idpp
There a few additional things we need to modify xml file manually to get the final form shown in the beginning
- First, we also need to set 0 for the 2nd/3rd arguments in the bfcMixer_TpcSvtSsd.C.
Those are flags to turn on SVT (2nd) and SSD (3rd) if they are 1. Since this example is
the request for Run6 p+p, we don't need SVT/SSD and hence must turn them off, i.e. set to be 0.
- Second, 100 % J/Psi -> e+e- has already been implemented in the "pams/sim/gstar/gstar_part.g" and
is available in the default STAR library so we actually don't need to check out pams and recompile.
In that case, you also need to remove "pams" in your "Localmakerlibs".
- Third, PA requests 2 different triggers in "st_jpsi" stream and one trigger in "st_physics" stream
so the xml files should be prepared for those two different streams with different triggers/inputs.
There would be couple of other things we have to do manually, which will depend on each request.
Please let us know your feedback in case you had some special modification on your xml file.
Production chain options
Production chain can be found in http://www.star.bnl.gov/devcgi/dbProdOptionRetrv.pl
Since the current embedding must use "VFMCE" chain option, other vertex finder chain options have to
be excluded. The relevant informations about vertex finder chain options can be found
Below is the current procedure how to set up production chain in the bfcMixer's
- ED will set up the proper bfcMixer macro with the correct chain options
-
EC and POC at PDSF will take a look and give feedback
-
Ask Lidia for her inputs, and verify the new chain with Lidia and Yuri
-
Enter all chains into Drupal embedding page for documentation
-
Commit bfcMixer into CVS
Below is the approved chains implemented in the bfcMixer at this moment (Jun/11/2010)
Chains approvied by Lidia
--------------------------------
P07ic CuCu production: TString prodP07icAuAu("P2005b DbV20070518 MakeEvent ITTF ToF ssddat spt SsdIt SvtIt pmdRaw OGridLeak OShortR OSpaceZ2 KeepSvtHit skip1row VFMCE -VFMinuit -hitfilt");
P08ic AuAu production: DbV20080418 B2007g ITTF adcOnly IAna KeepSvtHit VFMCE -hitfilt l3onl emcDY2 fpd ftpc trgd ZDCvtx svtIT ssdIT Corr5 -dstout
If spacecharge and gridleak corrections are on average instead of event by event then Corr5-> Corr4, OGridLeak3D, OSpaceZ2.
P08ie dAu production : DbV20090213 P2008 ITTF OSpaceZ2 OGridLeak3D beamLine, VFMCE TpcClu -VFMinuit -hitfilt
TString chain20pt("NoInput,PrepEmbed,gen_T,geomT,sim_T,trs,-ittf,-tpc_daq,nodefault);
P06id pp production : TString prodP06idpp("DbV20060729 pp2006b ITTF OSpaceZ2 OGridLeak3D VFMCE -VFPPVnoCTB -hitfilt");
P06ie pp production : TString prodP06iepp("DbV20060915 pp2006b ITTF OSpaceZ2 OGridLeak3D VFMCE -VFPPVnoCTB -hitfilt"); run# 7096005-7156040
TString prodP06iepp("DbV20061021 pp2006b ITTF OSpaceZ2 OGridLeak3D VFMCE -VFPPVnoCTB -hitfilt"); run# 7071001-709402
Locations of outputs, log files and back up of relevant codes at HPSS
Current output as well as log file location at HPSS is determined by the following scheme
/nersc/projects/starofl/embedding/${TRGSETUPNAME}/${PARTICLE}_&FSET;_${REAUESTID}/${PRODUCTION}.${LIBRARY}/${EMYEAR}/${EMDAY}
where "&FSET;" is 3 digit number determined at job submission (e.x. 100),
"${TRGSETUPNAME}" is trigger set up name (e.x. 2007ProductionMinBias),
"${PARTICLE}" is input MC particle name (e.x. JPsi),
"${REQUEST}" is the request id assigned for each embedding request,
"${PRODUCTION}" is the production tag for the real data (e.x. P07id),
"${LIBRARY}" is the library used for embedding simulation which is not
always matched with that for the real data production (e.x. SL08f),
"${EMYEAR}" and "${EMDAY}" are year and day number
extracted from the input file
The back up locations of relevant source codes, macros, scripts and xml files are
(starofl home) /home/starofl/embedding/CODE/${TRGSETUPNAME}/${PARTICLE}_${REQUESTID}/${PRODUCTION}.${LIBRARY}/${EMYEAR} (HPSS) /nersc/projects/starofl/embedding/CODE/${TRGSETUPNAME}/${PARTICLE}_${REQUESTID}/${PRODUCTION}.${LIBRARY}/${EMYEAR}
1. VFMCE chain option
Relevant discussions about VFMCE chain option can be found in the following links
2. Eta dip around eta ~ 0
Relevant discussions about eta dip problem can be found in
the bottom line from Yuri was
The problem was in StTrsMaker for two bfc options "TrsPileUp" and "TrsToF"
accounting particle time of flight in producing hits.
The above options activated TPC fiducial volume cut which removed a few cm near
membrane.
The cut has been removed and committed.
Hiroshi has confirmed that with this fix there is no dip at eta ~ 0.
Yuri
3. EmbeddingShortCut chain option
The details can be found in the following ticket in
The bottom line from Yuri was
EmbeddingShortCut means that TpcHitMover and dEdx makers
will not apply corrections for simulated data (IdTruth > 0 && IdTruth
< 10000 && QA > 0.95).
Trs has to have it. TrsRS should not have it.
Yuri
will not apply corrections for simulated data (IdTruth > 0 && IdTruth
< 10000 && QA > 0.95).
Trs has to have it. TrsRS should not have it.
Yuri
and
this option has really started to be used since release SL10c.
Till this release this option was always "ON" by default.
The only need for back propagation is when you will use release >= SL10c
with Trs. This correction will be done in dev for nightly tests.
Yuri
Till this release this option was always "ON" by default.
The only need for back propagation is when you will use release >= SL10c
with Trs. This correction will be done in dev for nightly tests.
Yuri
4. Bug in StAssociationMaker
See ticket in
For EC and ED(s)
Embedding instructions for EC and ED(s)
Last updated on Oct/23/2018 by Xianglei Zhu
Current Embedding Coordinator (EC): Xianglei Zhu (zhux@tsinghua.edu.cn)
Current Embedding Deputy (ED): Derek Anderson (derekwigwam9@tamu.edu)
Current NERSC Point Of Contact (POC): Jeff Porter (RJPorter@lbl.gov) and Jan Balewski (balewski@lbl.gov)
Revised history
- Jun/24/2018: initial version (copied from old instructions, still under construction)
Contents
- Instructions how to sample daq files and provide tags files
- Instructions how to set up xml file
- Production chain options
- Locations of outputs, log files and back up of relevant codes at HPSS
- Some useful discussions in hyper-news and bug-report
Please send me (zhux@rcf.rhic.bnl.gov) an e-mail if you have any questions/suggestions.
The typical xml file for the embedding job submission looks
<!-- Generated by StRoot/macros/embedding/get_embedding_xml.pl on Mon Aug 2 15:26:13 PDT 2010 --> <?xml version="1.0" encoding="utf-8"?> <job maxFilesPerProcess="1" fileListSyntax="paths"> <command> <!-- Load library --> starver SL07e <!-- Set tags file directory --> setenv EMBEDTAGDIR /eliza3/starprod/tags/ppProductionJPsi/P06id <!-- Set year and day from filename --> setenv EMYEAR `StRoot/macros/embedding/getYearDayFromFile.pl -y ${FILEBASENAME}` setenv EMDAY `StRoot/macros/embedding/getYearDayFromFile.pl -d ${FILEBASENAME}` <!-- Set log files area --> setenv EMLOGS /project/projectdirs/star/embedding <!-- Set HPSS outputs/LOG path --> setenv EMHPSS /nersc/projects/starofl/embedding/ppProductionJPsi/JPsi_&FSET;_20100601/P06id.SL07e/${EMYEAR}/${EMDAY} <!-- Print out EMYEAR and EMDAY and EMLOGS --> echo EMYEAR : $EMYEAR echo EMDAY : $EMDAY echo EMLOGS : $EMLOGS echo EMHPSS : $EMHPSS <!-- Start job --> echo 'Executing bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); ...' root4star -b <<EOF std::vector<Int_t> triggers; triggers.push_back(117705); triggers.push_back(137705); triggers.push_back(117701); .L StRoot/macros/embedding/bfcMixer_TpcSvtSsd.C bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); .q EOF ls -la . cp $EMLOGS/P06id/LOG/$JOBID.log ${FILEBASENAME}.$JOBID.log cp $EMLOGS/P06id/LOG/$JOBID.elog ${FILEBASENAME}.$JOBID.elog <!-- New command to organize log files --> mkdir -p $EMLOGS/P06id/JPsi_20100601/LOG/&FSET; mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/ <!-- Archive in HPSS --> hsi "mkdir -p $EMHPSS; prompt; cd $EMHPSS; mput *.root; mput ${FILEBASENAME}.$JOBID.log; mput ${FILEBASENAME}.$JOBID.elog" </command> <!-- Define locations of log/elog files --> <stdout URL="file:/project/projectdirs/star/embedding/P06id/LOG/$JOBID.log"/> <stderr URL="file:/project/projectdirs/star/embedding/P06id/LOG/$JOBID.elog"/> <!-- Input daq files --> <input URL="file:/eliza3/starprod/daq/2006/st*"/> <!-- csh/list files --> <Generator> <Location>/project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST</Location> </Generator> <!-- Put any locally-compiled stuffs into a sand-box --> <SandBox installer="ZIP"> <Package name="Localmakerlibs"> <File>file:./.sl44_gcc346/</File> <File>file:./StRoot/</File> <File>file:./pams/</File> </Package> </SandBox> </job>
Below is step by step instructions how to set it up for each request.
You can grab some xml files from other requests and modify manually
or can also create it by yourself with "StRoot/macros/embedding/get_embedding_xml.pl".
The option "-h or --help" will show all available options in "get_embedding_xml.pl".
1. Set up daq/tags files
Please contact POC at PDSF to locate daq/tags files if you don't find any daq/tags files
in the eliza disks at PDSF. The relevant descriptions in the xml file are
<!-- Input daq files --> <input URL="file:/eliza3/starprod/daq/2006/st*"/>
for daq files and
<!-- Set tags file directory --> setenv EMBEDTAGDIR /eliza3/starprod/tags/ppProductionJPsi/P06id
for tags files.
You can change the locations of daq/tags files in xml by "-daq [daq file path]" and "-tag [tags file path]" options like
> StRoot/macros/embedding/get_embedding_xml.pl -daq /eliza3/starprod/daq/2006 -tag /eliza3/starprod/tags/ppProductionJPsi/P06id
Note1-1: You can put the options in any order, so the command below gives the same result as above
> StRoot/macros/embedding/get_embedding_xml.pl -tag /eliza3/starprod/tags/ppProductionJPsi/P06id -daq /eliza3/starprod/daq/2006
Note1-2: If you have already created an xml file in your current directory,
"get_embedding_xml.pl" won't overwrite the previous xml file. If you want to overwrite it, put "-f" option.
2. Running job, archive outputs into HPSS
Below is the descriptions to run the job (bfcMixer), save log files, put outputs/logs into HPSS.
<!-- Start job --> echo 'Executing bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); ...' root4star -b <<EOF std::vector<Int_t> triggers; triggers.push_back(117705); triggers.push_back(137705); triggers.push_back(117701); .L StRoot/macros/embedding/bfcMixer_TpcSvtSsd.C bfcMixer_TpcSvtSsd(1000, 1, 1, "$INPUTFILE0", "$EMBEDTAGDIR/${FILEBASENAME}.tags.root", 0, 6.0, -1.5, 1.5, -200, 200, 160, 1, triggers, "P08ic", "FlatPt"); .q EOF ls -la . cp $EMLOGS/P06id/LOG/$JOBID.log ${FILEBASENAME}.$JOBID.log cp $EMLOGS/P06id/LOG/$JOBID.elog ${FILEBASENAME}.$JOBID.elog <!-- New command to organize log files --> mkdir -p $EMLOGS/P06id/JPsi_20100601/LOG/&FSET; mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/ <!-- Archive in HPSS --> hsi "mkdir -p $EMHPSS; prompt; cd $EMHPSS; mput *.root; mput ${FILEBASENAME}.$JOBID.log; mput ${FILEBASENAME}.$JOBID.elog"
The default bfcMixer is "StRoot/macros/embedding/bfcMixer_TpcSvtSsd.C".
The bfcMixer can be set by "-mixer [bfcMixer file]"
> StRoot/macros/embedding/get_embedding_xml.pl -mixer StRoot/macros/embedding/bfcMixer_Tpx.C
NOTE2-1: There are three different bfcMixer macros, bfcMixer_Tpx.C, bfcMixer_TpcSvtSsd.C and bfcMixer_TpcOnly.C.
You need to choose the proper bfcMixer macro depending on the Run;
<= Run4 : bfcMixer_TpcOnly.C
Run5 - Run7 : bfcMixer_TpcSvtSsd.C
>= Run8 : bfcMixer_Tpx.C
The library, production tag, trigger setup name and request number can be changed by using the following options;
"-production [production tag]", "-lib [library]", "-r [request number]" and "-trg [trigger setup name]"
> StRoot/macros/embedding/get_embedding_xml.pl -production P06id -lib SL07e -r 20100601 -trg ppProductionJPsi
The default library, production tag, and trigger setup name are SL08c, P08ic, 2007ProductionMinBias respectively
unless otherwise specified. These will be used for the locations of log files, scripts as well as the path in HPSS like
<!-- Load library --> starver SL07e ... ... ls -la . cp $EMLOGS/P06id/LOG/$JOBID.log ${FILEBASENAME}.$JOBID.log cp $EMLOGS/P06id/LOG/$JOBID.elog ${FILEBASENAME}.$JOBID.elog <!-- New command to organize log files --> mkdir -p $EMLOGS/P06id/JPsi_20100601/LOG/&FSET; mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/ <!-- Archive in HPSS --> hsi "mkdir -p $EMHPSS; prompt; cd $EMHPSS; mput *.root; mput ${FILEBASENAME}.$JOBID.log; mput ${FILEBASENAME}.$JOBID.elog" ... ... <!-- csh/list files --> <Generator> <Location>/project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST</Location> </Generator>
NOTE2-2: If the directories "/project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST"
doesn't exist, "get_embedding_xml.pl" will complain and doesn't generate xml like
Error: No /project/projectdirs/star/embedding/P06id/JPsi_20100601/LIST exists. Stop. Make sure you've put the correct path for generator file.
Currently, I didn't implement to automatically create "LOG" and "LIST" directories
in get_embedding_xml.pl. So you have to make those directories manually.
Please don't forget to make those directories group read/writable. Please contact me
if you are not clear what you should do (Hiroshi).
NOTE2-3: LOG directory structure has been changed to make them search easier,
and final log files will be moved to
mv $EMLOGS/P06id/LOG/$JOBID.* $EMLOGS/P06id/JPsi_20100601/LOG/&FSET;/
where we have introduced three additional directory "P06id" (production), "JPsi_20100601" (particle name and request id)
and "&FSET" (FSET number, which is determined during the job submission). This directory will be dynamically created
during the job submission, so you don't need to create it.
3. Arguments in the bfcMixer
You can find relevant informations for these numbers in the
STAR simulation request page, http://drupal.star.bnl.gov/STAR/starsimrequest.
Below is an example for the 2006 J/Psi in p + p 200 GeV request
Below is the details how to set each argument in the bfcMixer according to
the informations in the simrequest page
3-1. Particle id and particle name
The particle geantid can be found in "pams/sim/gstar/gstar_part.g".
You can see the J/Psi -> e+e- decay (100% B.R.)
Particle Jpsi code=160 TrkTyp=4 mass=3.096 charge=0 tlife=7.48e-21,pdg=443 bratio= { 1, } mode= { 203, }
"code=160" is the geantid for J/Psi so you can set the geantid and particle name by
"-geantid [GEANT3 id]" and "-particle [particle name]"
> StRoot/macros/embedding/get_embedding_xml.pl -geantid 160 -particle JPsi
Please don't put "/" in the particle name. The particle name is also used for the directory
in HPSS as well as in eliza disk to store the outputs. So "/" would be recognized as
directory.
3-2. Particle settings (how to simulate pt distribution)
Particle settings is usually "Flat in pt", generating the flat pt distribution of input MC particles,
which corresponds to the default option "FlatPt" in the last argument of bfcMixer.
There are two other options depending on how to simulate the input MC particles
"Strange" : Smear primary vertices in all (x,y,z) directions with vertex errors stored in tags files
"Spectrum" : Generate transverse momentum by pt*exp(-pt/T) shape, where T is inverse slope parameter
Default T is 0.3 GeV
These options can be changed by "-mode [option]"
> StRoot/macros/embedding/get_embedding_xml.pl -mode Strange
NOTE3-1: The mode option string is case insensitive, so "Strange", "strange", "STRANGE"
or any combination of letters can be accepted as long as it matches with "strange" string.
NOTE3-2: "Strange" option cannot be run in "VFMCE" chain option.
You should disable VFMCE by "-VFMCE" and put the appropriate vertex finder option back to the chain options.
NOTE3-3: "Spectrum" option can only be run after SL08f. We don't have any solutions at this moment except for
doing single MC simulation and embedding, separately.
3-3. Multiplicity
The default multiplicity is 1, i.e. throw 1 input MC particle per event. You can change it by "-mult [multiplicity]"
> StRoot/macros/embedding/get_embedding_xml.pl -mult 0.05
if the multiplicity is less than 1, the input MC particle will be generated
by the fraction of multiplicity from real data.
The 0.05 gives 5% of multiplicity per event, so the number of particles
will be varied event-by-event depending on the multiplicity.
In the current example, we don't need to modify the multiplicity.
3-4. Primary z-vertex range
The minimum and maximum z-vertex cuts can be set by
"-zmin [Minimum z-vertex value]" and "-zmax [Maximum z-vertex value]"
> StRoot/macros/embedding/get_embedding_xml.pl -zmin -30.0 -zmax 30.0
if PA requests some specific z-vertex cut. The default z-vertex cut is |vz| < 200 cm.
In the current J/Psi case, we don't need to put any z-vertex cut.
3-5. Rapidity and transverse momentum range
The rapidity and transverse momentum cuts can be set by
"-ymin [Minimum y value]", "-ymax [Maximum y value]",
"-ptmin [Minimum pt value]" and "-ptmax [Maximum pt value]"
> StRoot/macros/embedding/get_embedding_xml.pl -ymin -1.0 -ymax 1.0 -ptmin 0.0 -ptmax 6.0
NOTE3-4: The simulation request shows the eta range, but we'll throw in rapidity instead of eta.
The default rapidity and pt cuts are |y| < 1.5 and 0 < pt < 10 GeV/c, respectively.
For the current example, we only need to modify maximum pt cut off to 6 GeV/c
3-6. Trigger id's
If PA request trigger id(s), you can put them by using "-trigger [trigger id]"
> StRoot/macros/embedding/get_embedding_xml.pl -trigger 117705
you can add multiple triggers by adding more "-trigger [trigger id]" like
> StRoot/macros/embedding/get_embedding_xml.pl -trggier 117705 -trigger 137705 -trigger 117001
minor NOTE3-5: (This doesn't work, please don't try) I had try to implement the "-trigger" argument like
> StRoot/macros/embedding/get_embedding_xml.pl -trigger 117705 137705 ...
to accept multiple numbers with single "-trigger" option, but couldn't succeed with the current
perl version at PDSF/RCF, though it worked with newer perl in my Macbook (Hiroshi).
3-7. Production chain name
In order to use the correct chain option in the bfcMixer, you should set the production chain name like
StRoot/macros/embedding/get_embedding_xml.pl -prodname P06idpp
NOTE3-6: It might not be trivial what kind of chain options are available in the bfxMixer's.
Please take a look at the bfcMixer to make sure that the relevant chain option has been implemented.
Including the all relevant options in get_embedding_xml.pl, the command to produce the J/Psi xml file is
> StRoot/macros/embedding/get_embedding_xml.pl -f -daq /eliza3/starprod/daq/2006 -tag /eliza3/starprod/tags/ppProductionJPsi/P06id \ -production P06id -lib SL07e -r 20100601 -trg ppProductionJPsi -geantid 160 -particle JPsi -ptmax 6.0 -trigger 117705 -trigger 137705 -trigger 117701 \ -prodname P06idpp
There a few additional things we need to modify xml file manually to get the final form shown in the beginning
- First, we also need to set 0 for the 2nd/3rd arguments in the bfcMixer_TpcSvtSsd.C.
Those are flags to turn on SVT (2nd) and SSD (3rd) if they are 1. Since this example is
the request for Run6 p+p, we don't need SVT/SSD and hence must turn them off, i.e. set to be 0.
- Second, 100 % J/Psi -> e+e- has already been implemented in the "pams/sim/gstar/gstar_part.g" and
is available in the default STAR library so we actually don't need to check out pams and recompile.
In that case, you also need to remove "pams" in your "Localmakerlibs".
- Third, PA requests 2 different triggers in "st_jpsi" stream and one trigger in "st_physics" stream
so the xml files should be prepared for those two different streams with different triggers/inputs.
There would be couple of other things we have to do manually, which will depend on each request.
Please let us know your feedback in case you had some special modification on your xml file.
Production chain options
Production chain can be found in http://www.star.bnl.gov/devcgi/dbProdOptionRetrv.pl
Since the current embedding must use "VFMCE" chain option, other vertex finder chain options have to
be excluded. The relevant informations about vertex finder chain options can be found
Below is the current procedure how to set up production chain in the bfcMixer's
- ED will set up the proper bfcMixer macro with the correct chain options
- EC and POC at PDSF will take a look and give feedback
- Ask Lidia for her inputs, and verify the new chain with Lidia and Yuri
- Enter all chains into Drupal embedding page for documentation
- Commit bfcMixer into CVS
Below is the approved chains implemented in the bfcMixer at this moment (Jun/11/2010)
Chains approvied by Lidia
--------------------------------
P07ic CuCu production: TString prodP07icAuAu("P2005b DbV20070518 MakeEvent ITTF ToF ssddat spt SsdIt SvtIt pmdRaw OGridLeak OShortR OSpaceZ2 KeepSvtHit skip1row VFMCE -VFMinuit -hitfilt");
P08ic AuAu production: DbV20080418 B2007g ITTF adcOnly IAna KeepSvtHit VFMCE -hitfilt l3onl emcDY2 fpd ftpc trgd ZDCvtx svtIT ssdIT Corr5 -dstout
If spacecharge and gridleak corrections are on average instead of event by event then Corr5-> Corr4, OGridLeak3D, OSpaceZ2.
P08ie dAu production : DbV20090213 P2008 ITTF OSpaceZ2 OGridLeak3D beamLine, VFMCE TpcClu -VFMinuit -hitfilt
TString chain20pt("NoInput,PrepEmbed,gen_T,geomT,sim_T,trs,-ittf,-tpc_daq,nodefault);
P06id pp production : TString prodP06idpp("DbV20060729 pp2006b ITTF OSpaceZ2 OGridLeak3D VFMCE -VFPPVnoCTB -hitfilt");
P06ie pp production : TString prodP06iepp("DbV20060915 pp2006b ITTF OSpaceZ2 OGridLeak3D VFMCE -VFPPVnoCTB -hitfilt"); run# 7096005-7156040
TString prodP06iepp("DbV20061021 pp2006b ITTF OSpaceZ2 OGridLeak3D VFMCE -VFPPVnoCTB -hitfilt"); run# 7071001-709402
Locations of outputs, log files and back up of relevant codes at HPSS
Current output as well as log file location at HPSS is determined by the following scheme
/nersc/projects/starofl/embedding/${TRGSETUPNAME}/${PARTICLE}_&FSET;_${REAUESTID}/${PRODUCTION}.${LIBRARY}/${EMYEAR}/${EMDAY}
where "&FSET;" is 3 digit number determined at job submission (e.x. 100),
"${TRGSETUPNAME}" is trigger set up name (e.x. 2007ProductionMinBias),
"${PARTICLE}" is input MC particle name (e.x. JPsi),
"${REQUEST}" is the request id assigned for each embedding request,
"${PRODUCTION}" is the production tag for the real data (e.x. P07id),
"${LIBRARY}" is the library used for embedding simulation which is not
always matched with that for the real data production (e.x. SL08f),
"${EMYEAR}" and "${EMDAY}" are year and day number
extracted from the input file
The back up locations of relevant source codes, macros, scripts and xml files are
(starofl home) /home/starofl/embedding/CODE/${TRGSETUPNAME}/${PARTICLE}_${REQUESTID}/${PRODUCTION}.${LIBRARY}/${EMYEAR} (HPSS) /nersc/projects/starofl/embedding/CODE/${TRGSETUPNAME}/${PARTICLE}_${REQUESTID}/${PRODUCTION}.${LIBRARY}/${EMYEAR}
1. VFMCE chain option
Relevant discussions about VFMCE chain option can be found in the following links
2. Eta dip around eta ~ 0
Relevant discussions about eta dip problem can be found in
the bottom line from Yuri was
The problem was in StTrsMaker for two bfc options "TrsPileUp" and "TrsToF"
accounting particle time of flight in producing hits.
The above options activated TPC fiducial volume cut which removed a few cm near
membrane.
The cut has been removed and committed.
Hiroshi has confirmed that with this fix there is no dip at eta ~ 0.
Yuri
3. EmbeddingShortCut chain option
The details can be found in the following ticket in
The bottom line from Yuri was
EmbeddingShortCut means that TpcHitMover and dEdx makers
will not apply corrections for simulated data (IdTruth > 0 && IdTruth
< 10000 && QA > 0.95).
Trs has to have it. TrsRS should not have it.
Yuri
will not apply corrections for simulated data (IdTruth > 0 && IdTruth
< 10000 && QA > 0.95).
Trs has to have it. TrsRS should not have it.
Yuri
and
this option has really started to be used since release SL10c.
Till this release this option was always "ON" by default.
The only need for back propagation is when you will use release >= SL10c
with Trs. This correction will be done in dev for nightly tests.
Yuri
Till this release this option was always "ON" by default.
The only need for back propagation is when you will use release >= SL10c
with Trs. This correction will be done in dev for nightly tests.
Yuri
4. Bug in StAssociationMaker
See ticket in
For Embedding Helpers (OBSOLETE)
Embedding instructions for Embedding Helpers
This instructions provide for embedding helpers how to prepare/submit the embedding jobs at PDSF
You can also find the documentation about the embedding QA in the "Links" section.
NOTE: This is specific instructions at PDSF, some procedures may not work at RCF
Current Embedding Coordinator (EC): Terence Tarnowsky (tarnowsk@nscl.msu.edu)
Current Embedding Deputy (ED): Xianglei Zhu (zhux@tsinghua.edu.cn),
Christopher Powell (CBPowell@lbl.gov)
Current PDSF Point Of Contact (POC): Jeff Porter (RJPorter@lbl.gov)
Last updated on Nov/8/2011 by T. Tarnowsky
Revised history (Most recent first)
-
Nov/8/2011: Updated EC and ED information to bring it in line with other webpages.
-
Apr/12/2011: Update Section 3.1, remove obsolete procedure for StPrepEmbedMaker
-
Feb/17/2011: Update Section 3.2
-
Nov/17/2010: Update to get afs token to access afs area
-
Sep/08/2010: Update memory limit in the batch at PDSF (Section 1)
-
Aug/25/2010: Update Section 3.2
-
May/25/2010: Update Section 3 (include klog, library and NOTE about VFMCE)
-
May/21/2010: Added "How to setup your environment at PDSF ?" and "Links"
-
May/20/2010: Updated instructions
Contents
1. How to set up your environment at PDSF ?
Please go to the PDSF-wiki page to set up your environment at PDSF,
in case you have any troubles.
Please also don't forget to subscribe "Embedding issues and discussions" hypernews
forum (embedding-hn@www.star.bnl.gov).
Important note:
Please have a look at the "common issue: memory limit in batch"
and follow the procedure how to increase the memory limit in the batch jobs.
All EH should make this change before submitting any embedding production jobs.
2. Set up daq and tags files at PDSF
Please contact POC at PDSF to locate proper daq and tags files at eliza disks at PDSF.
3. Production area setup
3-0. Get afs token to access CVS
Please don't forget this procedure (How to klog to my RHIC afs account ?) before using cvs.
Below is the copy from the link above what you need to do in order to get afs token to access CVS
PDSF does not belong to the rhic.bnl.gov afs cell (the default cell is nersc),
so you have to specify the rhic.bnl.gov cell explicitly.
Additionally, your PDSF username may be different than on RACF.
If so, you need to specify your afs account name explicitly as well.
If so, you need to specify your afs account name explicitly as well.
> klog -cell rhic -principal YourRCFUserName
3-1. Check out relevant macros/scripts and Makers from CVS
> cvs co StRoot/macros/embedding > cvs co StRoot/St_geant_Maker
"StRoot/macros/embedding" should contain all relevant macros/scripts for the embedding productions. "StRoot/St_geant_Maker" area has "StPrepEmbedMaker" which is the main code to process the embedding.
****** THIS PROCEDURE IS OBSOLETE, SO YOU DON'T NEED TO FOLLOW ANYMORE ******
NOTE: Under the current environment at PDSF, you will not be able to compile St_geant_Maker by cons. So you need move "StRoot/St_geant_Maker/Embed/StPrepEmbedMaker.*" one directory up, then compile by cons
> mv StRoot/St_geant_Maker/Embed/StPrepEmbedMaker* StRoot/St_geant_Maker/ > starver ${library} > cons
****************************************************************************************************************
Please don't forget to set the proper library "${library}" before compiling the source codes.
In following sections, we assume you have already did "starver ${library}".
The requested library can be found in simulation request page.
NOTE: Any productions prior to P07ie should be simulated under SL07e due to the lack of "VFMCE"
chain options in older libraries, even if PA's request a specific library (like SL06d).
3-2. Set up input MC particle
The input MC particle will be specified by using the geantid (see GEANT3 manual around page number 61 for pre-defined particle).
Take a look at "$STAR/pams/sim/gstar/gstar_part.g", if you cannot find the geantid for the requested particle
> cvs co pams/sim/gstar > cons
Note that you need to check out the whole gstar directory in order to properly load gstar library.
The particle you need to simulate must be in either "$STAR/pams/sim/gstar/gstar_part.g" or
local "pams/sim/gstar/gstar_part.g" code. Please also take a look at the details about the description
of particles in "For EC and ED(s)", Section 3.1.
Please contact EC or ED if you don't find requested particles in the "pams/sim/gstar/gstar_part.g".
3-3. Set up bfcMixer macro
Please contact EC or ED whether bfcMixer (either bfcMixer_TpcSvtSsd.C or bfcMixer_Tpx.C) is ready to submit or not, and confirm which bfcMixer should be used for the current request.
3-4. Set up xml file for job submission
3-5. Other code modifications
For other code modifications needed, please check the libraries installed locally under the star embedding production account (starofl), and contact POC at PDSF for assistance.
4. Submit jobs
4-1. Local sandbox
Local compiled codes/libraries are archived as "Localmakerlibs.zip" and used for each job submission.
Default local sandbox should be
<!-- Put any locally-compiled stuffs into a sand-box --> <SandBox installer="ZIP"> <Package name="Localmakerlibs"> <File>file:./.sl44_gcc346/</File> <File>file:./StRoot/</File> <File>file:./pams/</File> </Package> </SandBox>
in your xml file. If you have anything other than above codes, please include them.
Please contact EC or ED if you are not clear enough which codes you need to include.
4-2. Submit jobs by scheduler
Once everything is prepared, you can submit jobs like
> star-submit-template -template embed_template.xml -entities FSET=200
where "embed_template.xml" is the xml file, and FSET is the unique 3 digit number
for a given request (typically starting from 100 or 101).
The embedding templates at PDSF are set up to write the output files to HPSS at the end of each job.
When all jobs in one FSET are finished, the outputs need to be retrieved from HPSS.
Please contact POC at PDSF which eliza disk can be used for your outputs.
The xml file "schedTemplateExp.xml", which will be automatically generated once you submit job by scheduler,
should be kept for each FSET job under "setup" directory.
And please don't submit the same FSET jobs, that will overwrite your previous outputs in HPSS.
4-3. Re-submitting jobs
Sometime you may need to modify something under "StRoot" or "pams", and recompile to fix some problems.
Each time you recompiled your local codes, you should clean up the current "Localmakerlibs.zip" and
"Localmakerlibs.package/" before starting resubmission. If you forgot to clean up the older "Localmakerlibs",
then the modification in the local codes will not reflect in the resubmitting jobs.
5. Back up macros, scripts, xml file and codes
Please contact EC or ED for backing up all relevant macros, scripts and codes in the local starofl area as well as HPSS at PDSF. Please also remind to clean up all relevant LOG and LIST files that have been generated by scheduler after PWG QA is done by PA.
6. Links
Obsolete documentations
Embedding Production Setup
This page describes how to set up embedding production. This procedure needs to be followed for any set of daq files/production version that requires embedding. Since this typically involves hacking the reconstruction chain, it is not advised that the typical STAR PA attempt this step. Please coordinate with a local embedding coordinator, and the overal Embedding coordinator (Olga).Note: The documentation here is very terse; it will be enriched as the documentation as a whole is iterated on. Patience is appreciated.
Get daq files from RCF.
| Grab a set of daq files from RCF which cover the lifetime of the run, the luminosity range experienced, and the conditions for the production. |
Rerun standard production but without corrections.
| bfc.C macros are located under ~starofl/bfc. Edit the submit.[Production] script to point to the daq files loaded (as above). |
Put tags files on disk.
| The results of the previous jobs will be .tags.root files located on HPSS. Retrieve the files, set a pointer for the tags files in the Production-specific directory under ~starofl/embedding. |
Now you're ready to start production.
Embedding Setup Off-site
Introduction
The purpose of this document is to describe step-by-step the setting up of embedding infrastructure on a remote site i.e. not at it's current home which is PDSF. It is based on the experience of setting up embedding at Birmingham's NP cluster (Bham). I will try to maintain a distinction between steps which are necessary in general and those which were specific to porting things to Bham. It should also be a useful guide for those wanting to run embedding at PDSF and needing to copy the relevant files into a suitable directory structure.Pre-requisites
Before trying to set up embedding on a remote site you should have:- a working local installation of the STAR library in which you are interested (or be satisified with your AFS-based library performance).
- a working mirror of the star database (or be satisfied with your connection to the BNL hosted db).
Collect scripts
The scripts are currently housed at PDSF in the 'starofl' account area. At the time of writing (and the time at which I set up embedding in Bham) they are not archived in CVS. The suggested way to collect them is to copy them into a directory in your own PDSF home account then tar and export it for installation on your local cluster. The top directory for embedding is /u/starofl/embedding . Under this directory there are several subdirectories of interest.- Those named after each production, e.g. P06ib which contain mixer macro and perl scripts
- Common which contains further subdirectories lists and csh and a submission perl script
- GSTAR which contains the kumac for running the simulation
mkdir embedding
cd embedding
mkdir Common
mkdir Common/lists
mkdir Common/csh
mkdir GSTAR
mkdir P06ib
mkdir P06ib/setup
Now it needs populating with the relevant files. In the following /u/user/embedding as an example of your new embedding directory in your user home directory.
cd /u/user/embedding
cp /u/starofl/embedding/getVerticesFromTags_v4.C .
cp -R /u/starofl/embedding/P06ib/EmbeddingLib_v4_noFTPC/ P06ib/
cp /u/starofl/embedding/P06ib/Embedding_sge_noFTPC.pl P06ib/
cp /u/starofl/embedding/P06ib/bfcMixer_v4_noFTPC.C P06ib/
cp /u/starofl/embedding/P06ib/submit.starofl.pl P06ib/submit.user.pl
cp /u/starofl/embedding/P06ib/setup/Piminus_101_spectra.setup P06ib/setup/
cp /u/starofl/embedding/GSTAR/phasespace_P06ib_revfullfield.kumac GSTAR/
cp /u/starofl/embedding/GSTAR/phasespace_P06ib_fullfield.kumac GSTAR/
cp /u/starofl/embedding/Common/submit_sge.pl Common/
You now have all the files need to run embedding. There are further links to make but as you are going to export them to your own cluster you need to make the links afterwards.
Alternatively you can run embedding on PDSF from your home directory. There are a number of change to make first though because the various perl scripts have some paths relating to the starofl account inside them.
For those planning to export to a remote site you should tar and/or scp the data. I would recommend tar so that you can have the original package preserved in case something goes wrong. E.g.
tar -cvf embedding.tar embedding/
scp embedding.tar remoteuser@mycluster.blah.blah:/home/remoteuser
Obviously this step is unnecessary if you intend to run from your PDSF account although you may still want to create a tar file so that you can undo any changes which are wrong.
Login to your remote cluster and extract the archive. E.gcd /home/remoteuser
tar -xvf embedding.tar
Script changes
The most obvious thing you will find are a number of places inside the perl scripts where the path or location for other scripts appears in the code. These must be changed accordingly.
P06ib/Embedding_sge_noFTPC.pl
changes to e.g.changes to e.g.changes to e.g.changes to e.g.
P06ib/EmbeddingLib_v4_noFTPC/Process_object.pm
changes to e.g.
changes to e.g.
This is because the location of tcsh was different and probably will be for you too.
Common/submit_sge.pl
changes to e.g.
Change relates to parsing the name of the directory with daq files in to extract the 'data vault' and 'magnetic field' which form part of job name and are used by Embedding_sge_noFTPC.pl (This may not make much sense right now and needs the detailed docs on each component. It is actually just a way to pass a file list with the same basename as the job). In the original script the path to the data is something like/dante3/starprod/daq/2005/cuProductionMinBias/FullField
whereas on Bham cluster it is/star/data1/daq/2005/cuProductionMinBias/FullField
and thus the pattern match in perl has to change in order to extract the same information. If you have a choice then choose your directory names with care!changes to e.g.
Change relates to the line printing the job submission shell script that this perl script writes and submits. The first line had to be changed such that it can correctly be identified as a sh script. I am not sure how original can ever have worked?changes to e.g.
This line prints part of the job submission script where the options for the job are specified. In SGE the job options can be in the file and not just on the command line. The extra options for Bham relate to our SGE setup. The-q
option provides the name of the queue to use, otherwise it uses the default which I did not want in this case. The other extra options are to make the environment and working diretory correct as they were not the default for us. This is very specific to each cluster. If your cluster does not have SGE then I imagine extensive changes to the part writing the job submission script would be necessary. The scripts use the ability of SGE to have job arrays of similar jobs so you would have to emulate that somehow.
- getVerticesFromTags_v4.C - none
- GSTAR/phasespace_P06ib_fullfield.kumac, GSTAR/phasespace_P06ib_fullfield.kumac - actually there are changes but they only relate to redefining particle decay modes for (anti-)Ξ and (anti-)Ω to go 100% to charged modes of interest. This is only relevant for strangeness group
- P06ib/bfcMixer_v4_noFTPC.C - checked carefully that
chain3->SetFlagsline actually sets the same flags since Andrew and I had to change the same flags e.g. add GeantOut option after I made orginal copy - P06ib/EmbeddingLib_v4_noFTPC/Chain_object.pm - none
- P06ib/EmbeddingLib_v4_noFTPC/EmbeddingUtilities.pm - there are lines where you may have to add the run numbers of the daq files which you are using so that they are recognised as either full field or reversed full field. In this example (Cu+Cu embedding in P06ib) the lines begin
and. This is also something that Andrew and I both changed after I made the original copy. - P06ib/submit.user.pl - changes here relate to setup that you want to run and not to the cluster or directory you are using i.e. which setup file to use, what daq directories to use and any pattern match on the file names (usually for testing purposes to avoid filling the cluster with useless jobs) although you probably want to change the
line! - P06ib/setup/Piminus_101_spectra.setup - any changes here relate to the simulation parameters of the job that you want to do and not to the cluster or directory you are using
Create links
A number of links are required. For example in the /u/starofl/embedding/P06ib there are the following links:daq_dir_2005_cuPMBFF -> /dante3/starprod/daq/2005/cuProductionMinBias/FullFielddaq_dir_2005_cuPMBRFF -> /dante3/starprod/daq/2005/cuProductionMinBias/ReversedFullFielddaq_dir_2005_cuPMBHTFF -> /eliza5/starprod/daq/2005/cucuProductionHT/FullField/daq_dir_2005_cuPMBHTRFF -> /eliza5/starprod/daq/2005/cucuProductionHT/ReversedFullFieldtags_dir_cuHT_2005 -> /eliza5/starprod/embedding/tags/P06ibdata -> /eliza12/starprod/embedding/datalists ->../Common/listscsh-> ../Common/cshLOG-> ../Common/LOG
tags_dir_cu_2005 -> /dante3/starprod/tags/P06ib/2005
That is it! Some things will probably need to be adapted to your circumstances but it should give you a good idea of what to do
Author: Lee Barnby, University of Birmingham (using starembed account)
Modified: A. Rose, Lawrence Berkeley National Laboratory (using starembed account)
Modified Birmingham Files
Upload of modified embedding infrastructure files used on Birmingham NP cluster for Cu+Cu for (anti-)Λ and K0S embedding request.Production Management
1) Usually embedding jobs are run in "HPSS" mode so the files end up in HPSS (via FTP). To transfer them from HPSS to disk copy the perl script ~starofl/hjort/getEmbed.pl and modify as needed. This script does at least two things that are not possible with, e.g., a command line hsi command: it only gets the files needed (usually the .geant and .event files) and it changes the permissions after the transfers. Note that if you do the transfers shortly after running the jobs the files will probably still be on the HPSS disk cache and transfers will be much fast than getting the files from tapes.2) To clean up old embedding files make your own copy of ~starofl/hjort/embedAge.pl and use as needed. Note that $accThresh determines the maximum access time in days of files that will not be deleted.
Running Embedding
This page describes how to run embedding jobs once the daq files and tags files are in place (see other page about embedding production setup).Basics:
Embedding code is located in production specific directories: ~starofl/embedding/P0xxx. The basic job submission template is typically called submit.starofl.pl in that directory.Jobs are usually run by user starofl but personal accounts with group starprod membership will work, too (but test first as the group starprod write permissions typically are not in place by default).
The script to submit a set of jobs is submit.[user].pl. The script should be modified to submit an embedding set from the the configuration file
~starofl/embedding/[Production]/setup/[Particle]_[set]_[ID].setup
where
[Particle] is the particle type submitted (Piminus for GEANTID=9, as set inside file)
[set] is the file set submitted (more on this later)
[ID] is the embedding request number