CSMStatusUtils

This is CSMStatusUtils, which outputs the status of either calorimeter to text and root status files.  Documentation is by Dave Relyea.  The code can be found at

StRoot/StEmcPool/CSMStatusUtils

History:
I was given code from both Joanna and Thorsten to figure out the status of the calorimeter towers over all pp production runs.  I merged the two sets of code, and and created a package called CSMStatusUtils.

Method:
The first piece of code in CSMSU takes every run and fills a 2-d histogram of ALL channels vs hit number (in ADC counts, from 0 to 150).  A second routine then combines these histograms from run subsets into single histograms for each run.  From the second routine and on down, the EEMC and BEMC are done entirely separately; the code needs to be run twice, once for each detector.

The code has an algorithm which then takes runs in each fill and combines them until the average number of hits above pedestal for all channels is greater than 100.  If runs are left over at the end of a fill, their statistics are added to prior runs in the fill.

For each combined set of runs, the code puts each channel through a series of tests.  It finds the pedestal (and writes it out to a file, btw) and determines if the pedestal is abnormally wide, or whether it falls outside acceptable limits (ADC channels 0 to 3, or 147 to 150).  It compares all towers' mean number of hits (ten sigma) above pedestal, and then flags towers which have 10x as many hits as the average, or 40x fewer.  Finally, it looks for stuck bits (either on or off) in the 1, 2, 4, or 8 position, and flags channels with stuck bits.

The code writes out a table (in text format) for each set of runs with the status of each channel clearly marked.  This table is also written in ROOT format, to be read by existing BEMC algorithms.  Also written is a hot tower plot, so the hot tower results can be eyeballed.  The code can also write out gif files of the spectra of every channel that failed a status test, so long as the number of channels in a given run set that failed the test is less than 25 (**for 2004 pp, gif files were not written out**).  Finally, the code creates a nice html file containing links to html subfiles detailing the channel status for each run set, which in turn link to the gif files.

As a final step, the code takes the text files and creates a new series of text status files with the results in differential format, meaning channels whose status didn't change from run set to run set are omitted.  However, since some channels fall very near the thresholds of certain status tests (for instance, channels whose pedestals sit at 2.9 ADC counts), I make the requirement that the channel's status must not have changed more than ten percent of the time over all runs sets, excluding runs in which all channels were bad (for nominal production running, this needs to be done, of course!).  If it has, it is marked bad once at the beginning, and then does not appear in any of the differential files.


************************************************************************
Running:
To run CSMSU, the first step is to use the FileCatalog to create a list of all the files you wish to analyze.  The command I use is typically something like:

get_file_list.pl -keys 'path,filename' -cond 'production=P04ik,tpc=1,emc=1, trgsetupname=productionPP||productionPPnoEndcap||pp||PP,filename~st_physics,filetype=daq_reco_mudst' -onefile -limit 100000 -distinct > allthephysicsfiles

Note that the output format I use is just 'path,filename', and I keep the :: delimiter that the FileCatalog uses.  My next step is to call

CSMSU/scripts/analysis0 allthephysicsfiles

(YOU NEED TO CHANGE THE HEADER IN THIS FILE TO YOUR OWN OUTPUT DIRECTORY)

which takes the "allthephysicsfiles" file from FileCatalog, splits it up into groups of 20 miniruns, and submits the entire processing job to batch.  Note - if I knew how to use the XML submission scripts, I would, but the online documentation for them doesn't mention how to code up your macro (.C) file such that the XML header file will work.  No matter.

PLEASE NOTE: Each minirun will generate about a 200k file.  This adds up to ENORMOUS disk space for large runs.  The 2004 pp run takes up about 1.4 Gig.  The 2004 Au-Au run would be even larger.  Thus, I really need to learn how to
use the XML submission scripts.

(FROM HERE ON, SCRIPT FILES CALL MACROS IN THE MACROS DIRECTORY.  YOU NEED TO CHANGE THE DEFAULT ARGUMENTS OF THE MACROS TO YOUR WORKING DIRECTORY.  YOU CAN DO THIS IN THE SCRIPT, OR JUST MAKE YOUR OWN SCRIPT, SINCE THEY'RE ALL TRIVIAL ONE LINE SCRIPTS ANYWAY.  SORRY ABOUT THE CAPS.)

After all miniruns have been processed, the next step is to combine them into runs.  The script "analysis1" does this.  

Next, you want to run the actual status code on the files.  The script "analysis2" does this.  PLEASE NOTE:  this macro requires an x window, as root needs to be able to Draw certain things.  I don't know how to do this in batch,
so I always run this interactively.  It's not a good solution, but for now, it's a solution.

Finally, you want to generate the ".root" status files and the concatenated status files (to alert you to changes in calorimeter tower status).  The script "analysis3" does this.

StBSMDStatusMaker

This page gives a brief overview of the code Frank Simon developed to create SMD Status Tables from MuDSTs.

Some features and limitiations of the code:

• Create one status table per fill for each SMD plane (if enough statistics are available)
• all triggers are used to maximize statistics (this can be a potential problem if there are hot trigger towers)
• pedestals are taken from the data base (MuDST SMD data are zero-suppressed)
• db-readable status tables can be created. Currently there are only to status flags implemented: good and bad. Adding more "variety" is straigth forward
• can be used with the STAR scheduler, no specific ordering of the input files required (although some structure in the job submission is recommended, see below)

Basic ideas behind the code:

• based on Daves tower status code, but: there are some very important differences
• One job runs over several MuDST files, when a new fill number is encountered, a new output file is opened. That way, the jobs can be run with the STAR scheduler and they can use files on distributed storage, since no particular ordering of the input files is needed
• for each fill, a file with 18000 channel TH2Fs storing amplitude information for each SMD channel is created (this method of dealing with random file order is a bit disk space hungry, so make sure enough space is available (~ 10 GB for 2005 pp); this can be deleted after the next step), other information such as time stamps and pedestals are stored in text files
• As a next step, the large number of files created by jobs on MuDSTs is consolidated into one file per fill
• Status tables are created from each of those files (one per fill, if statistics are sufficient, otherwise no status table for that fill is created)
• db readable files are produced from these status tables

Running the code

• Copy and compile the code in StRoot/StBSMDStatusMaker
• Copy the macro that runs the code: RunStatus.C
• Create scheduler scripts to submit your jobs. For pp2005, 50 jobs per job seems to give jobs with useful runtimes. In order to not get a totally randomized fill distribution in your jobs, submit them by day. A macro that creates a .csh that you can use to submit jobs by day is CreateSubmitScript.C, a template job describtion (you have to modify tha paths to suit your needs) is pp2005Template.xml
• Submit your jobs
• Once all jobs are done, create a list of all output files via ls * > FileList.list in the directory where your output ended up
• Consolidate data using the file list: run macro DoAdding.C after compiling AddHistograms.C (via .L AddHistograms.C++), this macro takes the directory where the files are located and the file list as arguments. This creates three files per fill: Fill*.root containing the histograms, Fill*.ped containing pedestal db information and Fill*.time containing the (approximate) start time of the fill
• Create a list of root files via ls Fill*.root > FileList.list and convert it into a list of fill numbers using the macro GetListOfFills.C. This maco takes the file list and a file name for the fill number list as argument
• Perform the status table creation. For that, compile the shared library StatusTools.C (via .L StatusTools.C++), then run the macro ProcessList.C, with arguments RunList (created previously) and the directory where the Fill* files are located. The output of this is a number of files per fill (root file, flag file and time stamp file). The flag and the time stamp file are needed to create the db readable status table, the root file contains histograms created during the analysis process.
• Create db readable status tables. This is done by running the macro WriteStatusFiles.C. This takes the directory where the flag and timestamp files are located as an argument. Two important notes:
  • This macro needs the full STAR environment (all other steps above except the running of jobs can be done on standalone machines)
  • The order in which the flag and time files are written (created) is crucial, since gSystem->GetDirEntry() is used to loop over all files. So care has to be taken if the files are copied from somewhere else
• Copy the db readable files to the database location in StarDb, and test them! Use TestStatusFiles.C for example.

For questions, please don't hesitate to contact me at fsimon@mit.edu!

BEMC Online Trigger Monitoring

During data taking the BEMC trigger information is monitored, and changes in the configuration (new pedestals, masks, etc.) are recorded.  The code relevant to this online trigger monitoring was developed by Oleksandr, and is checked into CVS here (usage instructions in the README file).  The scripts execute via a cronjob on the online machines on the starp network.  In particular, there exists directories with results from previous years at /ldaphome/onlmon/bemctrgdb20XX/ on onl08.starp.bnl.gov.

The final location for this information is in the offline DB, and the definitions for the tables are given here.  These DB tables are used by the StBemcTriggerSimu in the StTriggerUtilities package to replicate the BEMC trigger conditions for a particular timestamp. 

The DB tables can be uploaded while taking data or stored in ROOT files to be uploaded after data taking is complete.  To upload all the tables stored in ROOT files during data taking only a simple script is needed employing StBemcTablesWriter to read in a list of files and upload their information to the DB.  This script (uploadToDB.C) is checked into the macros directory of in CVS here.