Online QA (old)

 [Please note that the tools documented here are being phased out. A new Online QA is available.]

 

STAR Online Monitoring

The STAR online monitoring system consists of two applications currently running on a quad-CPU Intel-i386 Linux machine.

The first application is the evpServer. During data taking, its main functionality is to constantly read the most recent event from the event pool and fill user defined histograms which are written into a memory mapped file.

The second application is called the QEvpClient. Its purpose is to read the histograms provided by the evpServer from the memory mapped file and display them. While in the whole of STAR only one evpServer is running, multiple instances of QEvpClient are typically running.

The histogramming is done via the ROOT analysis package (http://root.cern.ch) which I have installed as version 5.08/00b. The letter Q signifies the use of Trolltech’s Qt (http://www.trolltech.com/) in the QEvpClient's gui. Qt is extended via the qtRoot (http://www.trolltech.com/) package provided by Valeri Fine.

See also:
Online Histogram Procedures

Installation and Testing

Source Code

Comparing to what you may have been accustom to from previous years (before 2009), the sources have moved and are currently located in cvs under OnlTools/OnlinePlots/.

For development, you should be in the 'dev' environment to build the online plots (run stardev).

Installation

  1. check out the latest version
    • cvs co OnlTools/OnlinePlots
  2. build
    • ./OnlTools/OnlinePlots/Scripts/CompileOnlinePlots.csh
  3. edit runtime script
    • vim EvpPlotServer.C

You may also need to edit some of the variables in the .rootrc file CompileOnlinePlots.csh generates (the default values may work). For detailed information about the arguments in EvpPlotServer.C, see Expert Modes.

For normal testing against an old run, in EvpPlotServer.C just edit the line after the "-path" option in argsS[] to specify either a run number in /a/ (when on the evp.starp machine) or a daq file (commonly found under /star/data03 on RCAS).  Then uncomment the evpServerMain::main line.

You may want to make a copy of this file for running the presenter (a common practice is to make one copy named server.C, and one copy named presenter.C).  Just make sure that evpServerMain::main is commented out and evpMainPresenter::main is in.

Starting the server should be done with

 root4starN -b -q EvpPlotServer.C

If developing code, recompilation is achieved simply by using cons.

Disclaimer

Know what you are doing.
Especially, if you use the evpops account on the evp.starp machine. You might disturb experiment operations.

 

 

Expert Modes

After successful installation, all scripts are located in the ~operator/pplots/ directory. The following expert-only options exist:

Server

The evpServer can be started with the following options by editing EvpPlotServer.C:

-nogui no graphical user interface will be opened
-disabled if GUI is present, user interaction is disabled
-live immediately switch to live mode after start-up
-start start requesting data immediately after start-up
-end launch end-of-run-action, when end of run is detected (prints all active panels into a pdf file and saves to data base)
-path <pathname> switch off live mode and read data from <pathname>
-map <mapfilename> do not write to default memory mapped file, write to <mapfilename>
-nocheck do not check for other running instances of evpServer
-nevents <number> process only <number> events for each run
-quit quit at the end of the run
-h print list off command line options

 

Presenter

When the Presenter is started without any additional arguments, the live mode will read from the default memory mapped file.
Edit EvpPlotServer.C (and uncomment the presenter option) and specify a pathname to the memory map file, e.g. test.map, which will be used instead of the default memory mapped file name.

Example: If you are developing code and want to run the Server and Presenter without interfering with the experiment monitoring, you would specify a map file for the server and have the presenter read that map file.

 

Histogram Groups

A histogram group, as specified in the base class HistogramGroup, consists off a set of histogram that usually will be printed onto one canvas.

A class of type HistogramGroup is responsible for creating, deleting, filling, resetting, and drawing of its histograms. It also holds information about the detector groups and event groups for which the group should be active two unsigned int (Jeff L ).

When processing event from the event pool, the online monitoring is examining every histogram group whether its detector and event group match current event’s respective groups. Only if a match if found, the HistogramGroup is filled. The fill function is passed pointers to the evpReader and the to data: virtual bool fill(evpReader*, char* datap)

With every new run, a HistogramGroups reset() function is called.

HistogramGroup.h:
#ifndef HistogramGroup_h
#define HistogramGroup_h
#include "TH1.h"
#include "TH2.h"
#include "TH3.h"
#include "TCanvas.h"
#include "TString.h"

class evpReader;
class TMapFile;

#include <map>
#include <set>
#include <list>
#include <vector>
#include <string>

using namespace std;

#include "Rtypes.h"

class HistogramGroup : public TObject {

public:
HistogramGroup(const char* group, const char* subGroup, const char* trigger="any", const char* detector="any");
// HistogramGroup(const HistogramGroup&);
virtual ~HistogramGroup() { }

virtual bool fill(evpReader* evp, char* datap) { return true; }
virtual void draw(TCanvas* cc);
virtual void reset() {}

public:

const char* groupName() const { return mGroupName.Data(); }
const char* subGroupName() const { return mSubGroupName.Data(); }
const char* triggerName() const { return mTriggerName.Data(); }
const char* detectorName() const { return mDetectorName.Data(); }
const char* id() const { return mId.Data(); }
bool operator<(const HistogramGroup& hg) const;

protected:
TString mGroupName;
TString mSubGroupName;
TString mTriggerName;
TString mDetectorName;
TString mId;

public:
bool testBits(unsigned int trigger, unsigned int detector);
void setActive(bool b) { mActive = b; }
bool active() { return mActive; }

protected:
char* pre(const char* a);
void pre(vector<TH1*>& );
unsigned int mTriggerBits;
unsigned int mDetectorBits;
bool mActive;
ClassDef(HistogramGroup,1) ;
};

#endif

Online QA Rate Limitations

Some interest has been expressed to obtain increased statistics for at least some quantities shown in the Online QA plots. We will first review what is currently available, and then discuss approaches for change.

CURRENT SET-UP

  • The DAQ group makes events available in the event pool. These events typically arrive at a rate of nearly 10 Hz, and are controlled to be specific (generally minbias-like, full-detector) triggers.
  • The Online QA program evpServer digests this data by grabbing the most recent event in the pool whenever it finishes with an event (pausing if nothing newer than the just-processed event is available). Processing minbias-like pp events on the current 2.4 GHz dual-CPU EVP machine (see Appendix for specs) takes approximately 1 second (1 Hz), but the code is not written and compiled to be parallelized.

POSSIBILITIES

  1. Data Selectivity:
    • A different mix of events can be fed into the event pool. For example, events with only fast detectors can be processed at a rate of about ? Hz by evpServer (still under investigation). This would be a very easy way to increase statistics for fast detectors, though the increase would still be rather mild.
    • Online QA could selectively process only portions of each event for a large fraction of the incoming events. Again, not too much effort, and mildy helpful for a few detectors.
  2. Parallelism:
    • The evpServer code could be re-written and compiled for multi-threading/parallel-processing. This would be a major effort (rough estimate of 2-3 FTE-months, depending on the developer's familiarity with the topics), but would provide access to the multiple CPUs already available on the EVP machine (though evpServer is not the only process running on this machine and it would need to compete for resources). This would be able to benefit all detectors with rate increases of perhaps x2 with the current EVP machine.
    • Multiple instances of evpServer could run either on the same EVP machine, or even on multiple machines. This is likely easier to implement than the multi-threading/parallel-processing approach withing a single program.
  3. Hardware:
    • New hardware is becoming available with 16 or more cores. Development along the line of parallelism on a single machine could lead to an order of magnitude increase in evpServer processing rates.
    • Running evpServer on multiple machines obviously implies additional computers would need to be bought.
    • There is very little room currently for hardware with increases CPU speeds (e.g. 3 GHz). Simply replacing the current EVP machine with a dual 3 GHz CPU machine would likely yield only marginal gains (much less than x2) in processing rates. Additional hardware only helps if parallelism is developed.
  4. Event Pool Filling Rates:
    • Unless the above-described techniques of data selectivity and/or parallelism are employed, the current event pool filling rate is not a limiting factor.
    • Additionally, the DAQ group believes that the current setup can handle something like 50-75 Hz of full-detector events through merely reconfiguration (almost no additional work). It is unlikely that such a rate would be limiting on a single machine without combining data selectivity and parallelism techniques together.

Appendix: Specs of current EVP machine

2 x 2.4GHz Xeons with Hyperthreading
2GB of memory (perhaps PC2400)
~1+ TB of disk (6x250GB disks in a RAID 5 array)
4 NICs, 2 presently in use (Gb/sec from DAQ, 100MB/sec over "starp")

Starting the Online Plots

Locations & account

The Online Plots are run from evp.starp.bnl.gov, typically under the 'evpops' account.  In the Control Room, the computer named rts01 is the "standard" location to have them displayed.  The standard scripts should be stored in ~evpops/pplots/ and ~evpops.

Running

Running the presenter

First check The OnlinePlots icon on rts01 in the Control Room looks like this: (if you ave a better png image for this purpose, send it to Wayne Betts). It launches a small script to start the Presenter (you will need the evpops account password).

Using the icon effectively does the following steps for you (which you can do manually from any node):

  1. Log into evp.starp.bnl.gov as 'evpops'. Ask your shift leader for the password to do so.
    • ssh evp.starp.bnl.gov -l evpops
  2. Change to the ~evpops/pplots/ directory
    • cd ~/pplots/
  3. Start the histogram presenter
    • root4starN livePresenter.C

Or simply run the LaunchPresenter.csh script in ~evpops. Due to an as-yet unsolved bug, it may take several tries to get the presenter running.

 

Starting the Server (only necessary if a Server is not already running on evp, which should be an unusual event)

  1. Log into evp.starp.bnl.gov as user 'evpops'. Ask your shift leader for the password to do so.
    • ssh evp.starp.bnl.gov -l evpops
  2. Make sure the server is not already running. No harm should be caused if you attempt to run multiple servers at the time, other than log files being filled up.
    • ps x | grep liveServer.C
    • kill <process number>
  3. Check if ServerLoop is running.  If it is the server should restart automatically.  Otherwise you need to start it.
    • ps x | grep ServerLoop
  4. Start ServerLoop (so that it will continue even if you log out)
    • nohup ./ServerLoop >&! log.ServerLoop &
  5. Check the output of in the log file
    • tail -f pplots/server.out

 

If the server fails to start it may be necessary to recompile it and make sure a new memory map file gets used:

  • cd pplots
  • rm evpEventServer.map
  • cons

Then try the above steps again.

 

For more details about the available options, see Expert Modes