Attendees: Leve H., Dmitry A., Jérôme L., Wayne B., Gene V.B., Jeff L.

Minutes:

Databases [Dmitry]:

• Conditions collectors started (all except beam information related ones)
  • Everything running appears to be OK and status can be checked from web
  • Beam info needs cdev, provided by C-AD once their operations get going for the Run (also includes magnetic field setting; expect data around the time we turn on the STAR magnet)
    • Absence of this data means RunLog does not finalize runs and information is not migrated to Offline, but this is expected to be OK for now for anyone doing online-only tests (requested to post a note to rts-hn about this)
  • No migration issues expected, but TPC anode voltage recording practices need to be ironed out in the context of the Run 11 data (discuss with Maxim)
• Online DBs flushed
  • As expected, a few failed links and missing web pages were noted (by users too) and fixed
• Once cdev is running, will be ready for turn on / testing of FastOffline

ShiftLog [Leve]:

• As expected, excluded from and not affected by DB flush
• Manual updated for instructions on changing run status (may receive further review & changes from Jerome)

Online nodes [Wayne]:

• AFS has been stable on the newer evp machine since last Friday
  • Jeff's been using the machine (and AFS) without incident
  • Will likely switch OnlinePlots back to this machine (from evp2, the older machine) next week
    • Documentation requested for switching between these machines
• Replacement of dean at the stage of OS installation on new hardware
  • Timeline for replacement should be well before Run starts, as disk space on the new machine has been requested for online use
• Trigger commissioning is a high priority part of the request for online node resources (Wayne will contact Pibero directly)
• Nothing to report on OS upgrades

Online QA - jevp demo [Jeff]:

• Code
  • Subsystems have an xxxBuilder class (e.g. tpcBuilder, eemcBuilder, etc.) which inherits from a jevp base class for plots
  • Plot objects can have multiple histograms added to them, as well as other graphical objects like lines and circles (as long as they are of some "component" type?)
• Configuration
  • Editor allows hierarchical arrangement of plots into tabs and sub-tabs, under larger Sets (e.g. Shift, 2009, ESMD)
    • Features drag & drop for re-arrangement
  • Tabs can have collective properties for all plots underneath (such as setting all to have the same maximum)
  • Plots can have individual properties (such as log y scale)
• Running
  • One server starts up several builders as separate processes
  • Data is sent to the builders from a file or an event pool
  • Presenter runs separately to display plots
    • Similar basic look & feel to current OnlinePlots presenter
    • Clicking on a plot brings up a reference side-by-side
      • Drag & drop to add new reference, or select a different reference from panel of choices
      • Suggested: tag references with run from which they're made, and timestamp (already includes a comment)
      • Deleting references is simple, but not so simple as to be done accidentally
      • Mild concerns to be wary of reference bloat and organization as things evolve at the experiment (including collision species/energies change)
  • PDFs generated and uploaded to RunLog DB for each Set
    • Include table of contents
    • Concerns of space this will consume on the onldb server and redundancy of plots
      • Run 10 PDFs used more than half the available disk space, and it is reasonable to expect a doubling (or more) of the required space for Run 11
      • Wayne & Dmitry will assess the current storage on onldb [post-meeting statement that space may be fine; follow up next week]
      • Requirements still need to be more accurately defined
        • Will depend on what gets axed/kept from review by subsystems & trigger board
        • Would be helped by real data, but that's a bit late

Next week: demo of Offline QA for AutoQA with reference histograms [Gene]

Minutes:

Attendees:Dmitry A., Leve H., Matt A., Wayne B., Jeff L., Jérôme L., G. Van Buren

Databases [Dmitry]:

• Online backup now using the new NAS system
  • Daily backup of all three ports with a retention time of 7 days
  • Currently have 2+ TB of space, which is probably more than enough for even 14 days retention
  • Potential problem with permissions due to NFS mount and different user IDs on different systems, but not a problem presently
  • Email alerts of problems from NAS goes to Wayne & Dmitry
• Flush of online DBs not yet done
  • Reasoning is that still in testing at STAR and this can add up to significant amount of data
  • ...but we're not sure which test data people will want to keep associated to Run 11
  • Decision made to go forward with the flush and not continue waiting for testing to get further along
  • NB: ShiftLog (and some other) DBs and tables are skipped in the flush; ShiftLog is already recording for Run 11
• Shift Signup GUI has been re-written
  • Demo shown
  • Some details of new features still need implementation (working with Jérôme)
  • All old features are in place; could replace old codes at any time (pending bug checks)
  • Deployment schedule not fixed by any deadlines
• Isolated nodes for FastOffline in Run 11
  • Not in place, but Dmitry will write up a config file for this

ShiftLog [Leve]:

• Nothing until new web server
  • No progress on new webserver

Online nodes [Wayne & Matt]:

• OS upgrades:
  
  • FTPC and PMDsc done/replaced
  • To be done: Bond, EMCsc (coordinate with users), STARUtilities (coordinate with C-AD), EMC01, Beatrice, L3display (Jeff notes the need for QT4 on this node for display programs)
• FUSE now working on all linux pool nodes
• gcc standardized on all linux pool nodes
• Recent rise in instability of the linux pool nodes: several have halted and/or needed rebooted in the past two weeks
  • Previous solution of disabling USB controller not helpful for this (that solution is still in effect)
  • No obvious environmental changes, but seems likely given the pattern
  • These nodes are ~5.5 years old (hard to believe they would show age problems within a couple weeks of each other)
  • Similar nodes are in use for offline DBs and not showing problems (located in BCF)
• Newer EVP machine experiencing AFS issues
  • Access given to John McCarthy to help diagnose
  • OnlinePlots will be switched to use old EVP machine for the time being (Gene & Jeff will arrange)

Attendees: Jeff L., Jérôme L., Wayne B., Gene V.B., Leve H., Dmitry A.

Minutes:

jevp : the new online QA package

• Deployment intended before run starts
  • Will run concurrently with OnlinePlots package on the evp.starp machine
    • That machine is believed, at this point, to be sufficiently up to the task
• Infrastructure ready at 90% (old adage: last 10% takes 90% of the time)
  • Using QtRoot
• New architecture has 3 components
  • server: repository for plots
  • builder: stand-alone process for reading event pool, filling hists, and ships plots of them to the server in a ROOT container class (StJEvpPlotSet)
    • StJevpPlotSet contains the histograms, plus lines and texts, and axis scales
  • display: it asks the server for the plots it wants to present at any given time (and tabs); not the content of other plots
• All components communicate over ethernet and can be on separate machines
  • Multiple builder instances will all read the same files simultaneously
  • Suggestion to possibly leverage other available online nodes
    • Would need to mount evp.starp: /a
    • Won't pursue until a need is shown
• Deployed all existing plots to server (999 exactly!)
  • Divided into 17 builders: generic, daq, hlt, l3, each subsystem
• Main features:
  • Easily configured plotting
    • There is an xml configuration file, plus an editor for managing the xml file
    • Hierarchy of tabs and plots easily changed (directory-like arrangement)
    • Top level is shift, HLT, other expert sets
    • Same plot can be in more than one place
    • Control over rows and columns, log scaling (part of plot definition, but over-ridable in the configuration), plot maxima and minima (setting can span multiple plots)
    • Display comes up with shift histograms by default
  • Reference histogram capability
    • Clicking on a hist brings up the data and reference histograms (and possible old references)
    • This page allows one to set the data as the reference along with a comment
    • Discussion of expanding this capability for automated analysis and alerts
      • Analytic comparisons be done in the builder (that's where the analysis happens)
      • Suggestions to keep this in mind as programming continues
• Future Planned Features:
  • Conditional suppression of unwanted plots
    • Distinction between not-in-run, and broken
    • Some plots shown only if some state exists
  • Fast trigger stream: possible send all trigger data (not DAQ data) to the event pool
  • Monitors from within the package (e.g. how long does each builder take)
    • Needs to be coordinated in the server
• Additional To-Do tasks:
  • Run stop/start isn't well-coordinated yet (when to clear histograms for start of a new run; codes need more understanding of state)
  • Basic shift histogram group needs to be set-up
    • Plan: present a set of plots to the trigger board for feedback & input
  • Need testing with real data for each subsystem for possible code-copying mistakes
  • Need extended operation testing (e.g. does it have memory leaks? eventually crashes?)
    • Already some concerns about ownership of objects in the plotting code (an issue we had with the OnlinePlots package)
  • Plan to run display component locally on rts01
    • Needs STAR environment to do so
    • Interest in a second display screen
    • Interest in upgrading rts01 hardware
      • Wayne agreed to spec out new hardware
• Plan to learn as we go (from problems during the Run) of what are the optimal ways to present data which make the problem(s) evident
• Plan for now is to demo the jevp package in 3 weeks time

Other discussion topics:

• Rotations of DBs done for new year, but not all hyperlinks have been rotated
• Online disk space usage needs: users need to be polled (e.g. spin QA programs and data), and who the users are needs to be determined
• Solaris phase out: systems (slow controls and trigger) are not in S&C control; admins need to be convinced that support is gone and catastrophes loom (e.g. ITD can disconnect machines from the network)
• Switch to using dbbak for daily DB backups not yet done
• DB flush planned for Monday, and that should be it until online daemons start up (awaits EPICS ramp up)
• Jérôme mentioned workflow tests are in order for QA-RunLog-ShiftLog interconnections
• Discussion of RTS/ShiftLeader status editing/updating (continued post-meeting):
  • Conclusion seemed to be that the current set-up is what is needed, but that edits/updates should be in the hands of shift leaders. This policy should be made very clear and known.