Minutes 

Attendees: Dmitry A., Wayne B., Leve H., Jérôme L., Gene V.B., Jan B., Maxim N.

Online webserver:

• Wayne installed: SSH key client, osiris
• ITD backups nominally working
  • Warning message noted to ITD
  • Restore from backup (targeting machine with warning messages) by Wayne went fine
• Synchronization with secondary new webserver done manually, not yet automatic
• Tomcat restarted for Leve's scheduler statistics redeployment
  • Jerome's service idea should be copied from old server so that intentional downtimes don't trip the watcher

ShiftLog:

• Detector checklist being posted to the ShiftLog regularly now. A suggestion from Jan to make it easier was to make a shortcut to the checklist document from the ShiftLeader's desktop (post-meeting Leve learned that each new checklist is given a new document name; finding the document isn't that difficult anyhow)

Online user support:

• Enabling migration off sc.starp and onto linux pool
  
  • Dmitry notes that Corba libs are present on linux pool
  • Jan says Brian Page is in the midst of trying to move two components:
    • cdev reader (needs Corba libs, will try compiling)
    • DB writer (Gene will provide Brian with a working example code)
• Two online disks (noted by Wayne in previous meetings): replaced
• Some additional network issues this week
  • Initially fixed with a switch replacement
  • ...but addition failures occurred
  • Wayne will try to fix during access later today (post-meeting update: this went successfully; topic closed)
• starutilities replacement: usage extension currently with "soft" limit
  
  • Matt could not install C-AD software in XP mode on new box
  • next possibility is dedicated XP machine (needs a software extension from ITD)
• Other tasks (noted in previous meetings) in queue

Databases:

• Shift sign-up un-subscriptions now exclude trainees [Dmitry]
• Feedback to slow controls to implement monitoring of their automated services (are they running? is data flowing? for now, only Dmitry's dbPlots and an environment monitor)
• TPC anode voltage migration was not running, but Dmitry will turn it on (we are aware of its shortcomings)
• Dmitry noted RHIC clock seems to be variable even during fills (Gene will follow up)
• Some discussion of when DB snapshots run; Jan seems OK with packaging at midnight for 4am usage for cloud jobs
• Dmitry notes that there's an open discussion about recorded prescales being wrong. If this is corrected later in the DB, it shouldn't be a problem for offline users as long as they access from the DB and not from the DAQ files. Needs to be sorted out by trigger group first.

FastOffline:

• Laser files were being skipped; fixed by Jerome
• Automatic drift velocity calibrations started (and working)

QA:

• OnlinePlots had 3 problems spurred by update of starnew to SL11a library
  
  • Macros used (Load.C) needed modified loading order of libraries (fixed)
  • Numerous histograms getting skipped, and others in unusual order in output PDFs (RT 2085)
    • Temporary solution: run in starpro (SL10k) patched for TOF/VPD updates
  • Code for inserting PDFs into database should use 'REPLACE INTO' instead of 'INSERT INTO', but we only have a binary of the current version
    • Dmitry had source code from an old version which Gene edited to work as desired now (update: codes committed to CVS)
• Offline QA histogram filling needed to update code for TPC padrow determination
• AutoCombining of offline histograms stopped working (Elizabeth investigating) (update: fixed for a changed function interface in QA daemon)
• No references yet, but shift crew (Lanny) looking at data
  • Call for setting references will go out when physics readiness is achieved (many problems in current data)

This meeting is the second of this kind. Meetings are being set to discuss and converge as far as the TPC voltage issue is concerned.

Several action items and discussions were started since the first meeting (see especially this thread).

Minutes 

Attendees: Wayne B., Leve H., Jérôme L., Gene V.B.

ShiftLog [Leve]:

• New server:
  • Ready:
    • App copied to new server (Wayne)
      • Successfully passed stability tests (pounding on it)
    • All WARs deployed
    • Scripts ready to flush cache and re-copy WARs
  • To do:
    • Will re-check links to and from RunLog
    • Watcher script not on yet
    • Verify Offline QA Shift Report connections

Online infrastructure [Wayne]:

• New web server:
  • Links on homepage checked
    • Some issues with DB access and privilege from current node name (dean2)
  • New jpgraph version installed in same location as the old one
  • Disk exports are not in place, so content isn't being updated currently
  • Will do another sync over the weekend or early next week, in addition to the final one when we swap servers
  • Ganglia to be installed
  • User accounts to be added
• New web server backup:
  • Configuration kept the same as main new web server
  • ...but content isn't sync-ed
• Password rotation completed
• mysql-devel installed on online linux nodes (RT 2068)
• Two online disk failures (on onl09 (non-critical) RT 2065 and on onlldap (critical) RT 2067)
  • RAID still providing for the file systems in the interim
• ITD network backups failing (RT 2069), but resolution from ITD may have just come through
• Machine replacements:
  • emcsc ready (needs either deployed by tomorrow or ITD block needs postponed)
  • emcspin ready (Monday)
  • starutilities awaiting C-AD software

FastOffline [Jerome]:

• Many files coming into the system, but all are tests, pulsers, etc., so not being processed
• Some missing detector setup, beam energy, etc. information: could mean misconfiguration, or potentially not properly propagated data

QA [Gene]:

• OnlinePlots switched back to newer EVP server
  • Switching steps still undocumented
• Pre-combining of FastOffline files still needs tested and configured to be the default

 Minutes

Attendees: Leve H., Dmitry A., Jerome L., Wayne B., Gene V.B.

Databases [Dmitry]:

• All DB collectors running now
  • Only TPC voltages absent due to no data yet
• Slow Controls Archiver not yet running to avoid collecting value-less data
  • With Run start imminent, this needs to get going
  •  Dmitry will follow up with Yury G.
• Potential to benefit from an unused EPICS feature to read multiple channels in one request (we have been reading one channel at a time across the board), which could cut down the overhead in obtaining data
  • Unknown why this wasn't previously used, so testing with caution to learn (perhaps the multiple channel data comes in a burst which could fail for some reason)
• New shift sign-up release is imminent (possibly today)
  • Only minor features getting final adjustments (features have been [node:19902 "previously presented"])
• RunLog working fine: new runs appearing, but all being marked bad
  • Jerome notes that bad runs won't come through FastOffline, which has been turned on already
• New DB nodes
  • Tested and ready for use
  • New configuration (relevant for FastOffline use) not yet in place (Jerome and Dmitry will work this out)
• Online DB plots working and logs show usage
  • Only unavailable quantities are those collected from the Slow Controls Archiver

ShiftLog [Leve]:

• New manual is printed out and in place at the counting house
• ShiftLeader desktop computer has all the necessary icons, correctly linked and tested, including making a ShiftLog entry
• New web server not critical (stable operation on old web server)

Online infrastructure [Wayne & Jerome]:

• OS upgrade
  • One critical: a replacement node for usage in monitoring chilled water (old machine is Windows 2000) is awaiting ITD processing.
    • Follow-up with ITD on Monday if nothing transpires by then
  • Other machines in the queue ("bond", "beatrice", "l3disp")
• Online web server replacement
  • Currently replicating old machine on new one: file copying and version checking
    • ShiftLog request to continue with same Tomcat version (a known quantity) instead of upgrading in hopes of improved stability
  • Spin resource request involves storage to be delivered on new server
• Uncertainty in status of UPS services for computers at the experiment
  • Testing today before the Run starts
• Network request for east FMS/FPD racks completed
• Online linux pool has been stable for the past ~3 weeks (after a couple weeks of apparently random reboots)

FastOffline [Jerome]:

• Ready, running, and waiting for data

QA [Gene]:

• OnlinePlots still running on older evp machine (but stable)
  • Will look into moving it back over to the new node
  • Need Jeff L. to flip some switches (and document it)
• Offline QA
  • Finished implementing tools to update just specific histograms in reference set
  • QA Shift set of histograms will be flushed next week and started anew
    • Histograms only go into the set if given a description, and only stay in the set if a reference is set

We are expecting some collider operations imminently, so data will likely flow through the entire system over the next week.

Attendees: Leve H., Dmitry A., Jérôme L., Wayne B., Gene V.B., Jeff L.

Minutes:

Databases [Dmitry]:

• Conditions collectors started (all except beam information related ones)
  • Everything running appears to be OK and status can be checked from web
  • Beam info needs cdev, provided by C-AD once their operations get going for the Run (also includes magnetic field setting; expect data around the time we turn on the STAR magnet)
    • Absence of this data means RunLog does not finalize runs and information is not migrated to Offline, but this is expected to be OK for now for anyone doing online-only tests (requested to post a note to rts-hn about this)
  • No migration issues expected, but TPC anode voltage recording practices need to be ironed out in the context of the Run 11 data (discuss with Maxim)
• Online DBs flushed
  • As expected, a few failed links and missing web pages were noted (by users too) and fixed
• Once cdev is running, will be ready for turn on / testing of FastOffline

ShiftLog [Leve]:

• As expected, excluded from and not affected by DB flush
• Manual updated for instructions on changing run status (may receive further review & changes from Jerome)

Online nodes [Wayne]:

• AFS has been stable on the newer evp machine since last Friday
  • Jeff's been using the machine (and AFS) without incident
  • Will likely switch OnlinePlots back to this machine (from evp2, the older machine) next week
    • Documentation requested for switching between these machines
• Replacement of dean at the stage of OS installation on new hardware
  • Timeline for replacement should be well before Run starts, as disk space on the new machine has been requested for online use
• Trigger commissioning is a high priority part of the request for online node resources (Wayne will contact Pibero directly)
• Nothing to report on OS upgrades

Online QA - jevp demo [Jeff]:

• Code
  • Subsystems have an xxxBuilder class (e.g. tpcBuilder, eemcBuilder, etc.) which inherits from a jevp base class for plots
  • Plot objects can have multiple histograms added to them, as well as other graphical objects like lines and circles (as long as they are of some "component" type?)
• Configuration
  • Editor allows hierarchical arrangement of plots into tabs and sub-tabs, under larger Sets (e.g. Shift, 2009, ESMD)
    • Features drag & drop for re-arrangement
  • Tabs can have collective properties for all plots underneath (such as setting all to have the same maximum)
  • Plots can have individual properties (such as log y scale)
• Running
  • One server starts up several builders as separate processes
  • Data is sent to the builders from a file or an event pool
  • Presenter runs separately to display plots
    • Similar basic look & feel to current OnlinePlots presenter
    • Clicking on a plot brings up a reference side-by-side
      • Drag & drop to add new reference, or select a different reference from panel of choices
      • Suggested: tag references with run from which they're made, and timestamp (already includes a comment)
      • Deleting references is simple, but not so simple as to be done accidentally
      • Mild concerns to be wary of reference bloat and organization as things evolve at the experiment (including collision species/energies change)
  • PDFs generated and uploaded to RunLog DB for each Set
    • Include table of contents
    • Concerns of space this will consume on the onldb server and redundancy of plots
      • Run 10 PDFs used more than half the available disk space, and it is reasonable to expect a doubling (or more) of the required space for Run 11
      • Wayne & Dmitry will assess the current storage on onldb [post-meeting statement that space may be fine; follow up next week]
      • Requirements still need to be more accurately defined
        • Will depend on what gets axed/kept from review by subsystems & trigger board
        • Would be helped by real data, but that's a bit late

Next week: demo of Offline QA for AutoQA with reference histograms [Gene]