Minutes 

Attendees: Dmitry A., Wayne B., Leve H., Jérôme L., Gene V.B., Jan B., Maxim N.

Online webserver:

• Wayne installed: SSH key client, osiris
• ITD backups nominally working
  • Warning message noted to ITD
  • Restore from backup (targeting machine with warning messages) by Wayne went fine
• Synchronization with secondary new webserver done manually, not yet automatic
• Tomcat restarted for Leve's scheduler statistics redeployment
  • Jerome's service idea should be copied from old server so that intentional downtimes don't trip the watcher

ShiftLog:

• Detector checklist being posted to the ShiftLog regularly now. A suggestion from Jan to make it easier was to make a shortcut to the checklist document from the ShiftLeader's desktop (post-meeting Leve learned that each new checklist is given a new document name; finding the document isn't that difficult anyhow)

Online user support:

• Enabling migration off sc.starp and onto linux pool
  
  • Dmitry notes that Corba libs are present on linux pool
  • Jan says Brian Page is in the midst of trying to move two components:
    • cdev reader (needs Corba libs, will try compiling)
    • DB writer (Gene will provide Brian with a working example code)
• Two online disks (noted by Wayne in previous meetings): replaced
• Some additional network issues this week
  • Initially fixed with a switch replacement
  • ...but addition failures occurred
  • Wayne will try to fix during access later today (post-meeting update: this went successfully; topic closed)
• starutilities replacement: usage extension currently with "soft" limit
  
  • Matt could not install C-AD software in XP mode on new box
  • next possibility is dedicated XP machine (needs a software extension from ITD)
• Other tasks (noted in previous meetings) in queue

Databases:

• Shift sign-up un-subscriptions now exclude trainees [Dmitry]
• Feedback to slow controls to implement monitoring of their automated services (are they running? is data flowing? for now, only Dmitry's dbPlots and an environment monitor)
• TPC anode voltage migration was not running, but Dmitry will turn it on (we are aware of its shortcomings)
• Dmitry noted RHIC clock seems to be variable even during fills (Gene will follow up)
• Some discussion of when DB snapshots run; Jan seems OK with packaging at midnight for 4am usage for cloud jobs
• Dmitry notes that there's an open discussion about recorded prescales being wrong. If this is corrected later in the DB, it shouldn't be a problem for offline users as long as they access from the DB and not from the DAQ files. Needs to be sorted out by trigger group first.

FastOffline:

• Laser files were being skipped; fixed by Jerome
• Automatic drift velocity calibrations started (and working)

QA:

• OnlinePlots had 3 problems spurred by update of starnew to SL11a library
  
  • Macros used (Load.C) needed modified loading order of libraries (fixed)
  • Numerous histograms getting skipped, and others in unusual order in output PDFs (RT 2085)
    • Temporary solution: run in starpro (SL10k) patched for TOF/VPD updates
  • Code for inserting PDFs into database should use 'REPLACE INTO' instead of 'INSERT INTO', but we only have a binary of the current version
    • Dmitry had source code from an old version which Gene edited to work as desired now (update: codes committed to CVS)
• Offline QA histogram filling needed to update code for TPC padrow determination
• AutoCombining of offline histograms stopped working (Elizabeth investigating) (update: fixed for a changed function interface in QA daemon)
• No references yet, but shift crew (Lanny) looking at data
  • Call for setting references will go out when physics readiness is achieved (many problems in current data)

Minutes 

Attendees: Dmitry A., Wayne B., Leve H., Jérôme L., Gene V.B.

ShiftLog [Leve]:

• Checked that links work both ways with RunLog
• Re-deployed to fix an issue with a function used to tell if we're in a shift-taking period
  • Dmitry suggested to modify this next year to not use dates

Online infrastructure [Wayne]:

• New webserver:
  • Database access now working fine
  • Watcher script needs turned on
  • cron jobs (and associated files) need to be copied
  • Announcement for a Monday swap:
    • 2pm (Wayne, Jerome, Leve, Dmitry should be available to handle whatever may arise - Dmitry already knows one configuration he needs to change during the swap)
    • NFS will be turned off on the old server, sync will be done, IPs swapped, and NFS on.
• Two online disks (noted last week) still need replacement
• Event Display
  • Not currently functional (we lost our maintainer; Jerome is trying to prop it back up for now)
  • Pieces in multiple locations for online display plus offline (images dumped for web display)
  • Not critical (old online event display still functional), but valuable for PR
• Long-standing request (not critical) to be able to read from HPSS online (via data carousel)
  • Jerome looking for any road blocks (e.g. firewall)
• Machine replacements:
  • emcsc done
  • starutilities and ftpcgas not yet done (scheduled for work with relevant parties early next week)
• All Windows nodes rebooted with security patches
• rts01 replacement request merged into a larger purchase request (RT 2070)
• Request for help restoring old FTPC event builders (RT 2066)
  • Possible solution of other hardware if this is too much effort
• FMS/Spin request for replacement machines (RT 2073 and 2074). Not critical, but latter is important for remote access (current machine is inside WAH and not on network)

Databases [Dmitry]:

• After new nodes go into production pool, 2 old nodes will be targeted for FastOffline + nightly tests
  
  • Assessment of new node speed/power still ongoing
• onldb1 and 2 were up for 1+ years, so rebooted on purpose to check state
  • Collector daemons did not automatically start, but otherwise OK
  • Dmitry suggested leaving it this way since reboot need is so rare and automatic can take a while to get caught up (has to look through and skip all runs from the year)
  • Gene suggested making it automatic in case Dmitry's not around to restart daemons (though shift leaders can potentially do it themselves). We'll go with this.
• QA files (OnlinePlots + Jevp) stored for RunLog can now exceed 4 GB without needing manual rollover
• TPC Anode Voltges:
  
  • For now, continue with methods of collection from last year
  • Dmitry tested a supposedly-optimized "bulk reading" methods and found them to be no better than channel-by-channel (actually worse as it appears to be the same + overhead: 13-15 seconds to collect all values versus less than 1 second); dead end.
  • Possible new methods of collection under development (Tonko & the TPC group)
  • Feedback to the TPC group that wider "dead band zones" could reduce traffic and help alleviate problems due to load

FastOffline [Jerome]:

• Same as last week: many files coming into the system, but all are tests, pulsers, etc., so not being processed
• Gene notes some possible tweaks to the chain may be needed for TPC calibration status (awaiting TPC feedback)
• Dmitry need to verify with Lidia some exported info from from onldb1

QA [Gene]:

• Offline QA documentation updated (Lanny & Gene)
• Jerome notes the need to monitor and document and issues with QA Shift
• Pre-combining of FastOffline files still needs tested and configured to be the default