2008 - EMC calibration workshop

The EMC calibration workshop was launched as with a start up Email to starmail on 9/4/2008 10:31. A copy resides below. The review report was requested to be delivered by October the 29th 2008.

See also for further references:

• EMC Calibration Workshop agenda

Dear STAR collaborators,

	Driven by findings and (constructive) criticisms of
the past operation workshop on the core team's attention to EMC
Physics, I have flagged the calorimetric physics issues as high
priority for the core team to address and  a key project for
the year.

	As corrective action, I proposed and discussed internally
to the S&C project (including the software coordinators) the
need for a workshop which high-level goal would be to define,
discuss, map the Physics goals and deliverables of our research
program to EMC tasks timelines, milestones and effort levels
needed to reach those goals (by physics topics, analysis). To
answer the requirements needs, and thanks to Bedanga for being
on board with this, specific questions will be asked of the PWG
(via the PWGC). Questions will serve as direct input to the
workshop.


---
	The workshop will take place at BNL on September
29 and 30th (just after the analysis meeting) in the ITD seminar
room (Bldg 515) and I have asked (and charged) Gene Van Buren,
calibration coordinator, to help with the logistic and
post-workshop report.



---
	As a product of the workshop, I have asked Gene Van Buren,
in consultation with the EMC sub-system management (Will Jacobs
as project coordinator and Matthew Walker as software coordinator)
to steer an effort to write a summary report which would emphasize
the roadmap ahead, and the (human) resources available/needed as
well as what we would not be able to accomplish shall we have
missing workforce. Such document would then serve as a clear
statement for an incoming operation workshop where the needs
would be re-iterated, quantification in our hands. The report
would be due by October 17th, well in time for the operation
workshop in November.

	Many thanks to Will Jacobs for his understanding, Matthew
Walker for his prompt response and assistance and Gene Van Buren
for pulling the troops and generating interest as well as taking
this task on the S&C global plan to the level of seriousness it
deserves.

	With hopes you will attend, participate in providing
feedback (through your PWG) and make this workshop a success.
	

-- 
                 Dr. Jerome LAURET
                 RHIC/STAR Software and Computing project Leader
      ,,,,,      Physics Department, Brookhaven National Laboratory
     ( o o )     Bldg 510a, Upton, NY 11973
 ---m---U---m---------------------------------------------
 E-mail: jlauret@bnl.gov

2008 - TOF software review

The Time Of Flight sub-system was called for a software readiness and integration review in the Fall of 2008. The commitee's charges are available below in the related documents section.

Related documents follows:

• Commitee and TOF group review charges
• PSN0467 : TOF software review report, Fall 2008
   
• Agenda for the You do not have access to view this node
• TOF documentation

2011 - Sti, CA and Stv tracking component review

This page will keep information related to the 2011 tracking component review. The review will cover the state of the Cellular Automaton (CA) seed finding component as well as the Virtual Monte-Carlo based tracker (Stv) and their relevance to STAR's future need in terms of tracking capabilities.

• Project goals
• Why a review?
• Review charges
• Review committee
• Material

Project goals

After a successful review of the ITTF/Sti tracker in 2004, the STAR collaboration have approved the move to the new framework bringing at the time unprecedented new capabilities to the experiment and physics analysis.Sti allowed the STAR reconstruction approach to integrate to its tracking other detector sub-systems by providing method to integrate simple geometry models and allow to extrapolate track to the non-TPC detector planes therefore, correlating information across detector sub-systems. In 2005, the STAR production switched to a Sti based production and we have run in this mode ever since.

However, careful architecture considerations revealed a few areas where improvements seemed needed. Those are:

• The need to maintain two different geometry models (one for reconstruction, one for simulation) increasing workforce load at a time when STAR is both active and ambitious in its future program as well as running thin on detector sub-system code developer. Beyond workforce considerations
  • The two separate geometries have consequences on embedding and simulation hence, our ability to bring efficiency corrections to the next level of accuracy.
  • Material budgets were found to be ill-accounted in reconstruction (dead-material were not properly modeled in the Sti framework). The use of a common geometry model would have removed this issue
• Sti has some tracking restrictions - geometries made of planes and volumes perpendicular to the beam cannot be treated due to a technical choice (detector elements are organized in planes // to the beam, sub-systems assumed to be composed of elements replicated in Phi). This would preclude tracking in detectors such as the FGT.
  • Our goal was to create an extended set of functionalities providing a truly complete integrated tracking approach, allowing the inclusion of hit information from other detectors (a key goal the inclusion of detector hits placed in the forward direction)
• The use Monte-Carlo based propagators would allow better access to Eloss, better predictors and track swimming allowing for tracking in non constant B field (this is also not possible in Sti)

Additional considerations for the future of STAR were

• A single yet flexible geometry model would allow STAR to be ready for GeantX (5 and TGeo based)
• A flexible geometry model would allow STAR to better cope with STAR to eSTAR migration (geometry morphing)
• A revitalize framework would allow addressing long standing issues of event mode in simulation
  • While STAR has a FORtran based approach allowing integration of some event generators, many have appeared pure C++ based, making their integration to the STAR simulation framework difficult. A new generic model would allow a "plug-and-play" approach.
  • The use of non-perfect geometries (miss-aligned) have been lacking in the simulation framework and would be advisable
• Novel algorithm have appeared in the community, leveraging and harvesting the power of multi-core and many-core architectures. Investigating speed and efficient gains and evaluate the best use of resources is necessary for STAR demanding physics program. Equally important, those new algorithm (Cellular Automaton being one) are opening to online tracking algorithm (GPU based).

Based on those considerations, several projects were launch and encouraged

• CA based tracking - the sudy of the CBM/Alice Cellular Automaton algorithm for seed finding was launched in collaboration with our GSI colleagues.  Multi-core aware, the simple algorithm is thought to provide speed gains over the seed finding. Further work could spurse from this evaluation (online HLT) if successful. The algorithm was showed to be portable to STAR, thanks to Yuri Fisyak and Ivan Kisel team, and the product of this evaluation to be tested.
• The VMC project - a three part project (VMC tracking, VMC geometry, VMC simulation framework), the VMC geometry (a.k.a. aGML) has rapidly matured under the care of Jason Webb. The VMC trakcing (a.k.a. Stv) has been developed by Victor Perevoztchikov and thought to provide equal efficiency than Sti (as well as implement all the features listed above).

We propose to review the aGML, CA and Stv components of our framework reshape.

NB: Beyond the scope of this review, a key goal for VMC is to allow the inclusion of newer Geant version and hence, getting ready to step away from Geant3 (barely maintainable), the FORtran baggage (Zebra and portability issues on 64 bits architectures) and remove the need for a special verison of root (root4star) hard-binding root and STAR specific needed runtime non-dynamic libraries.

Why a review?

• All R&D projects are reviewed in STAR
  • Initial approach was to proceed with a "soft" PWG evaluation but (on second thoughts) not really an options …
  • An internal STAR review process should (and will) be established
• Advantages
  • A review process provides strong and independent backing  of the projects
  • A review process provides  an independent set of guidance to management (S&C and PWG) on path forward
  • Collaboration wide scrutiny and endorsement across PWG lessen the risks of  finding problems later
• Reminder: ITTF / Sti was not carried without problems
  • Sti review missed  the UPC PWG’s feedback –problems found a-posteriori diverted attention and workforce in solving it
  • Problem are seen in HBT and fluctuation analysis when Run 4 is compared to Run 10
    • HBT issues were not seen at Sti evaluation – Is it an analysis problem? Something else?
• A review will also provide a good time to re-establish a solid baseline and get feedback from the PWG on opened issues if any
  • This is even more so important that STAR is moving forward to a new set of detectors and high-precision measurements

Review charges

See attachment at the bottom of this page.

Review committee

Status:

• 2011/08/12 Intent of a review brought to management (charges to be written).
                      Action items was to suggest a set of names for the committee set.
• 2011/08/18 Committee members suggestions provided at management meeting. Spokesperson decides he will contact chair.
• 2011/09/02 Charges sent to management for comments along a note that the charges may be long (text is both for committee and reviewee). No feedback outside the provided self-provided note.
• 2011/10/07 Chair contacted - process of selecting committee being worked out (Spokesperson or)
• 2011/10/13 Spokesperson delegate committee forming to review Chair (Olga Evdokimov), S&C Leader (Jerome Lauret) and PAC (Xin Dong)
• 2011/10/15 Committee assembled
• 2011/10/31 Draft agenda made
• 2011/11/01 Agenda presented and feedback requested
• 2011/11/08 Final agenda crystalized

Members:

• Olga Evdokimov (chair)
• Claude Pruneau
• Jim Thomas
• Renee Fatemi                [EVO]
• Aihong Tang
• Thomas Ullrich              [EVO]
• Jan Balewski                 [EVO]
• Anselm Vossen

The agenda is ready and available at You do not have access to view this node.

Material

Below is a list of cross-references to other documents:

• An Hypernews tracking evaluation private list was created to hold information and discussions on the way of convergence toward the review
   
• Documentation
  • Adding a New Detector to STAR
    • AgML Tutorials
    • AgML Example: The Beam Beam Counters
    • AgML vs AgSTAR Comparison

• Meetings
  • You do not have access to view this node
  • You do not have access to view this node
  • You do not have access to view this node
  • You do not have access to view this node
  • You do not have access to view this node
  • You do not have access to view this node
  • You do not have access to view this node
  • You do not have access to view this node
     
• Data and simulation samples, tools, ... (guidance given at You do not have access to view this node)
  • Tools
    • Nightly test stats for the "eval" area
    • Job status for tester's data sample
  • Data selections
    • Location summary
    • Real data   : Summary of tracking testers
    • Simulations: Estimated disk space and time requirements for Sti/Stv/CA evaluation sample
       
• Nightly build (AutoBuild)
  • gcc version 4.3.2 - this has a CVS update and report of differences
  • gcc version 4.5.1 - this follows the previous and compile in place all of what was updated

2021 TPC calibration review

• Committee members:
  • Zhangbu Xu (Chair), Bill Christie, Carl Gagliardi, Frank Geurts, Marian Ivanov, Ivan Kisel, Tonko Ljubicic, Maria Stefaniak, Jim Thomas, Qinghua Xu, Xin Dong (ex officio), Rongrong Ma (ex officio)
• Committee formed on Jan. 26, 2021. Announcement to starmail.
• Review meeting and presentations
  
• Final report and Presentation

2021 TPC calibration review

Content Management system in STAR

Introduction

This project started in 2005 as a service task aimed to provide a seamless port of the online Web server for document self-maintenance and easy access. The initial description follows. It was motivated by the poor maintenance and log term support of the pages available online and the need for quick page creation for keeping help, instructions and procedures up to date in a multiple user and group environment context. Also, we imagined that shift crew could drop comments on existing pages and hoped for the documentation of our operation to be more interactive and iterative with immediate feedback process. Plone was envisioned at the time but the task was opened to an evaluation based on requirements provided below.

Initial service task

This task would include the evaluation and deployment of the a content management system (CMS) on the online Web server. While most CMS uses a virtual file system, its ability to manage web content through a database is of particular interest. Especially, the approach would allow for a Web automatic mirroring and recovery. We propose the task to include

• A smooth port of the main page available at http://online.star.bnl.gov/ to the CMS base system. This may include look and feel, color scheme etc ... at minimum, content. For example, the main tools should appear as left menu for easy navigation as a general web template.
• The evaluation of the database approach should be made and in place before a transition from the old style to a CMS based online web server.
• The CMS system layout should provide accessible branches for detector sub-systems, each sub-system may manage their branch separately. The branches will be most helpful to keep documentation on the diverse subsystem and provide a remote editable mechanism allowing easy management and modification.
• Depending on time and experience, the development could be applied to the offline Web server to some extend. An area where it could benefit best is the tutorial area and the QA area. Within the plone system, users would be able to leave comments, add new tutorials etc ... a manager would then organize or make available to the public (flexibility of access to be discussed).

Project timelines and project facts

Facts:

• We wanted a framework rather than a tool
• There were more than 762 main stream CMS at the start of the project
• "best" of today is not the best tomorrow
• We therefore decided to start the project with a focus on requirements rather than a specific solutions

 Timelines:

Project requirements

The following requirements were set for the project:

Technical requirements

STAR requirements

• Easy page creation
  • Web based editor – accessibility from anywhere
  • WYSIWYG editor, What You See Is What You Get - help novice and advanced users. Feature not a mandate (plain HTML allowed)
  • Assisted Help to layout & design
    • i.e. no special html, xhtml, xml knowledge required
• Support for
  • Attachments, images,  …
  • Auto-update search (i.e. search index and is self maintained withoit the need for external scripts or program)
• Community (popular) tools supported - may trends corresond to real modern world needs
  • Blogs, Comments
  • Polls, Calendar, Conferences, Meetings …
  • ... many more via modules (a-la-perl extension)
• Group management
  • PWG, Sub-systems, Activities, …
• Powerful search (feature rich selector as far as possible)
• Common visual theme for all pages (auto)
• Community support (leverage development from a wider base support)

Functional requirements

The following functional requirements were either requested or desired for a smooth (sup)port of previous deployment.

• Page content must be allowed to be public or require an authentication
• Meetings/agenda should allow for a non-public sections. Public content should remain to the strict minimal "advertisement" level and not reveal information internal to STAR.
  • (comments? summary? section support?)
• Talks provided as attachments should not be public (PWG requested)
• Groups should extent beyond sub-systems and PWG (technical groups, sub-groups)

Related presentation

• You do not have access to view this node

Inner Silicon Tracking

Multi-core CPU era task force

Introduction

On 7/12/2007 23:42, a task force was assembled to evaluate the future of the STAR software
and its evolution in the un-avoidable multi-core era of hardware realities.

The task force was composed of: Claude Pruneau (Chair), Andrew Rose, Jeff Landgraf, Victor Perevozchikov, Adam Kocolosk. The task force was later joined by Alex Wither from the RCF as the local support personnel were interested in this activity.

The charges and background information are attached at the bottom of this page.

The initial Email announcement launching the task force follows:

Startup Email (7/12/2007 23:42)

Date: Thu, 12 Jul 2007 23:42:40 -0400
From: Jerome LAURET <jlauret@bnl.gov>
To: pruneau claude <aa7526@wayne.edu>, Andrew Rose <AARose@lbl.gov>, 
 Jeff Landgraf <jml@bnl.gov>,
 Victor Perevozchikov <perev@bnl.gov>, Adam Kocoloski <kocolosk@MIT.EDU>
Subject: Multi-core CPU era task force

        Dear Claude, Adam, Victor, Jeff and Andrew,

        Thank you once again for volunteering to participate to
serve on a task force aimed to evaluate the future of our software
and work habits in the un-avoidable multi-core era which is upon
us. While I do not want to sound too dire, I believe the emergence
of this new direction in the market has potentials to fundamentally
steer code developers and facility personnel into directions they
would not have otherwise taken.

        The work and feedback you would provide on this task force
would surely be important to the S&C project as depending on
your findings, we may have to change the course of our "single-thread"
software development. Of course, I am thinking of the fundamental
question in my mind: where and how could we make use of threading
if at all possible or are we "fine" as it is and should instead
rely on the developments made in areas such as ROOT libraries.

        In all cases, out of your work, I am seeking either
guidance and recommendation as per possible improvements and/or
project development we would need to start soon to address the
identified issues or at least, a quantification of the "acceptable
loss" based on cost/performance studies. As a side note, I have
also been in discussion with the facility personnel and they may
be interested in participating to this task force (TBC) so, we
may add additional members later.


        To guide this review, I include a background historical
document and initial charges. I would have liked to work more on
the charges (including adding my expectations of this review as
stated in this Email) but I also wanted to get them out of the
door before leaving for the V-days. Would would be great would
be that, during my absence, you start discussing the topic and
upon my return, I would like to discuss with you on whether or
not you have identified key questions which are not in the charges
but need addressing. I would also like by then to identify a chair
for this task force  - the chair would be calling for meetings,
coordinate the discussions and organize the writing of a report
which ultimately, will be the result of this task force.

        Hope this will go well,

        Thank you again for being on board and my apologies for
dropping this and leaving at the same time.


-- 
              ,,,,,
             ( o o )
          --m---U---m--
              Jerome

-

Follow up EMail (8/3/2007 15:34)

Date: Fri, 03 Aug 2007 15:34:56 -0400
From: Jerome LAURET <jlauret@bnl.gov>
CC: pruneau claude <aa7526@wayne.edu>, Andrew Rose <AARose@lbl.gov>, 
 Jeff Landgraf <jml@bnl.gov>,
 Victor Perevozchikov <perev@bnl.gov>, Adam Kocoloski <kocolosk@MIT.EDU>, 
 Alexander Withers <alexw@bnl.gov>
BCC: Tim Hallman <hallman@bnl.gov>
Subject: Multi-core CPU era task force


        Dear all,

        First of all, I would like to mention that I am very pleased
that Claude came forward and offered to be the chair of this task force.
Claude's experience will certainly be an asset in this process. Thank
you.

Second news: after consulting with Micheal Ernst (Facility director
for the RACF) and Tony Chan (Linux group manager) as well as Alex
Withers from the Linux group, I am pleased to mention that Alex
has kindly accepted to serve on this task force. Alex's experience
in the facility planing and work on batch system as well as aspects
of how to make use of the multi-core trends in the parallel nascent
era of virtualization may shade some lights on issues to identify
and bring additional concepts and recommendations as per adapting
our framework and/or software to take best advantage of the multi-core
machines. I further discussed today with Micheal Ernst of the
possibility to have dedicated hardware shall testing be needed for
this task force to complete their work - the answer was positive
(and Alex may help with the communication in that regard).

        Finally, as Claude has mentioned, I would very much like for
this group to converge so a report could be provided by the end of
October at the latest (mid-October best). This time frame is not
arbitrary but is at the beginning of the fiscal year and at the
beginning of the agency solicitations for new ideas. A report by
then would allow shaping development we may possibly need for our
future.


        With all the best for your work,

Background work

The following documents were produced by the task-force members and archived here for historical purpose (and possibly providing a starting point in future).

• First meeting notes - Claude Pruneau
• CPU benchmarking (plenty of memeory context) - Andrew Rose
• Concurrent Computing for C++ Applications

CPU and memeory usage on the the farm - Alex Wither

Opteron (CPU / memory)

Xeon (CPU / memory)

CAS & CRS CPU usage, month and year

Outcome & Summary

A reminder as per the need for a reoprt was sent on 10/3/2007 to the chair (with a side track discussion on other issues which seemed to have taken attention). To accomodate for the busy times, a second reminder was sent on 11/19/2007 with a new due date for the end of november. Sub-sequent reminders were sent on the 12/10/2007 and 1/10/2008.

The task force has not deliverred the report as requested. A summary was sent in an Email as follow:

... a summary of the activities/conclusions of the committee.


... during the first meeting, all participants agreed that if
there was anything to be done, it would be on reconstruction. Members
of the committee felt that GEANT related activities are not in the
perview of STAR and should not be STAR's responsibility.  In view also
of what we did next it also appears that not much would actually be
gained.  We also discussed (1st meeting) the possibility of
multi-treading some aspects of user analysis. e.g. io, and perhaps some
aspects of processing.  Here people argued that there is too much
variability in type of analyses carried by STAR users. And it is not
clear that multi-treading would be in anyway faster - while adding much
complexity to infrastructure - if not to the user code.


Members of the committee thus decided to consider reconstruction processes only.

In subsequent meetings, we realized (based on some references test
conducted in the Industry) that perhaps not much would be gained if a
given node (say 4 cores) can be loaded with 4 or 5 jobs simultaneously
and provided sufficient RAM is available to avoid memory swapping to
disk.

Alex, and Andrew carried some tests. Alex's test were not really
conclusive because of various problems with RCF. Andrew's test however
clearly demonstrated that the wall clock time essentially does not
change if you execute 1 or 4 jobs on a 4-core node. So the effective
throughput of a multicore node scales essentially with the number of
cores. No need for complexity involving multithreading.  Instant
benefits.

Cost:   PDSF and RCF are already committed according to Alex and
Andrew to the purchase of multicore machines. This decision is driven
in part by cost effectiveness and by power requirements. 1 four core
machine consumes less power, and is less expensive than 4 1-core
machine. Additionally, that's where the whole computing industry is
going...


So it is clear the benefits of multicore technology are real and immediate without 
invocation of multitreading.

Possible exceptions to this conclusion would be for online
processing of data for trigger purposes or perhaps for fast diagnostic
of the quality of the data. Diagnostics (in STAR) are usually based on
a fairly large dataset so the advantage of multi-threading are dubious
at best in this case because the througput for one event is then
irrelevant - and it is the aggregate throuput that matters.

Online triggering is then the only justifiable case for use of
multithreading.  Multithreading would in principle enable faster
throughput for each event thereby enabling sophisticated algorithms.
This is however a very special case and it is not clear that adapting
the whole star software for this purpose is a worthy endeavor - that's
your call.

I should say in closing that the mood of the committee was overall
quite pessimistic from the onset. Perhaps a different group of people
could provide  a slightly different point of view - but I really doubt
it.

Projects and proposals

This page will either have requirements document or project description for R&D related activity in S&C (or defined activities hopefully in progress).

1. We proposed an R&D development within the ROOT framework to support full schema evolution as described in the project description
2. We worked on the elaboration of the KISTI proposal to join STAR
3. We supported a network upgrade for the RCF backbone (sent via Micheal Ernst to Tom Ludlam Physics departement chair at BNL on April 16th 2008, discussed at the Spokesperson meeting on April 4th 2008)
4. For 2008 Network Requirements Workshop for the DOE/SC Nuclear Physics Program Office, we provided background material as below (STAR sections and summary, the Phenix portion was taken verbatim from their contribution)
5. Trigger emulation / simualtion framework: discussions from 20070528 and following Emails.

Ongoing activities

Internal projects and sub-systems task lists

• The TPC sub-system task list
  Status: sent to the advisory board and spokesperson office on April 11th 2008
• Computing operation and support: IO performance measurements
• Opened project and activities listing

Tasks and projects

Computing operation: IO performance measurements

Goals:

• Provide a documented reference of IO performance tests made toward several configuration sin both disk formatting and RAID level space under non-constrained hardware considerations.
• The base line would help making future configuration choices when it comes to hardware provisioning of servers (services) such as database servers, grid gatekeepers, network IO doors, etc...

Steps and tasks:

1. Survey community work on the topic of IO performance of drives especially topics concerning
   1. Effect of disk format on performance
   2. Effect of parallelism on performance
   3. Effect of software raid (Linux) performance and responsiveness (load impact on node under stress)
   4. Software RAID level and performance impacts
   5. Kernel parameter tweaks impacting IO performance (good examples are efforts of DAQ group, review consequence)
       
2. Prepare a baseline IO test suite for measuring IO performance (read and write) under two mode Possible test suite could follow what was used in the  IO performance page . Other tools welcomed based upon survey recommendations.
   • single stream IO
   • multi stream IO (parallel IO)
      
3. Use a test node and measure IO performance under the diverse reviewed configurations. A few constraints on choice of hardware are needed to avoid biasing the performance results
   • The node should have sufficient memory to accommodate for the tests (2 GB of memory or more is assumed to be large sufficient to accommodate for any tests)
   • OS must support software RAID
   • Disks used for the test should be isolated from system drive to avoid performance degradation
   • Node should have more than two drives (including system disk) and ideally, at least 4 (3+1)
      
4. Present result as a function of disk formatting, RAID level and/or number of drives added in both absolute values (values for each configuration) and differentials (gain when moving from one configuration to another).

Status: See results on Disk IO testing, comparative study 2008.

Opened project and activities listing

A summary of ongoing and incoming projects was sent to the software coordinators for feedback. The document refers to projects listed in this section under Projects and proposals.

The list below does NOT include general tasks such as the one described as part of the S&C core team roles as defined in the Organization job descriptions documents . Examples of which would be global tracking with Silicon including HFT, geometry maintenance and updates or otherwise calibration or production tasks as typically carried for the past few years. Neither does this list include improvements we need for areas such as online computing (many infrastructure issues, including networking an area of responsibility which has been unclear at best) nor activities such as the development and enhancement of the Drupal project (requirements and plans sent here).

The list includes:

• Closer look at Calorimetry issues if any (2007 operation workshop feedback follow-up related to calibration being too"TPC centric" and not addressing Physics qualities). Proposed a workshop with goals to:
  • gather requirements from the PWG (statements from the operation workshop in 2007 seemed to have taken the EMC coordinators by surprised as per what resolution was needed to achieve Physics goals)
  • discuss with experts technical details and implementation, unrolling / deployment and timing
Status: Underway, see report from a review as PSN0465 : EMC Calibrations Workshop report, fall 2008


• Db related: load balancing improvements, monitoring and performance measurements, resource discovery, distributed database
  Status: underway.
  References: You do not have access to view this node
   
• Trigger simulations - (some fleshed out on May 2007 as mentioned in this S&C meeting and attached below). The general idea was to provide a framework to allow trigger emulation / simulation offline for studying rejection/selection effects either applying trigger algorithms on real data (minimum bias) or via true simulation or allow re-applying trigger algorithm to triggered sample (higher threshold for example)
  Status: nowhere close to where it should be
  References: trigger simulation discussions meeting notes and Email communications.
   
• Embedding framework reshape.
  Status: underway (need full eval with SVT and SSD integrated)
   
• Unified online/offline framework including integration of online reader offline and offline tools online (leveraging knowledge, minimizing work). This task would address comments and concerns that whenever code is developed online (for PPlot purposes for example), it also needs to be developed offline within separate and very different reader approaches. At a higher level, dramatic memory overwrite offline occurred in early 2007 due to the lack of synchronization between structure sizes (information did NOT propagate and was not adjusted offline by the software sub-system coordinator of interest; an entire production had to be re-run).
  Status: tasked and underway, first version delivered in 2008, usage of "cons" and regression testing in principle in place (TBC in 2009 run)
   
• EventDisplay revisited
  Status: underway (are we done? need new review follow-up after the pre-review meeting made in 2007)
   
• VMC - realistic geometry / geometry description
  Status: Project on hold due to reconstruction issues, resumed July 2008.
   
• Forward tracking (radial field issue). May have importance for FGT project upon schedule understanding.
  Status: depend on previous item and would be tasked whenever forward tracking need would be better defined.
   
• Old framework cleanup, table cleanup, drop old formats and historical baggage. In principle a framework tasks, this is bound to introduce instabilities during which assembling a production library would be challenging. This need to be tasked outside major development projects.
  Status: only depend on production of Year 7/8 start-up
   
• Multi-core CPU era - Task force assembled in 2007 (Multi-core CPU era task force) had an unfortunate conclusion that the work would be too hard hence not necessary. Unfortunately, market development and aggressive company progression toward even more packed CPU and core indicates the future must integrate this new paradigm. First attempts should target the "obvious".
  Status: First status and proposal made at ACAT08 (changing chains to accommodate for possible parallelism). Investigated possibility of parallelism at library level and core algorithm (tracking). Talks at ACAT08 very informative.
   
• Automated QA (project draft available, Kolmogorov etc... discussed and summarized here)
  Status: no project drafted yet, only live discussions and Email communications.
   
• Automated calibration. The main project objective is to move toward a more automated calibration framework whereas migration from one chain to another chain (distortion correction) would be triggered by a criteria (resolution, convergence) rather than a manual change. This work may leverage the FastOffline framework (which was a first attempt to make automated calibration a reality; currently modified by hand and the trigger mechanism is not present / implemented)
  Status: Project description available . Summer 08 service task.
   
• IO schema evolution (reduction of file size by dropping redundant variables but with full transparency to users)
  Status: Project started as planned on July 16th with goals drafted on page Projects and proposals. Project deliverables were achieved (tested from a custom ROOT version now in the ROOT main CVS). Future release will include a fully functional schema evolution as specified in our document. Integration will be needed.
  Project team: Jerome Lauret (coordination), Valeri Fine (STAR tetsing), Philippe Canal (ROOT team)
   
• Distributed storage improvement (Efficient dynamic disk population). This project would aim to restore the dynamic disk population of datasets on distributed disk as well as a prioritization mechanism (and possibly bandwidth throttling) so user cannot over-subscribe storage, causing past observed massive delete/restore dropping efficiency.
  Status: under-graduate thesis done ; model to improve IO in/out of HPSS is defined and need implementation.
   
• Efficient multi-site data transfer (coordination of data movement), this project aims to address multi-Tier2 data transfer support and help organize / best utilize the bandwidth out of BNL. A second part of this project aims at data placement on Grid whereas a "task" working on a dataset is to be scheduled with use of existing staged files at sites or possible pre-staging or migration of files from any site to any site (a bit ambitious).
  Status: Project started as a computer science PhD program (thesis submitted). Work scheduled over a 3 years period and deliverable would need to be put in perspectives of Grid project deliverables.
   
• Distributed production and monitoring system, job monitoring, centralized production requests interface
  Status: work tasked within the production team.
   
• FileCatalg improvement. The FileCatalog in STAR was developed from in-house knowledge and support (starting from service work). The catalog now hold 15 Million records (scalability beyond is a concern) and its access possibly inefficient. An initial design diverging from Meta-Data catalog, File Catalog, Replica Catalog has allowed for a quick start and the development of additional infrastructure but has also lead to the replication of the Meta Data information, making hard to maintain consistency of the Catalogs across sites. Federating the Catalogs and using all site's information simultaneously has been marginal to not possible, making a global namespace (replicas) not possible. The lack of this component will directly affect grid realities.
  Status: Ongoing (see Catalog centralized load management, resolving slow querries).

Wish list (for now):

• Online tracking & High Level trigger. This may depend on a trigger simulation framework (it would have benefited from it for sure) or may be an opportunity to revive the issue and shape anew focused (and reduced in scope) project.
  Status: How to fit this additional activity is under debate. First discussion held at BNL on 2008/07/10 and followed later by additional meetings. This activity moved to the "upgrade" activity.

STAR/RCF resource plans

• General fund
• External fund summary
• 2008 requirements
  • Allocation within budget
• Post plan adjustements
  • BlueArc layout within Physical stores, BA, ...
  • Proposal for reshape

General fund

 The level of funding planned for 2008 was:

• According to the RHIC mid-term strategic planning for 2006-2011 document, the budget for 2008 was projected to be 2140 k$ (table 7-2) with a note that and additional 2 M $ additional would be needed between FY08 and FY10 (to accommodate for network infrastructure, storage robotics and silo expansion and general infrastructure changes)
• The budget planned for FY08 in FY07 was 2.5 M$, accounting for recovering by 0.5 M$ already present past years shortfalls
• The current budget available is 1.7 M$ with a 1.5 M$ usable base fund.

External funds

Following previous years "outsourcing" of funds approach, an note was sent to the STAR collaboration (Subject: RCF requirements & purchase) on 3/31/2008 12:18. The pricing offered was 4.2 $/GB i.e. 4.3 k$/TB of usable space. Based on the 2007 RCF requirement learning experience (pricing was based on vendor's total space rather than usable), the price was firmed, fixed and guaranteed as "not higher than 4.2 $/GB" by the facility director Micheal Ernst at the March 27th liaison meeting.

The institutions external fund profile for 2008 is as follows:

Penn State university provided (late) funds for 1 TB worth.

*** WORK IN PROGRESS ***

Requirements 

The requirements for FY08 are determined based on 

The initial STAR requirements provided for the RHIC mid-term strategic plan can be found here

STAR resource requirements FY05-FY12

The initial raw data projected was 870 TB (+310 TB).

The RAW data volume taken by STAR in FY08 (shorter run) is given by the HPSS usage (RAW COS) as showed below:


A total of 165 TB was accumulated far below expected data projections by a factor of 2. The run was however declared as meeting (to exceeding) goals comparing to the STAR initial BUR.

Some notes:

• STAR made extensive use this year of fast triggers
   
• Based on those numbers, we assumed that
  • The CPU requirements of 1532 kSI2k (+1071 kSI2k) would equally scale, hence a minimal requirement of +215 kSI2K should be accounted for
  • A bigger pool of distributed storage would allow for more flexibility: it would allow for re-considering multiple (if not most of) the datasets to be placed on disk in Xrootd pool + it would allow (modulo expanding beyond the 1.2 replication baseline) to better load balance the resources.
  • The distributed disk planing accounted for 365 TB of storage (1 pass production, small fraction of past results on disk). We targeted 800 TB of disk space (about twice the initial amount).

Allocations within total budgets

scenario B = scenario A + external funds

Post purchase actions

BlueArc disk layout before the new storage commissioning

Reshape proposal

Missing information and progress records:

• 2008/08/14 13:44 - Answer from the RCF as per the above plan being approved (and commented it seemed easy)
  • Two caveats: ETA cannot be provided until migration starts (one test example) to get a more accurate estimate
  • While virtualmount point are swapped between one storage pool to another, there may be a fluke in access (will need infomring institutions / production disk will be handled by hard-dismount)
     
• 2008/08/14 13:42 - Sent an Email requesting infomration regarding disk manager and/or policies for PSU, UKY and RICE - Email sent to council rep and/or designated rep on August 14th 2008
  • Answer from UKY 2008/08/14 13:56 >> Disk space manager=Renee Fatemi, policy = MIT policy
  • Answer from PSU 2008/08/15 16:07 >> Policy is standard
     
• 2008/08/18
  • Achieved actions marked in Italic + date
  • Date in Italic are ongoing actions
  • If two dates appear, the first is the start of the action and the second the end

CPU and bulk storage purchase 2009

The assumed CPU profile will be:

• 2 GB of memory per core
• Nearly 6 TB of disk space per node
• Several CPU model will be investigated for best price/performance ratio (bulk purchase pricing matters in this purchase hence coordination between STAR/Phenix is likely needed) - currentely being considered are
  
  • Xeon 5550 @ 3350 SI2k (scenario A)
  • Xeon 5560 @ 3526 SI2k (scenario B)

The share between space and CPU is as below within the following caveats:

• THe additional massive amount of storage (+170 TB for production) requires a secondary Titan head and the proper network switches. The total cost is projected to be ~  50k$ and we agreed to leave a ~ 20k$ unspent fund to move in this direction (cost shared with facility budget)

Disk space for FY09

Institution disk space

The below is what was gathered as the call sent to starsoft "Inquiry - institutional disk space for FY09" (with delay, a copy was sent to starmail on the 14^th of April 2009). The deadline was provided as the end of Tuesday the 14^th 2009, feedback was accepted until Wednesday the 15^th (anything afterward could have been ignored).

The pricing on the table is as initially advertised i.e. a BlueArc Titan 3200 based solution at 4.3 k$/ TB for fiber channel based storage. For a discussion of fiber channel versus SATA, please consult this posting in starsofi. A quick performance overview of the Titan 3200 is showed below:

Solution enables over 60,000 user sessions and thousands of compute nodes to be served concurrently.

The first scalability statement is over the top comparing to RHIC/STAR need but the second is by far reached at the RCF environment.

Production space

SATA based solution will be priced at 2.2 k$ / TB. While the price is lower than the fiber channel solution (and may be tempting), this solution is NOT recommended for institutional disk as the scalability for read IO at the level we are accustom to is doubtful (doubtful is probably an under-statement as we know by 5 years ago experience we will have to apply IO throttling).

As a space for production however (and considering resource constrained demanding cheaper solutions coupled with a Xrootd fast IO based aggregation solution which will remain the primary source of data access to users), the bet is that it will work if used as a buffer space (production jobs write locallyto the worker nodes, move files to central disk at the end as an additional copy along an HPSS data migration). There will be minimal guarantees of read performance access for analysis on those "production reserved" storage.

One unit of Thumper at 20k$ / 33 TB usable will be also purchased and tried out in special context. This solution is even less scalable and hence, requires a reduced amount of users and IO. The space targeted for this lower end may include (TBC):

• data06 & data07 (2 TB) - reserved for specific projects and not meant for analysis, performance would not an issue
• data08                (2 TB) - meant for Grid, IO is minimal there but we may need to measure data transfers compatible with KISTI based production
• /star/rcf               (5 TB) - production log space (delayed IO, mostly a one time saving and will be fine)

Final breakdown

Post procurement 1 space topology

Following the Disk space for FY09, here is the new space topology and space allocation. 

Projects & proposals

This page is under constructions. Most projects are stil under the Projects and proposals page and not revised.

• Supplemental funds were requested from DOE to help with infrastructure issues for both STAR & Phenix (and in predicion of a difficult FY10 funding cycle). The document is attached below as Supplemental-justification-v0 7.jl_.pdf
• CloudSpan: Enabling Scientific Computing Across Cloud and Grid Platforms proposal was granted a Phase-I SBIR. This proposal is made in collaboration with Virkaz Tech.
• Customizable Web Service for Efficient Access to Distributed Nuclear Physics Relational Databases proposal was granted a Phase-II award.

CPU and bulk storage purchase 2010

Institutional disk space summary

Announcement for institutional disk space was made in starmail on 2010/04/26 12:31.

To date, the following requests were made (either in $ or in TB):

The storage cost for 2010 was estimated at 3.4k$ / TB. Detail pricing below.

Central storage cost estimates

Since the past storage stretches on the number of servers and scalability, we would (must) buy a pair of mercury servers which recently cost us $95,937. The storage itself would be based on a recent pricing i.e. a recent configuration quoted it as: (96) 1TB SATA drives, price $85,231 + $2,500 installation yielding to 54 TB usable. STAR's target is 50 TB for production +5+10 TB for institution (it will fit and can be slightly expanded). Total cost is hence:

$95,937 + $85,231 + $2,500  =  $183,668 / 54TB = 3401/TB

Detail cost projections may indicate (depending on global volume) a possibly better pricing: the installation price (a haf a day of work for a tech from BlueArc) is fixed and each server pair could hold more than the planned storage (hence the cost for two servers is also fixed). Below a few configurations:

CPU estimates, choices, checks

Projected / allowed CPU need additional based on funding guidance (see CSN0474 : The STAR Computing Resource Plan, 2009): 7440 kSi2k / 2436 kSi2k    - projected to be 43% shortage
Projected distributed storage under the same condition (dd model has hidden assumptions): 417 TB / 495 TB  - projected to be at acquired level 130% off optimal solution

The decision was to go for 1U machine, switch to the 2 TB drive Hitachi HUA722020ALA330 SATA 3.0 Gbps drive to compensate from drive space loss (4 slots instead of 6 in a 2U). The number of network ports was verified to be adequate for our below projection. The 1U configuration allows recovering more CPU power / desnity. Also, the goal is to move to a boosted memory configuration and enable hyper-threading growing from 8 Batch slots to a consistent 16 slots per node (so another x2 although the performance scaling will not be x2 due to the nature of hyper-threading). Finally, it was decided NOT to retire the older machines this year but keep them on until next year.

Planned numbers

• Distributed storage additional: 1009.4 TB
  • Only 3/4th of this space is usage * 90% for high watermarking hence, we end up with 681 TB of new storage. The assumption is that one of teh 2 TB disk will go to support production and user analysis (the likely proper number is 1 TB, hence a 16% effect and margin TBC).
  • The total required space for considering all production passes within a year is 1440 TB.
  • The accumulated total usable distributed storage is 277 TB - the total space is hence planned to be 958 TB with the assumptions above (possibly 30% shortfall or only 15% if we recover 1 TB from the OS+TEMP disk).
  • Conclusion: distributed storage will remain constrained as planned (not all productions will be available but near all).
• The total centralized disk needed for 2010 was 50 TB. The final number will be a 81 TB unit - 24 TB for institutional support = 57 TB storage.
  • Conclusion: The central storage will have a small margin of flexibility allowing expansion of simu space and other similar areas
• The total needed CPU required was projected to be 11634 kSI2k
  • Within the current procurement, the total CPU will reach 8191 kSI2k with 1U nodes (would have been 6827 kSI2k for 2U nodes).
  • Our shortfall will be ~ 30% off the theoretical projected needs. Initial projection was a fall by 43% (so a 13% gain by balancing cost between storage, memory and CPU).
  • Assuming the hyper-threading will allow for at least a gail factor of x1.4 (TBC but evidence through beta-testing indicates this is likely), the shortfall may be as little as 16% shortfall. This number is within reacheable enhanced duty factor.
  • Conclusion: the shortfall, if the initial projections remain accurate, is assumed to be from 16 to 30%.

Reality checks:

• Event size estimates DAQ + Event size estimates reco - implicitly done in You do not have access to view this node - the bottom line is that the space is tracking fine but overall, we exceeded our goals not by 50% as initially though but by 79% +/- 2%.
• Processing time estimates (extracted from FastOflfine, final times are unclear due to slow time caused by the so-called Speeding up DB access using SSD or Memory (see also Effect of stream data on database performance, a 2010 study).

Accounting check - post install

Since we had many problems with missmatch of purchased/provided space with the RCF in past years, keeping track of the space accounting is a good idea. Below is an account of where the space went (we should total to 55 TB of production space and 26 TB of institution space).

There should be a 0.5 TB unallocated here and there.

Software effort level by sub-systems