Site information

This page will provide information specific to the STAR Grid sites.

BNL

GK Infrastructure

Gatekeeper Infrastructure

This page was last updated on May 17, 2016.

The nodes for STAR's grid-related activities at BNL are as follows:

Color coding

Black: in production (please, do NOT modify without prior warning)
Green: machine was setup for testing particular component or setup
Red : status unknown
Blue: available for upgrade upon approval

Grid Machine	Usage	Notes	Hardware Make and Model	OS version, default gcc version	Hardware arrangement	OSG base	Condor
stargrid01	FROM BNL, submit grid jobs from this node		Dell PowerEdge 2950 dual quad-core Xeon E5440 (2.83 GHz/ 1.333 GHz FSB), 16 GB RAM	RHEL Client 5.11, gcc 4.3.2	6 x 1TB SATA2: 1GB /boot (/dev/md0) is RAID 1 across all six drives There are 3 RAID 1 arrays using pairs of disks (eg. /dev/sda2 and /dev/sdb2 are one array). The various local mount points and swap space are logical volumes scattered across these RAIDed pairs. There are 2.68 TB of unassigned space in the current configuration. NIC: 2 x 1Gb/s (one in use for RACF IPMI/remote administration on private network)	OSG 3.2.25 Client software stack for job submission	8.2.8-1.4 (part of OSG install -- only for grid submission, not part of RACF condor)
stargrid02	file transfer (gridftp) server	Attention: on stargrid02, the mappings formerly were all grid mappings (i.e. to VO group accounts: osgstar, engage, ligo, etc...) On May 17, 2016, this was changed to map STAR VO users to individual user accounts (matching the behaviour of stargrid03 and stargrid04). This behavior may be changed back. (TBD) Former STAR-BNL site gatekeeper	Dell PowerEdge 2950 dual quad-core Xeon E5440 (2.83 GHz/ 1.333 GHz FSB), 16 GB RAM	RHEL Client 5.11, gcc 4.3.2	6 x 1TB SATA2: Configured the same as stargrid01 above NIC 2 x 1Gb/s (one in use for RACF IPMI/remote administration on private network)	OSG CE 3.1.23	7.6.10 (RCF RPM), NON-FUNCTIONAL (non-working configuration)
stargrid03	file transfer (gridftp) server	To transfer using STAR individual user mappings, please use this node or stargrid04	Dell PowerEdge 2950 dual quad-core Xeon E5440 (2.83 GHz/ 1.333 GHz FSB), 16 GB RAM	RHEL Client 5.11, gcc 4.3.2	6 x 1TB SATA2: Configured the same as stargrid01 above NIC 2 x 1Gb/s (one in use for RACF IPMI/remote administration on private network)	OSG CE 3.1.18	7.6.10 (RCF RPM), NON-FUNCTIONAL (non-working configuration)
stargrid04	file transfer (gridftp) server	To transfer using STAR individual user mappings, please use this node or stargrid03	Dell PowerEdge 2950 dual quad-core Xeon E5440 (2.83 GHz/ 1.333 GHz FSB), 16 GB RAM	RHEL Client 5.11, gcc 4.3.2	6 x 1TB SATA2: Configured the same as stargrid01 above NIC 2 x 1Gb/s (one in use for RACF IPMI/remote administration on private network)	OSG CE 3.1.23	7.6.10 (RCF RPM), NON-FUNCTIONAL (non-working configuration)

stargrid0[234] are using the VDT-supplied gums client (version 1.2.16).
stargrid02 has a local hack in condor.pm to adjust the condor parameters for STAR users with local accounts.

All nodes have GLOBUS_TCP_PORT_RANGE=20000,30000 and matching firewall conduits for Globus and other dynamic grid service ports.

General upgrade history (generally obsolete, but some valuable tidbits)
stargridXX (generally obsolete, but some valuable tidbits)
stargrid02 (generally obsolete, but some valuable tidbits)
stargrid04 (generally obsolete, but some valuable tidbits)

LBL

MIT

CMS Analysis Facility

MIT’s CMS Analysis Facility is a large Tier-2 computing center built for CMS user analyses. We’re looking into the viability of using it for STAR computing.

Initial Setup

First things first. I went to http://www2.lns.mit.edu/compserv/cms-acctappl.html and applied for a local account. The welcome message contained a link to the CMSAF User Guide found on this TWiki page.

AFS isn’t available on CMSAF, so I started a local tree at /osg/app/star/afs_rhic and began to copy over stuff. Here’s a list of what I copied so far (nodes are running SL 4.4):

CERNLIB
/afs/rhic.bnl.gov/asis/sl4/slc4_ia32_gcc345/cern

OPTSTAR
/afs/rhic.bnl.gov/i386_sl4/opt/star/sl44_gcc346

GROUP_DIR
/afs/rhic.bnl.gov/star/group

ROOT 5.12.00
/afs/rhic.bnl.gov/star/ROOT/5.12.00/root
/afs/rhic.bnl.gov/star/ROOT/5.12.00/.sl44_gcc346

SL07e (sl44_gcc346 only)
/afs/rhic.bnl.gov/star/packages/SL07e

I copied these precompiled libraries over instead of building them myself because of a tricky problem with the interactive nodes’ configuration. The main gateway node is a 64-bit machine, so regular attempts at compilation produce 64-bit libraries that we can’t use. CMSAF has a node reserved for 32-bit builds, but it’s running SL 3.0.5. We’re still working on a proper resolution of that problem. Perhaps we can force cons to do 32-bit compilations.

The environment scripts are working, although I had to add more hacks than I thought were necessary. I only changed the following files:

~/.login
~/.cshrc
$GROUP_DIR/site_post_setup.csh

It doesn’t seem possible to change the default login shell (chsh and ypchsh both fail), so when you login you need to type “tcsh” to get a working STAR environment (after copying my .login and .cshrc to your home directory, of course).

Basic interactive tests look good, and I’ve got a SUMS configuration that will do local job submissions to the Condor system (that’s a topic for another post). DB calls use the MIT database mirror. I think that’s all for now.

STAR Scheduler Configuration

I deployed a private build of SUMS (roughly 1.8.10) on CMSAF and made the following changes to globalConfig.xml to get local job submission working:

In the Queue List

In the Policy List

Now for the Dispatcher

And finally, here's the site configuration block

Database Mirror

MIT has a local slave connected to the STAR master database server. A dbServers.xml with the following content will allow you to connect to it:

<StDbServer>
<server> star1 </server>
<host> star1.lns.mit.edu </host>
<port> 3316 </port>
<socket> /tmp/mysql.3316.sock </socket>
</StDbServer>

For more information on selecting database mirrors please visit this page. You can also view a heartbeat of all the STAR database slaves here. Finally, if you're interested in setting up your own database slave, Michael DePhillips has put some preliminary instructions on the

Nonexistent node nid: 56.

Drupal page. Contact Michael for more info.

Guidelines For MIT Tier2 Job Requests

In order to facilitate the submission of jobs, all requests for the Tier2 must contain the following information. Note that, because we cannot maintain stardev on Tier2, all jobs must be run from a tagged release. It is the users responsibility to ensure that the requested job runs from a tagged release, with any necessary updates from CVS made explicit.

1. Tagged release of the STAR environment from which the job will be run, e.g. SL08a.

2. Link to all custom macros and/or kumacs.

3. Link to pams/ and StRoot/ directories containing any custom code, including all necessary CVS updates of the tagged release.

5. List of commands to be executed, i.e. the contents of the <job></job> in your submission XML.

One is also free to include a custom log4j.xml, but this is not necessary.

MIT Simulation Productions

Production Name	STAR Library	Species	Subprocesses	PYTHIA Library	BFC	Geometry	Notes
mit0000	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 4, CKIN(4) = 5
mit0001	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 5, CKIN(4) = 7
mit0002	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 7, CKIN(4) = 9
mit0003	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 9, CKIN(4) = 11
mit0004	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 11, CKIN(4) = 15
mit0005	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 15, CKIN(4) = 25
mit0006	SL08a	pp200	QCD 2->2	pythia6410PionFilter	"trs fss y2006c Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006c	CKIN(3) = 25, CKIN(4) = 35

Kumacs slightly modified to incorporate local pythia libraries from ppQCDprod.kumac and ppWprod.kumac provided by Jim Sowinski

Production Name	STAR Library	Species	Subprocesses	PYTHIA Library	BFC	Geometry	Notes
mit0007	SL08a	pp500	W	pythia6_410	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=10, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0008	SL08a	pp500	QCD 2->2	pythia6_410	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=20, CKIN(4)=30, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0009	SL08a	pp500	W	pythia6410FGTFilter	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=10, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit00010	SL08a	pp500	QCD 2->2	pythia6410FGTFilter	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=20, CKIN(4)=30, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0011	SL08a	pp500	QCD 2->2	pythia6410FGTFilterV2	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=5, CKIN(4)=10, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0012	SL08a	pp500	QCD 2->2	pythia6410FGTFilter	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=10, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0013	SL08a	pp500	QCD 2->2	pythia6410FGTFilterV2	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=15, CKIN(4)=20, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0014	SL08a	pp500	QCD 2->2	pythia6410FGTFilterV2	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=20, CKIN(4)=30, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0015	SL08a	pp500	QCD 2->2	pythia6410FGTFilterV2	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=30, CKIN(4)=50, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched
mit0016	SL08a	pp500	QCD 2->2	pythia6410FGTFilterV2	"trs -ssd upgr13 Idst IAna l0 tpcI fcf -ftpc Tree logger ITTF Sti StiRnd -IstIT -SvtIt -NoSvtIt SvtCL,svtDb -SsdIt MakeEvent McEvent geant evout geantout IdTruth bbcSim emcY2 EEfs bigbig -dstout fzin -MiniMcMk McEvOut clearmem -ctbMatchVtx VFPPV eemcDb beamLine"	upgr13	CKIN(3)=50, Custom BFC, vertex(0.1,-0.2,-60), beamLine matched

The seed for each file is given by 10000 * (Production Number) + (File Number). *The version of SL08c used is not the final version at RCF due to an unexpected update.

Production Name	STAR Library	Species	Subprocess	PYTHIA Library	BFC	Geometry	Notessuffix
mit0019	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=2, CKIN(4)=3, StGammaFilterMaker
mit0020	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=3, CKIN(4)=4, StGammaFilterMaker
mit0021	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=4, CKIN(4)=6, StGammaFilterMaker
mit0022	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=6, CKIN(4)=9, StGammaFilterMaker
mit0023	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=9, CKIN(4)=15, StGammaFilterMaker
mit0024	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=15, CKIN(4)=25, StGammaFilterMaker
mit0025	SL08c	pp200	Prompt Photon	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=25, CKIN(4)=35, StGammaFilterMaker
mit0026	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=2, CKIN(4)=3, StGammaFilterMaker
mit0027	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=3, CKIN(4)=4, StGammaFilterMaker
mit0028	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=4, CKIN(4)=6, StGammaFilterMaker
mit0029	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=6, CKIN(4)=9, StGammaFilterMaker
mit0030	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=9, CKIN(4)=15, StGammaFilterMaker
mit0031	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=15, CKIN(4)=25, StGammaFilterMaker
mit0032	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=25, CKIN(4)=35, StGammaFilterMaker
mit0033	SL08c	pp200	QCD	p6410BemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=35, CKIN(4)=65, StGammaFilterMaker

The seed for each file is given by 10000 * (Production Number) + (File Number). *The version of SL08c used is not the final version at RCF due to an unexpected update.

Production Name	STAR Library	Species	Subprocess	PYTHIA Library	BFC	Geometry	Notessuffix
mit0034	SL08c	pp200	Prompt Photon	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=2, CKIN(4)=3, StGammaFilterMaker
mit0035	SL08c	pp200	Prompt Photon	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=3, CKIN(4)=4, StGammaFilterMaker
mit0036	SL08c	pp200	Prompt Photon	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=4, CKIN(4)=6, StGammaFilterMaker
mit0037	SL08c	pp200	Prompt Photon	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=6, CKIN(4)=9, StGammaFilterMaker
mit0038	SL08c	pp200	Prompt Photon	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=9, CKIN(4)=15, StGammaFilterMaker
mit0039	SL08c	pp200	Prompt Photon	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=15, CKIN(4)=25, StGammaFilterMaker
mit0040	SL08c	pp200	QCD	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=2, CKIN(4)=3, StGammaFilterMaker
mit0041	SL08c	pp200	QCD	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=3, CKIN(4)=4, StGammaFilterMaker
mit0042	SL08c	pp200	QCD	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=4, CKIN(4)=6, StGammaFilterMaker
mit0043	SL08c	pp200	QCD	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=6, CKIN(4)=9, StGammaFilterMaker
mit0044	SL08c	pp200	QCD	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=9, CKIN(4)=15, StGammaFilterMaker
mit0045	SL08c	pp200	QCD	p6410EemcGammaFilter	"trs fss y2006g Idst IAna l0 tpcI fcf ftpc Tree logger ITTF Sti VFPPV bbcSim tofsim tags emcY2 EEfs evout -dstout IdTruth geantout big fzin MiniMcMk clearmem eemcDb beamLine sdt20050727"	y2006g	CKIN(3)=15, CKIN(4)=25, StGammaFilterMaker

STAR environment on OS X

This page is obsolete -- please see Mac port of STAR offline software for the current status

In order of decreasing importance:

pams - still can't get too far here. No idea how the whole gcc -> agetof -> g77 works to compile "Mortran". I know VMC is the future and all that, but I think we really do need pams in order to have a useful STAR cluster.
dynamic library paths - specifying a relative object pathname to g++ -o means that the created dylib always looks for itself in the current directory on OS X. In other words, the repository is useless. Need to figure out how to tell cons to send the absolute path when linking. Executables work just fine; it's only the .dylibs that have this problem.
starsim - crashing on startup (!!!!! ZFATAL called from MZIOCH) Hopefully this is related to pams problems, although I do remember having some trouble linking.
root4star - StarRoot,StUtilities,StarClassLibrary,St_base do not load automatically as I thought they were supposed to. How do we do this at BNL?
QtRoot - has it's own build system that didn't work out of the box for me. Disabled StEventDisplayMaker and St_geom_Maker until I figure this out.

Contents of $OPTSTAR

I went through the list of required packages in /afs/rhic.bnl.gov/star/common/AAAREADME and figured out which ones were installed by default in an Intel OS X 10.4.8 client. Here's what I found:

perl 5.8.6: /usr/bin/perl (slightly newer than requested 5.8.4)
make 3.8.0: /usr/bin/make -> gnumake
tar (??): /usr/bin/tar
flex 2.5.4: /usr/bin/flex
libXpm 4.11: /usr/X11R6/lib/libXpm.dylib
libpng: not found
mysql: not found
gcc 4.0.1: /usr/bin/gcc -> gcc-4.0 (yeah, I know. Apple does not support gcc 3.x in 10.4 for Intel! We can do gcc_select to go back to 3.3 on ppc though.)
dejagnu: not found
gdb 6.3.50: /usr/bin/gdb (instead of 5.2)
texinfo: not found
emacs 21.2.1: /usr/bin/emacs (instead of 20.7)
findutils: not found
fileutils: not found
cvs 1.11: /usr/bin/cvs
grep 2.5.1: /bin/grep (instead of 2.5.1a)
m4 1.4.2: /usr/bin/m4 (instead of 1.4.1)
autoconf 2.59: /usr/bin/autoconf (2.53)
automake 1.6.3: /usr/bin/automake
libtool (??): /usr/bin/libtool (1.5.8)

I was able to find nearly all of the missing packages in the unstable branch for Fink (Intel machine). I wouldn't worry about the "unstable" moniker; as long as you don't do a blind update-all it's certainly possible to stick to a solid config, and there are several packages on the list that are only available in unstable (only because they haven't yet gotten the votes to move them over to stable). I've gone ahead and installed some of the missing packages in a fresh Fink installation and will serve it up over NFS at /Volumes/star1.lns.mit.edu/STAR/opt/star/osx48_i386_gcc401 (with a power_macintosh_gcc401 to match, although a more consistent $STAR_HOST_SYS would probably have been osx48_ppc_gcc401).

Here's a summary table of the packages installed in $OPTSTAR for the two OS X architectures at MIT. Note that many of these packages have additional dependencies, so the full list of installed packages on each system (attached at the bottom of the page) is actually much longer.

package	version
Fortran compiler	gfortran 4.2 (i386), g77 3.4.3 (ppc)
libpng	1.2.12
mysql	5.0.16-1002 (5.0.27 will break!)
dejagnu	skipped
texinfo	4.8
findutils	4.2.20
fileutils	5.96
qt-x11	3.3.7
slang	1.4.9
doxygen	1.4.6
lynx	2.8.5
ImageMagick	6.2.8
nedit	5.5
astyle	1.15.3 (ppc only)
unixodbc	2.2.11
myodbc	not available (2.50.39, if we want it)
libxml	2.6.26

I also looked for required perlmods in Fink. I stuck with the default Perl 5.86, so the modules that say e.g. pm588 required I did not install. I found that some of the modules are already part of core. If the older ones hosted by STAR are still needed, let me know. Virtual package means that it came with the OS already:

perlmod	version
Compress-Zlib	virtual package
DateManip	5.42a
DBI	1.53
DBD-mysql	3.0008
Digest-MD5	core module
HTML-Parser	virtual package
HTML-Tagset	3.10
libnet	not available
libwww-perl	5.805
LWPng-alpha	not available
MD5	not available
MIME-Base64	3.05
Proc-ProcessTable	0.39-cvs20040222-sf77
Statistics-Descriptive	2.6
Storable	core module
Time-HiRes	core module
URI	virtual package
XML-NamespaceSupport	1.08
XML-SAX	0.14
XML-Simple	2.16

There were some additional perlmods that install_perlmods listed as "Linux only" but Fink offered to install:

perlmod	version
GD	2.30
perlindex	not available
Pod-Escapes	1.04
Pod-Simple	3.04
Tk	804.026
Tk-HistEntry	not available
Tk-Pod	not available

Questions:

what was with all those soft-links (/usr/bin/sed -> /bin/sed, etc.) that Jerome had me make? Will they be needed on every machine running STAR environment (that's a problem), or just on the one he was compiling on?
is perl in /usr/bin sufficient or do we need to put it in $OPTSTAR as directed in AAAREADME?
what to do about mysql? Is 5.0 back-compatible, or do we only need development headers and shared libraries?

Building PYTHIA dylibs with gfortran

The default makePythia6.macosx won't work out of the box for 10.4, since it requires g77. Here's what I did to get the libraries built for Pythia 5:
$ gfortran -c jetset74.f $ gfortran -c pythia5707.f $ echo 'void MAIN__() {}' > main.c $ gcc -c main.c $ gcc -dynamiclib -flat_namespace -single_module -undefined dynamic_lookup -install_name $OPTSTAR/lib/libPythia.dylib -o libPythia.dylib *.o $ sudo cp libPythia.dylib $OPTSTAR/lib/. and for Pythia 6: $ export MACOSX_DEPLOYMENT_TARGET=10.4 $ gfortran -c pythia6319.f In file pythia6319.f:50551 IF (AAMAX.EQ.0D0) PAUSE 'SINGULAR MATRIX IN PYLDCM' 1 Warning: Obsolete: PAUSE statement at (1) $ gfortran -fno-second-underscore -c tpythia6_called_from_cc.F $ echo 'void MAIN__() {}' > main.c $ gcc -c main.c $ gcc -c pythia6_common_address.c $ gcc -dynamiclib -flat_namespace -single_module -undefined dynamic_lookup -install_name $OPTSTAR/lib/libPythia6.dylib -o libPythia6.dylib main.o tpythia6_called_from_cc.o pythia6*.o $ ln -s libPythia6.dylib libPythia6.so $ sudo cp libPythia6.* $OPTSTAR/lib/.

CERNLIB notes

All the CERNLIB libraries are static and the binaries depend only on system libraries, so the whole installation should be portable. For PowerPC I had a CERNLIB 2005 build left over from a different Fink installation, so I just copied those binaries and libraries to the new location and downloaded the headers from CERN. Fink doesn't support CERNLIB on Intel Macs, so for this build I used Robert Hatcher's excellent shell script:

http://home.fnal.gov/~rhatcher/macosx/readme.html

Hatcher's binaries link against the gfortran dylib, so I made sure to build them with gfortran from $OPTSTAR.

CERNLIB 2005 doesn't include libshift.a, but STAR really wants to link against it. Here's a hack from Robert Hatcher to build your own cat > fakeshift.c < eof int rshift_(int* in, int* ishft) { return *in >> *ishft; } int ishft_(int* in, int* ishft) { if (*ishft == 0) return *in; if (*ishft > 0) return *in << *ishft; else return *in >> *ishft; } EOF gcc -O -fPIC -c fakeshift.c fi g77 -fPIC -c getarg_stub.f ar cr libshift.a fakeshift.o eof

ROOT build notes

Following the instructions at http://www.star.bnl.gov/STAR/comp/root/building_root.html was basically fine. Here was my configure command for rootdeb:
./configure macosx --build=debug --enable-qt --enable-table --enable-pythia6 --enable-pythia --with-pythia-libdir=$OPTSTAR/lib --with-pythia6-libdir=$OPTSTAR/lib --with-qt-incdir=$OPTSTAR/include/qt which resulted in the final list Enabled support for asimage, astiff, builtin_afterimage, builtin_freetype, builtin_pcre, builtin_zlib, cern, cintex, exceptions, krb5, ldap, mathcore, mysql, odbc, opengl, pch, pythia, pythia6, python, qt, qtgsi, reflex, shared, ssl, table, thread, winrtdebug, xml, xrootd. I did run into a few snags:

MakeRootDir.pl didn't find my /usr/X11R6/bin/lndir automatically (even though that was in my $PATH) so I had edit the script and do it manually.
Had to run MakeRootDir.pl twice to get root and rootdeb directory structures in place, editing the script in between.
CVS was a mess. I had to drill down into each subdirectory that needed updating, and even then it puked out conflicts instead of patching the files, so I had to trash the originals first. Also, I'm fairly sure that root5/qt/inc/TQtWidget.h should have been included in the v5-12-00f tag, since my first attempt at compiling failed without the HEAD version of that file.

Hacking the environment scripts

set rhflavor = "osx48_" in STAR_SYS to get the name I chose for $STAR_HOST_SYS
I installed Qt in $OPTSTAR, so group_env.csh fails to find it

Building STAR software

I'm working with a checked out copy of the STAR software and modifying codes when necessary if the fix is obvious. So far I've got the following cons working: cons %QtRoot %StEventDisplayMaker %pams %St_dst_Maker %St_geom_Maker St_dst_Maker tries to subtract an int and a struct! Pams is a crazy mess of VAX-style Fortran STRUCTURES, but we really need it in order to run starsim. I haven't delved too deeply into the QtRoot-related stuff; I'm sure Valeri can help when the time comes. Hopefully we can get these things fixed without too much delay.

Power PC notes

why does everything insist on linking with libshift? It's not a part of CERNLIB 2005, so I used Hatcher's hack to get around it and stuck libshift.a in $OPTSTAR/lib
libnsl is not needed on OS X, so we don't link against it anymore
remove -dynamiclib and -single_module for executables
cfortran.h can't identify our Fortran compiler -- define it as f2c
asps/Simulation/starsim/deccc/fputools.c won't compile under power pc (contains assembly code!) -- skip it for now
g++ root4star brings out lots of linking issues; one killer seems to be that libpacklib from Fink is missing fzicv symbol.
- one very hack solution: install gfortran, use it to build CERNLIB with Hatcher script, replace libpacklib.a, copy libgcc.a and libgfortran.a from gcc 4.2.0 into $OPTSTAR/lib or other, then link against them explicitly
- needed to -lstarsim to get gufile, srndmc symbols defined
<malloc.h> -- on Mac they decided to put this in /usr/include/malloc, so we add this to path in ConsDefs.pm
cons wanted to link starsim using gcc and statically include libstdc++; on Mac we'll let g++ do the work. Also, -lstarsim seems to be included too early in the chain. Need to talk to Jerome about proper way to fix this, but for now I can hack a fix.
PAMS -- ACK!

Problems requiring changes to codes:

struct mallinfo isn't available on OS X
- for now we surround any mallinfo with #ifndef __APPLE__; Frank Laue says there may be a workaround
'fabs' was not declared in this scope
- add <cmath> in header
TCL.h from ROOT conflicts with system tcl.h because of case-insensitive FS
- TCL.h renamed to TCernLib.h in newer ROOT versions (ROOT bug 19313)
- copied TCL.h to TCernLib.h myself and added #ifdef __APPLE__ #include "TCernLib.h"
- this problem will go away when we patch/upgrade ROOT
passing U_Int to StMatrix::inverse() when it wants a size_t
- changed input to size_t (only affected StFtpcTrackingParams)
abs(float) is not legal
- change to fabs(float) and #include <cmath>

Intel notes

Basic problem here is (im)maturity of gfortran. Current Fink unstable version 4.2.0-20060617 still does not include some instrinsic symbols (lshift, lstat) that we expect to be there. Newer versions do have these symbols, and as soon as Fink updates I'll give it another go. I may try installing gcc 4.3 from source in the meantime, but it's not a high priority. Note that Intel machines should be able to run the Power PC build in translated mode with some hacking of the paths (force $STAR_HOST_SYS = osx48_power_macintosh_gcc401).

Xgrid

Summary of Apple's Xgrid cluster software and the steps we've taken to get it up and running at MIT.

http://deltag5.lns.mit.edu/xgrid/

Xgrid jobmanager status report

xgrid.pm can submit and cancel jobs successfully, haven't tested "poll" since the server is running WS-GRAM.
Xgrid SEG module monitors jobs successfully. Current version of Xgrid logs directly to /var/log/system.log (only readable by admin group), so there's a permissions issue to resolve there. My understanding is that the SEG module can run with elevated permissions if needed, but at the moment I'm using ACLs to explicitly allow user "globus" to read the system.log. Unfortunately the ACLs get reset when the logs are rotated nightly.
CVS is up-to-date, but I can't promise that all of the Globus packaging stuff actually works. I ended up installing both Perl module and the C library into my Globus installation by hand.
Current test environment uses SimpleCA, but I've applied for a server certificate at pki1.doegrids.org as part of the STAR VO.

Important Outstanding Issues

streaming stdout/stderr and stagingOut files is a little tricky. Xgrid requires an explicit call to "xgrid -job results", otherwise it just keeps all job info in the controller DB. I haven't yet figured out where to inject this system call in the WS-GRAM job life cycle, so I'm asking for help on gram-dev@globus.org.
Need to decide how to do authentication. Xgrid offers two options on the extreme ends of the spectrum. On the one hand we can use a common password for all users, and on the other hand we can use K5 tickets. Submitting a job using WS-GRAM involves a roundtrip user account -> container account -> user account via sudo, and I don't know how to forward a TGT for the user account through all of that. I looked around and saw a "pkinit" effort that promised to do passwordless generation of TGTs from grid certs, but it doesn't seem like it's quite ready for primetime.

USP

This is a copy of the web page that contains a log of the Sao Paulo grid activities. For the full documentaion, please go to http://stars.if.usp.br:8080/~suaide/grid/

Installation

In order to be fully integrated to the STAR GRID you need to have the following items installed and running (the order I present the items are the same order I installed them in the cluster). There are other software to install before full integration but this is the actual status of the integration.

Installing the batch system (SGE)

We decided to install the SGE because it is the same system used in PDSF (so it is scheduler compatible) and it is free. The SGE web site is here. You can donwload the latest version from their website.

Instructions to install SGE

Download from the SGE web site
gunzip and untar the file
cd to the directory

In the installation directory there are two pdf files. The sge-install.pdf contains instruction on how to install the system. The sge-admin.pdf contains instructions how to maintain the system and create batch queues. Our procedure to install the system was:

In the batch system server (in our case, STAR1)

Create the SGE_ROOT directory. In our case, mkdir /home/sge-root. This directory HAS to be available in all the exec nodes
copy the entire content of the installation directory to the SGE_ROOT directory
add the lines bellow to your /etc/services file
sge_execd      19001/udp
sge_qmaster     19000/tcp
sge_qmaster     19000/udp
sge_execd        19001/tcp
cd to the SGE_ROOT directory
Type ./install_qmaster
follow the instructions in the screen. In our case, the answers to the questions were:

Do you want to install Grid Engine under an user id other than >root< (y/n) >> n
$SGE_ROOT = /home/sge-root
Enter cell name >> star
Do you want to select another qmaster spool directory (y/n) [n] >> n
verify and set the file permissions of your distribution (y/n) [y] >> y
Are all hosts of your cluster in a single DNS domain (y/n) [y] >> y
Please choose a spooling method (berkeleydb|classic) [berkeleydb] >> classic
You can change at any time the group id range in your cluster configuration. Please enter a range >> 20000-21000
The pathname of the spool directory of the execution hosts. Default: [/home/sge-root/star/spool] >> [ENTER]
Please enter an email address in the form >user@foo.com<. Default: [none] >> [PUT YOUR EMAIL]
Do you want to change the configuration parameters (y/n) [n] >> n
We can install the startup script that will start qmaster/scheduler at machine boot (y/n) [y] >> y
Adding Grid Engine hosts. Do you want to use a file which contains the list of hosts (y/n) [n] >> n
Host(s): star1 star2 star3 star4 ...... (ADD ALL HOSTS THAT WILL BE CONTROLED BY THE BATCH SYSTEM)
Do you want to add your shadow host(s) now? (y/n) [y] >> n
Scheduler Tuning. Default configuration is [1] >> 1
Proceed with the default answers until the end of the script

You have installed the master system. To make sure the system will start at boot time type
ln -s /etc/init.d/sgemaster /etc/rc3.d/S95sgemaster
ln -s /etc/init.d/sgemaster /etc/rc5.d/S95sgemaster

Install the execution nodes (including the server, if it will be a exec node). This needs to be done in ALL exec nodes

add the lines bellow to your /etc/services file
sge_execd        19001/udp
sge_qmaster     19000/tcp
sge_qmaster     19000/udp
sge_execd        19001/tcp
cd to your SGE_ROOT directory
type ./install_execd

Answer the question about the SGE_ROOT directory location
Please enter cell name which you used for the qmaster. >> star
Do you want to configure a local spool directory for this host (y/n) [n] >> n
We can install the startup script that will start execd at machine boot (y/n) [y] >> y
Do you want to add a default queue instance for this host (y/n) [y] >> n (WE WILL CREATE A QUEUE LATER)
follow the default instructions until the end

You have now installed the master system. To start the system at boot time. type
ln -s /etc/init.d/sgeexecd /etc/rc3.d/S96sgeexecd
ln -s /etc/init.d/sgeexecd /etc/rc5.d/S96sgeexecd

Install a default queue to your batch system

type qmon
It opens a GUI window where you can configure all the batch system.
Click in the buttom QUEUE CONTROL
It opens another screen with the queues you have in your system
Click on ADD
Fill the instructions. See the file sge-admin.pdf for instructions. It is very simple.

Installing GANGLIA

Aditional information from STAR web site

You can download the ganglia packages from their web site. You need to install the following packages:

gmond - the monitoring system. Should be installed in ALL machines in the cluster
gmetad - the gathering information system. Should be installed in the machine that will collect the data (in our case, STAR1)
the web front end. This is nice to have but not essential. It creates a web page, like this one, with all the information in your cluster. You should have a web server running in the collector machine (STAR1) for this to work
rrdtool - this is a package that creates the plots in the web page. Necessary only if you have the web frontend.

To install Ganglia, proceed with the following

In each machine in the cluster

Install the gmond package (change the name to match the version you are installing)
rpm -ivh ganglia-gmond-3.0.1-1.i386.rpm
edit the /etc/gmond.conf file. The only change I made in this file was
cluster {
name = "STAR"
}
Type
ln -s /etc/init.d/gmond /etc/rc5.d/S97gmond
ln -s /etc/init.d/gmond /etc/rc3.d/S97gmond
/etc/init.d/gmond stop
/etc/init.d/gmond start

In the collector machine (STAR1)

Install the gmetad, web and rrdtool packages (change the name to match the version you are installing)
rpm -ivh ganglia-gmetad-3.0.1-1.i386.rpm
rpm -ivh ganglia-web-3.0.1-1.noarch.rpm
rpm -ivh rrdtool-1.0.28-1.i386.rpm
edit the /etc/gmetad.conf file. The only change I made in this file was
data_source "STAR" 10 star1:8649 star2:8649 star3:8649 star4:8649 star5:8649
Type
ln -s /etc/init.d/gmetad /etc/rc5.d/S98gmetad
ln -s /etc/init.d/gmetad /etc/rc3.d/S98gmetad
/etc/init.d/gmetad stop
/etc/init.d/gmetad start

MonaLISA

Aditional information from STAR web site

To install Monalisa in your system you need to download the files from their web site. After you gunzip and untar the file you need to perform the following steps:

Create a monalisa user in your master computer and its home directory
cd to the monalisa installation dir
type ./install.sh
Answer the following questions:

Please specify an account for the MonALISA service [monalisa]: [ENTER]

Where do you want MonaLisa installed ? [/home/monalisa/MonaLisa] : [ENTER]
Path to the java home []: [enter the path name for your java distribution]
Please specify the farm name [star1]: [star]
Answer the next questions as you wish

Make sure that Monalisa will run after reboot by typing:
ln -s /etc/init.d/MLD /etc/rc5.d/S80MLD
ln -s /etc/init.d/MLD /etc/rc3.d/S80MLD
You need to edit the following files in the directory /home/monalisa/MonaLisa/Services

ml.properties
MonaLisa.ContactName=your name
MonaLisa.ContactEmail=xxx@yyyy.yyy
MonaLisa.LAT=-23.25
MonaLisa.LONG=-47.19
lia.Monitor.group=OSG, star (Note that we are being part of both OSG and STAR groups)
lia.Monitor.useIPaddress=xxx.xxx.xxx.xxx (your IP)
lia.Monitor.MIN_BIND_PORT=9000
lia.Monitor.MAX_BIND_PORT=9010

Need to tell MonaLisa that I am using SGE as a batch system. For this, edit the Service/CMD/site_env file and add
```
SGE_LOCATION=/home/sge-root
export SGE_LOCATION
SGE_ROOT=/home/sge-root
export SGE_ROOT
```

It is important to make sure these ports are not blocked by your firewal, in case your system is behind one.

To start the MonaLisa service just type /etc/init.d/MLD start

Requesting a GRID certificate

By the way, you will have to request (for Grid usage) a user certificate. For instructions, click on the link http://www.star.bnl.gov/STAR/comp/Grid/Infrastructure/#CERT

A grid installation will require a "host" certificate. Jerome told me he never asked for one really ...
The certificate arrived three days after I requested it (with some help from Jerome). I them followed
the instructions that came with the email to validade and export the certificate.

Installing OSG

I think this is the last step to be fully GRID integrated. I have not used the certificate I got up to now. Lets see. To install the OSG package I followed the instructions in the following web page

http://osg.ivdgl.org/twiki/bin/view/Documentation/OsgCEInstallGuide

The basic steps were

Make sure pacman is installed. For this I had to update python to a version above 2.3. Pacman is a package management system. It can be downloaded from here
create a directory at /home/grid. This is where I installed the grid stuff. Thid directory needs to be visible in all the cluster machines

I typed

export VDT_LOCATION=/home/grid
cd $VDT_LOCATION
pacman -get OSG:ce

I just followed the log and answered the questions.

The entire installation process took about 20 minutes or so but I imagine it depends on the network connection speed.

After this installation was done I typed source setup.sh to complete the installation. No messages in the screen...

Because our batch system is SGE, we need to install extra packages, as stated in the OSG documentation page. I typed:

pacman -get http://www.cs.wisc.edu/vdt/vdt_136_cache:Globus-SGE-Setup

and these extra packages were installed in a few seconds.

I just followed the instructions in the OSG installation guide and everything went fine. One important thing is related to firewall setup. If you have a firewall running with MASQUERADE, in which your private network is not accessible from the outside world, and your gatekeeper is not the firewall machine, remember to open the necessary ports (above 1024) and redirect the ports number 2119, 2811 and 2812 to your gatekeeper machine. The command depends on your firewall program. If using iptables, just add the following rule to your filter tables:

$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2119 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2119 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2135 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2135 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2136 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2136 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2811 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2811 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2812 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2812 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2912 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2912 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 7512 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 7512 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 8443 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 8443 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 19000 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 19000 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 19001 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 19001 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 20000:65000 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 20000:65000 -j DNAT --to $STAR1

where $GLOBALIP is the external IP of your firewall and $STAR1 is the IP of the machine running the GRID stuff.

I also had to modify the files /home/grid/setup.csh and setup.sh to fix the HOSTNAME and port range. I added, in each file:
setup.csh

setenv GLOBUS_TCP_PORT_RANGE "60000 65000"
setenv GLOBUS_HOSTNAME="stars.if.usp.br"

setup.sh

export GLOBUS_TCP_PORT_RANGE="60000 65000"
export GLOBUS_HOSTNAME="stars.if.usp.br"

This assures that the port range opened in the firewall will correspond to those used in the GRID environment. Also, because I run the firewall in masquerade mode, I had to set the proper hostname, otherwise it will pick the machine name, and I do not want that to happen.

GridCat and making things to work...

It is very interesting to add your grid node to GridCat. It is a map, just like MonaLisa but it performs periodical tests to your gatekeeper, making it easier to find out problems (and, if you got to this point, there should be a few of them)

To add your gatekeeper to GridCat, go to http://osg.ivdgl.org/twiki/bin/view/Integration/GridCat

You will have to fill a form, following the instructions in the following link:

http://osg.ivdgl.org/twiki/bin/view/Documentation/OsgCEInstallGuide#OSG_Registration

If everything goes right, when your application is aproved you will show up in the GridGat Map, located http://osg-cat.grid.iu.edu:8080

well, this is were debuggins starts. Every 2-3 hours the GridCat tests their gatekeepers and assign a status light for each one, based on tests results. The tests are basically:

Authentication test
Hello world test
Batch submition (depends on your batch system)

submit a job
query the status of the job
canceling the job

file transfer (gridFtp)

This is were I spent my last few days trying to resolve the issues. Thanks a lot for all the people in the STAR-GRID list that helped me a lot with suggestions. But I had to find out a lot of stuff... This is what google is made for.... The main issue is the fact that our cluster is behind a firewall configures qith masquerading.... It means that the internal IP's of the machines (including the gatekeeper) are not visible. All the machines have the same IP (the gateway IP) for the outside world.... I think I am the only one in the GRID with this kind of setup :)

How to turn authentication and hello world to green?

This is the easiest... Need to map the following certificates to your grid map (/etc/grid-security/grid-mapfile)

"/DC=org/DC=doegrids/OU=People/CN=Leigh Grundhoefer (GridCat) 693100" XXXX
"/DC=org/DC=doegrids/OU=People/CN=Bockjoo Kim 740786" XXXX

The username 'XXXX' is the local username in your cluster... After this certificates were added to my mapfile the first two tests turned green

How to turn the batch system test green

It seems that SGE is not the preferable batch system in the GRID... Too bad because it is really nice and SIMPLE. Because of this the OSG interface to OSG does not work right.... I hope the bugs are fixed in the next release but just to keep log of what I did (with a lot of hel) in case they forget to fix it :)

mis-ci-functions

This file, located at $VDT_LOCATION/MIS-CI/etc/misci/ is responsible for checking your system basically every 10 minutes and extract information about your cluster. It uses the batch system to grab information. Of course, it does not work with SGE. Replace the file with the version 0.2.7, located here. Please check if your version is newer than this one before replacing...

sge.pm

This file is located at $VDT_LOCATION/globus/lib/perl/Globus/GRAM/JobManager/
Please check the following

In the BEGIN section

if $SGE_ROOT, $SGE_CELL and the commands ($qsub, $qstat, etc) are defined properly

In the submit section

Locate the line

$ENV{"SGE_ROOT"} = $SGE_ROOT;

add the line

$ENV{"SGE_CELL"} = $SGE_CELL;

The same in the pool section
In the clear section

locate the line system("$qdel $job_id >/dev/null 2>/dev/null");
replace for the following

    $ENV{"SGE_ROOT"} = $SGE_ROOT;
    $ENV{"SGE_CELL"} = $SGE_CELL;
    $job_id =~ /(.*)\|(.*)\|(.*)/;
    $job_id = $1;
    system("$qdel $job_id");

This will make your batch tests turn green. It means people can submit jobs, query, cancel, etc. I hope I did not miss anything in here...

Making the gridFTP to work

This was the most difficult part because of my firewall configuration and thanks google for making reseach in the web easier...

Before, please check if the services are listed in your /etc/services file

  globus-gatekeeper       2119/tcp        # Added by the VDT
  gsiftp                  2811/tcp        # Added by the VDT
  gsiftp2                 2812/tcp        # Added by the VDT
  gsiftp                  2811/udp        # Added by the VDT
  gsiftp2                 2812/udp        # Added by the VDT

If not, add them...

I started testing file transfer between gatekeepers by logging into another gatekeeper, getting my proxy (grid-proxy-init) and do a file transfer with the command:

globus-url-copy -dbg file:///star/u/suaide/gram_job_mgr_13594.log gsiftp://stars.if.usp.br/home/star/c

The -dbg mean debug is turned on... Everything goes fine until it starts transfering the data (STOR /home/star/c). It hangs and times out. Researching on the web, I found a bug report at

http://bugzilla.globus.org/globus/show_bug.cgi?id=1127

And a quote in the bottom of the page:

" ... The wuftp based gridftp server is not supported behind a firewall. The problem is in reporting the external IP address in the PASV response. You can see this by using the -dbg flag to globus-url-copy. You will see the the PASV response specifies your internal IP address.

The server should, however, work for clients using PORT. ..."

which means I am doommed... Researching more the web I found some solutions and what I did was:

replace file /etc/xinetd.d/gsiftp for this one

service gsiftp
{
     socket_type = stream
     protocol = tcp
     wait = no
     user = root
     instances = UNLIMITED
     cps = 400 10
     server = /auto/home/grid/vdt/sbin/vdt-run-gsiftp2.sh
     disable = no
}

restarted xinetd

modified the file /hom/grid/globus/etc/gridftp.conf to

# Configuration for file the new (3.9.5) GridFTP Server
inetd 1
log_level ERROR,WARN,INFO,ALL
log_single /auto/home/grid/globus/var/log/gridftp.log
hostname "XXX.XXX.XXX.XXX"

XXX.XXX.XXX.XXX is the IP of the gateway for the outside world

And this worked!!!!

Now all tests are geen and I am happy and tired!!! There are still a few issues left, basically in the cluster information query (number of CPU's, batch queues, etc) that are related to mis-ci-functions (I think) and I will have a look latter.

Another important thing, if you plan to have a cluster running jobs from outside and making file transfers with gsiftp it is necessary that the directory /etc/grid-security is available in all machines in the cluster, even if they are not gatekeepers. Also, the grid setup should be executed in all the nodes (/home/grid/setup.csh). If not, when a job start running in one of the nodes and it attempts to transfer the file with globus-url-copy it will fail. The solution I used was to have the directory grid-security in the /home/grid and make symbolic links in all the nodes.