FAQs and Tips

Software Site Licenses

The answer is (almost) always: No!
Neither STAR nor BNL have site licences for any Microsoft product, Hummingbird Exceed, WinZIP, ssh.com's software or much of anything intended to run on individual users' desktops. Furthermore, for most purposes BNL-owned computers do not qualify for academic software licenses, though exceptions do exist.

FAQ: PDF creation

Without Adobe Acrobat (an expensive bit of software), this can be a daunting question. I am researching answers, some of which are available in my Windows software tips. Here is the gist of it in a nutshell as I write this -- there are online conversion services and OpenOffice is capable of exporting PDF documents.

FAQ: X Servers

I recommend trying the X Server that is available freely with Cygwin, for which I have created some documentation here: Cygwin Tips. If you can't make that work for you, then I next recommend a commercial product called Xmanager, available from http://www.netsarang.com. Last time I checked, you could still download a fully functional version for a time-limited evaluation period.

TIP: Windows Hibernation trick

• "Standby" puts the machine in a low power state from which it can be woken up nearly instantly with some stimulus, such as a keystroke or mouse movement (much like a screensaver) but the state requires a continuous power source.  The power required is quite small compared to normal running, but it can eventually deplete the battery (or crash hard if the power is lost in the case of a desktop).
• "Hibernate" actually dumps everything in memory to disk and turns off the computer, then upon restarting it reloads the saved memory and basically is back to where it was.  While hibernating, no power source is required.  It can't wake up quickly (it takes about as long as a normal bootup), but when it does wake up, (almost) everything is just the way you left it.  One caveat about networking is in order here:  Stateful connections (eg. ssh logins) are not likely to survive a hibernation mode (though you may be able to enable such a feature if you control both the client and server configurations), but most web browsing activity and email clients, which don't maintain an active connection, can happily resume where they left off.

Imagine:  the lightning is starting, and you've got 50 windows open on your desktop that would take an hour to restore from scratch.  You want to hibernate now!  Here's how to enable hibernating if it isn't showing up in the shutdown box: 
Open the Control Panels and open "Power Options".  Go to the "Hibernate" tab and make sure the the box to enable Hibernation is checked.  When you hit "Turn Off Computer" in the Start menu, if you still only see a Standby button, then try holding down a Shift key -- the Standby button should change to a Hibernate button.  Obvious, huh?

For the curious:
There are actually six (or seven depending on what you call "official") ACPI power states, but most motherboards/BIOSes only support a subset of these.  To learn more, try Googling "acpi power state", or you can start here as long as this link works.  (Note there is an error in the main post -- the S5 state is actually "Shutdown" in Microsoft's terminology). 
From the command line, you can play around with these things with such straightforward commands as:

%windir%\System32\rundll32.exe powrprof.dll,SetSuspendState 1 

Even more obvious, right?  If you like that, then try this on for size.

TIP: My new computer is broken!:

Printers

STAR's publicly available printers are listed below. 

There are additional printing resources available at BNL, such as large format paper, plotters, lamination and such.  Email us at starsupport 'at' bnl.gov and we might be able to help you locate such a resource.

Printing from the wireless (Corus) network

The "standard" way of printing from the wireless network is to go through ITD's CUPS server on the wireless network.  How to do this varies from OS to OS, but here is a Windows walkthrough.  The key thing is getting the URI for the printer into the right place:
 

• Open the Printers Control Panel and click "Add a Printer". 
• Select the option to add a network printer.  (Ignore the list of printers that it generates automatically).
• Click on the button or option for "the printer that I want isn't listed". 
• Select the option for a shared printer and enter the green URL from the list above for the printer you want.
  eg. http://cups.bnl.gov:631/printers/HP_Color_LaserJet_4700_2
• On the next window, select the hardware manufacturer and model (if not listed, let Windows search for additional models).
• Print a test page and cross your fingers... 
• If your test print does not come out, it doesn't necessarily mean your configuration is wrong - sometimes a problem occurs on the the CUPS server that prevents printing - it isn't always easy to tell where the fault lies.

Since printing through ITD's CUPS servers at BNL has not been very reliable, here are some less convenient alternatives to using the printers that you may find handy.  (Note that with these, you can even print on our printers while you are offsite - probably not something to do often, but might come in handy sometimes.)
 

1.  Use VPN.  But if you are avoiding the internal network altogether for some reason, or can't use the VPN client, then keep reading...

2.  Get your files to rcf2.rhic.bnl.gov and print from there.  Most of printers listed above have rcf print queues (hence the column "rcf2 queue name").  But if you want to use a printer for which there is no queue on rcf2, or you have a format or file type that you can't figure out how to print from rcf2, then the next tip might be what you need.

3.  SSH tunnels can provide a way to talk directly (sort-of) to almost any printer on the campus wired network.  At least as far as your laptop's print subsystem is concerned, you will be talking directly to the printer.  (This is especially nice if you want to make various configuration changes to the print job through a locally installed driver.)  But if you don't understand SSH tunnels, this is gonna look like gibberish:

Here is the basic idea, using the printer in the Control Room.
It assumes you have access to both the RSSH and STAR SSH gateways.

The ITD SSH gateways might also work in place of rssh (I haven't
tried them yet).  If they can talk directly to our printers,
then it would eliminate step C below.

A.  From your laptop:

ssh -A -L 9100:127.0.0.1:9100 <username>@rssh.rhic.bnl.gov

(Note 1:  -A is only useful if you are running an ssh-agent with a
loaded key, which I highly recommend)

(Note 2:   Unfortunately, the rssh gateways cannot talk directly to our
printers, so we have to create another tunnel to a node that can...  If the
ITD SSH gateways can communicate directly with the printers, then the
next hop would be unnecessary...)

B.  From the rssh session:

ssh -L 9100:130.199.162.165:9100 <username>@stargw1.starp.bnl.gov

(Note 1: 130.199.162.165 is the IP address of onlprinter1.star.bnl.gov -
it could be replaced with any printer's IP address on the wired network.)
(Note 2:  port 9100 is the HP JetDirect default port - non-HP printers
might not use this, and there are other ways of communicating with HP
network printers, so ymmv - but the general idea will work with most TCP 
communications, if you know the port number in use. 

C.  On your laptop, set up a local print queue as if you were going to
print directly to the printer over the network (with no intermediate
server), but instead of supplying the printer's IP address, use
127.0.0.1 instead.

D. Start printing...


If you close either of the ssh sessions above, you will have to
re-establish them before you can print again. 

The two ssh commands can be combined into one and you can create an alias to
save typing the whole thing each time.  (Or use PuTTY or some other GUI SSH client
wrapper to save these details for reuse.)

You could set up multiple printers this way, but to use them
simultaneously, you would need to use unique port numbers for each one
(though the port number at the end of the printer IP would stay 9100).

Quick (?) start guide for visitors with laptops

So you brought a laptop to BNL… and the first thing you want to do is get online, right?
Ok, here's a quick (?) guide to getting what you want without breaking too many rules.

Wired Options:

• Visitors' network: Dark purple jacks (usually labeled VNxx) are on a visitors' network and are effectively outside of the BNL firewall. They support DHCP and do not require any sort of registration to use. Being outside the firewall can be advantageous, but will prevent you from
  using some network services within BNL (printing, for instance). (The rest of this page is largely irrelevant if you are using the visitors' network.)

• BNL network: If it isn't dark purple (and it isn't a phone jack) then it is on the BNL network, which supports DHCP on most subnets. (NB. The 60/61 subnet (available in parts of 1006, including the WAH) has a locally managed DHCP server -- contact Wayne Betts to be added to the access list). All devices on the BNL networks are required to be registered based on the MAC address that is unique to each network interface. To help enforce this policy, if you request a DHCP
  address from an unregistered node, you will be assigned a restricted address. With a restricted IP address, your web browser will be automatically redirected to the BNL registration page, and you will be unable to surf anywhere else until you are registered.

  When registering a laptop, fill in "varies" for the location fields. For the computer name field, I recommend using "DHCP Client" (unless you have a static IP address of course).

  Previously registered users are encouraged to verify and update their registration information by going to http://register.bnl.gov from the machine to be updated.

  There you can also find out more about the registration system and find links to some useful information for network users.

   

Windows

This area is intended to provide information for STAR members to assist in configuring and using typical desktop/laptop PCs at BNL.

  Windows 2000/XP and Scientific Linux/Redhat Enterprise Linux are the preferred Operating Systems within STAR at BNL for desktop computing, though there is no formal requirement to use any particular OS.

  These pages are intended to be dynamic, subject to the constantly changing software world and user input.   Feedback from users -- what you find indispensable; what is misleading, confusing or flat-out wrong; and what is missing that you wish was here -- can help to significantly increase the value of these pages.

  Additional pages that are under consideration for creation:

• Windows installation checklist (the basic software and configuration that should probably be on every Windows PC)
• Linux installation checklist
• Common Linux details and useful links, such as Linux equivalents to software for Windows.
• Resources specific to the experiment operations (eg. common DAQ NFS mounts)
• Publically useable terminals

Getting a computer account in STAR

1. Introduction
2. Getting an account and performing work at BNL
   • New users in STAR
   • Previously a STAR user
   • Additional steps for everyone

Introduction

First of all, if you are a new user, WELCOME to the RHIC/STAR collaboration and experiment. STAR is located at Brookhaven National Laboratory and is one of the premier particle detectors in the world.

As a (new) STAR user, you will need to be granted access to our BNL Tier0 computing facility in order to have access to the offline and online infrastructure and resources. This includes accessing BNL from remote or directly while visiting us on site. Access includes access to data, experiment, mailing lists, desktop computer for visitors to name only those. As a National Facility under the Department of Energy (DOE) regulations, a few steps are required for this to happen. Please, follow them precisely and make sure you understand their relevance.

Note:

The DOE requires proper credentials for anyone accessing a computing "resource" and expect such individual to keep credentials up-to-date i.e. in good standing. It is YOUR responsibility to keep valid credentials with Brookhaven National Laboratory's offices. Credentials include: being a valid and active STAR member, having a valid and active guest/user ID and appointment, having and keeping proper trainings. Any missing component would cause an immediate closure of access to computing resources.

In many cases, we rely on account name matching the one created at the RCF (for example, Hypernews or Drupal accounts need exact match to be approved) - this is enforced so we can accurately rely on the work already done by the RCF personnel and only base our automation on "RCF account exist and is active". The RCF personnel work with the user's office and other agencies to verify your credentials.


If you were a STAR user before and seek to re-activate your account, this page also has information for you.

 

Getting an account and performing work at BNL

ALL USERS - Ensure/Verify you are affiliated to STAR in our records

• Ask your council representative that he/she sends your information to the collaboration's record keeping person (at this point in time, this person is Liz Mogavero).
  Note: Your council representative IS the one responsible for keeping the list of authors and active members at all times. We will not (and cannot) consider requests coming from other STAR members.
   
• Pro-actively check the presence of your name and record in our Phone Book.
  Note: If you are not in our Phone Book, you are simply NOT a STAR user as far as we know as our PhoneBook is the central repository of active STAR members as defined by the STAR council representatives. 
   

New users in STAR

1. Request a Guest appointment
   You must be sure you have a valid guest appointment with the BNL User Office.
   
   Note1: Requesting a Guest ID requires a procedure called “Foreign Visit and Assignment”. This procedure involves steps such as background checks with Counter Intelligence and approval from the Department of State. The procedure could take up to 60 days from the time it is started (sensitive countries may take 90 days).
   Note2 : If you have done this already and are a valid Guest, please go to to this section.
   
   
   • Go to the Guest Registration Form and complete the registration as instructed.
     • Purpose of Visit: likely "Research" but if you come for other purposes, chose as appropriate ("CRADA" or "Interview" may apply for example) 
     • Experiment/Facility:  "Physics Dept (RHIC/AGS)"
     • Facility Code: "RHIC"
     • Type of Research: "STAR"
     • Type of Access Requested: likely "Open Research" if you stated your visit purpose as "Research"
     • Subject Code for this Visit/Assignment: likely "General Physics"
        
2. Be patient and wait for further instructions and the approval.
   • ONLY AFTER THE FIRST STEPS will you be able to proceed with the rest of the instructions below.
   • We will assume that you, from now on, have a valid Guest appointment and hold a Guest/BNL ID.
      
3. Ensure you have the required and mandatory training
   You MUST take the Cyber Security training and course GE-CYBERSEC. This training is mandatory and access to the facility computing resources will NOT be granted without it.
   You are also requested to read the Personal User Agreement which describes your responsibilities, the reasonable use and scope of personal use of computing equipments. In recent years, the BNL User Office have requested for the form to be signed and returned for their records. Please, do not skip any of those steps.
    
4. Request an RCF account
   To request a new RCF account, start here. The fields are explained on this instruction page.
   Note: There is a "Contact information" field which is aimed to be filled using an existing RHIC member (holder of a valid account and appointment) who can vouch for you. Put your council representative or team lead name there OR (in case of interview / CRADA etc...) the name of your contact and host at BNL. DO NOT use your own name for this field. DO NOT use the name a person who is NOT yet a STAR Member.
    
5. Additional steps are described below.

Previously a STAR user

1. you cannot remember how to login and need access but you are in good standing (all training valid, your BNL appointment is valid)
2. you have let your training expire but you are a valid BNL guest (your appointment with BNL has NOT expired)
3. your BNL appointment is about to expire or has expired not long ago
4. you are a RHIC user (from another experiment), and now coming to STAR

• First of all, please make sure you are in the STAR PhoneBook as indicated here.
  If you were a member of another experiment before, you will be joining STAR as either a member of an existing institution or joining as a new institution. All membership handling are the responsibilities of the STAR council (approval of new institutions) or your council representative. In both cases, we MUST find your name in our PhoneBook records.
   
• Instructions for the several use cases above
  
  1. If you are in good standing but cannot remember your login information at the RCF facility, please see Account re-activation
      
  2. You have let your BNL training expire - likely, you have not renewed or taken GE-CYBERSEC training available from the training page (please locate in the list at the bottom the course named GE-CYBERSEC). Within 24 hours of the training being taken/renewed again, the privilege to access the BNL computing resources using your RCF account will be re-established (the process will be automatic).
      
  3. Your appointment is about to expire, or has expired not long ago - you will need to go to the Extension requests .
     The Guest Central interface will help identifying your status and appointment expiration. This form could be used by users already having a BNL Guest ID. If you have let your appointment expire for a long time however, this form may let you know (or not show your old BNL badge/guest ID at all). In such a case, you should consider yourself as a "New user" and follow the first set of instructions above.
     For an appointment renewal, the starting point will be the Guest Extension Form.
      
  4. If you were a user before and now coming to STAR, you will need to follow
     • The general principles of being recorded in our PhoneBook as described above.
     • Request a NEW account at the RCF (using a different username than the one you previously had in another experiment)
     • Keep your training up-to-date as before (GE-CYBERSEC) ...
        
  5. Additional steps are described below.

Additional steps for everyone

1. Generate and upload your SSH keys to ensure secure login
   You may now read SSH Keys and login to the SDCC and following information in this section.
    
2. Drupal access
   
   1. Log in to RCF node to verify your account username/password working
   2. Download 2-Factor Authentication app to your mobile device (application ranges from Google or Microsoft Authenticator, Duo Mobile, Authy, FreeOTP, Aegis, ...)
   3. Contact Dmitry Arkhipkin or Jerome Lauret on MatterMost (https://chat.sdcc.bnl.gov, choose "BNL Login" with your RCF username/password) to obtain the 2-Factor Authentication QR code.
   4. Use your RCF username/password + 2-Factor Authentication code (read from the app on your mobile device) to log in to drupal.

• Web Access
• You do not have access to view this node your ssh keys will need to be uploaded to a different interface if you have a need to login to the online setup - this is needed mostly by experts (not by all users in STAR).
• Software Infrastructure for general information
• Video Conferencing ... and the related comments at the bottom of some of those pages (viewable only if you are authenticated to Drupal).

Account re-activation

The instructions here are for users who have an account at the RCF but have unfortunately let their BNL appointment expire or do not know how to access their (old) account.

• Account expired or is disabled
• Forgot your Kerberos password but need to upload your SSH keys to log-in

Account expired or is disabled

First of all, please be sure you understand the requirements and rationals explained in Getting a computer account in STAR.
As soon as your appointment with BNL ends or expires, all access to BNL computing resources are closed / suspended and before re-establishing it, you MUST renew your appointment first. In such case, we will not provide you with any access which may include access to Drupal (personal account) and mailing lists.

The simplest way to proceed is to

• Check that you do have GE-CYBERSEC training. You can do this by checking your training records.
  • If you do not, please take this training NOW as any future request will be denied until this training is complete
• Send an Email to RT-RACF-UserAccounts@bnl.gov requesting re-activation of your account. Specify the account name (Unix account name, not your name) if you remember it. If you don't your full name may do. The RCF team will check your status (Cyber training, appointment status) and
  • if any is not valid, you will be notified and further actions will be needed.
  • If all is fine, they will re-activate your account after verifying with us that you are a valid STAR user. Please consult Getting a computer account in STAR for what this means ...

If your appointment has expired, you will need to renew it. Please, follow the instructions available here.

Chicken and Egg issue? Forgot your password but did not upload SSH keys

If your account is valid, so is your appointment  but you have not logged in the facility for a while and hence, are unable to upload your SSH keys (as described in SSH Keys and login to the SDCC and related documents) this may be for you.

You cannot access the upload page unless you have a valid password as the access to the RCF requires a double authentication scheme (Kerberos password + SSH key). In case you have forgotten your password, you have first to send an Email to the RCF at RT-RACF-UserAccounts@bnl.gov asking to reset your password, then thereafter go to the SSH key upload interface and proceed.

Drupal access

This page describes how you can obtain the access to the STAR drupal pages. Please understand that your Drupal access is now tight to a valid SDCC login - no SDCC account, no access to Drupal. This is because we integrated Drupal login to the common infrastructure (the login is Kerberos based). Here are the steps to gain access then:

1. Get a computer account in STAR, make sure you have a valid guest appointment and valid cyber training, then request a RCF account  

   https://drupal.star.bnl.gov/STAR/comp/sofi/facility-access/general-access
   
   
   

2. Generate SSH keys and upload them to the SDCC - this has formally little to do with access to Drupal but will allow you to log to the facility and verify you know your Kerberos password (it should have been sent to you via EMail during the account creating with instructions on how to change it)
   https://drupal.star.bnl.gov/STAR/comp/sofi/facility-access/ssh-keys
   Now Log in to RCF node- again, this is to ensure you have the proper
   1. ssh xxx@ssh.rhic.bnl.gov    (xxx is your username on RCF, enter the passphrase for your SSH key)
   2. kinit (enter your SDCC kerberos password - this is the one you will use for accessing Drupal)
       
3. Download 2-Factor Authentication app to your mobile device (application ranges from Google or Microsoft Authenticator, Duo Mobile, Authy, FreeOTP, Aegis, ...)
   
   
   
4. The FIRST time you log to Drupal, you will actually NOT need a second factor (leave the "code" box blank) but MUST generate it right away (the second login will require it).
   1. To generate it, use the "(re)Create 2FA login" in the left hand-side menu, leave all default option (you need a time based OTP), import the QRCode displayed.
   2. Once imported, a 6 Digits code should appear in your app - this is he "code" you will need to enter in future in the third field named "code". Note the code changes every 30 seconds.
5. IF you have logged out  without generating a QRCode or forgot to import and have no "code", then you will need to contact Dmitry Arkhipkin or Jerome Lauret on MatterMost to obtain your 2FA QR-code (either can generate it for you after the fact or start a chat with both). SDCC chat is available at https://chat.sdcc.bnl.gov (please use a Direct Message, not a channel). Use "BNL Login" and your SDCC account user/Kerberos password to login. Do as discussed above (import the QRCode, make sure an entry appears, test your login right away, ...)
    
6. You are set. Drupal login will now ask for your SDCC username and Kerberos password and a 6-digit code you read from the 2-Factor Authentication app.

SSH Keys and login to the SDCC

How to generate keys for about every platform ... and actually be able to log to the SDCC

• General / Introduction
• Instructions for Windows, Linux/Windows interoperability
• Uploading keys to the SDCC
• I have done all of this, what's next?

General

What you find below is especially useful for those of you that work on several machines and platforms in and out of BNL and need to use ssh key pairs to get into SDCC.

• If you use Linux only or Windows only everywhere, all you need is follow the instructions on the SDCC web site and you are all set (see especially their Unix SSH Key generation page).
• Otherwise this page is for you.

The findings on this web page are a combined effort of Jérôme Lauret, Jim Thomas, and Thomas Ullrich. All typos and mistakes on this page are my doing. I am also not going to discuss the wisdom of having to move private keys around - all I want to do is get things done.

The whole problem arises from the fact that there are 3 different formats to store ssh key-pairs and all are not compatible:

• ssh.com: Secure Shell is the company that invented the (now public) ssh protocol. They provide the (so far) best ssh version for Windows which is far nicer than PuTTY. Especially the File Browser provided is so much nicer than the scp command interface. It is free for academic/university sites.
• PuTTY: a free ssh tool for Windows.
• OpenSSH: runs on all Linux boxes and via cygwin on Windows.

Despite all claims, OpenSSH cannot export private keys into ssh.com format, nor can it import ssh.com private keys. Public keys seem to work but this is not what we want. So here is how it goes:

[A] Windows: follow one of the instructions below

1. PuTTY (Windows)
   1. Download puttygen.exe from the PuTTY download page. You only need it once, but it might be good to keep it in case you need to regenerate your keys.
   2. Start the program puttygen.exe
      • Under parameters pick SSH-2 (RSA) and 1024 for the size of the key in bits.
      • Then press the Generate button. You will be asked to move your mouse over the blank area.
      • Enter a passphrase in the referring fields. The passphrase is needed as it will correspond to a password. Make a mental note of it as keys will not be usable without it.
      • I recommend to save the "key fingerprint" too since you will need it at the SDCC web site when uploading your public key. Just save it in a plain text file.
        Note: You can always generate it later from Linux with ssh-keygen -l -f <key_file> but since you will need access to a Linux system to do this, it is important you keep a copy of this now so you could proceed with the rest of the instructions. The picture below shows where the important fields are
        
   3. Saving keys
      • Press Save Public Key. To not confuse all the keys you are going to generate I strongly recommend to call it rsa_putty.pub.
      • Next press Save Private Key. Type rsa_putty as a name when prompted. PuTTY will automatically name it rsa_putty.ppk. That's your private key.
        Don't quit puttygen yet. Now comes the important stuff.
      • In the menu bar pick Conversions->Export OpenSSH key. When prompted give a name that indicated that this is the private key for OpenSSH (Linux). I used rsa_openssh. There is no public key stored only the private. We will generate the public one from the private one later.
      • In the menu bar pick Conversions->Export ssh.com key. When prompted give a name that indicated that this is the private key for ssh.com. I used rsa_sshcom. Again, there is no public key stored only the private. We will generate the public one from the private one later.
   4. All done. Now you have essentially 4 files: public and private keys for putty and private keys for ssh.com and OpenSSH.
2. Getting ssh.com to work (Windows):
   1. Here I assume that you have SSHSecureShell (client) installed, that is the ssh.com version. Open a DOS (or cygwin) shell. We now need to generate a public key from the private key we got from puttygen. Best is to change into the directory where your private key is stored and type: ssh-keygen2 -D rsa_sshcom . Note that the command has a '2' at the end. This will generate a file called rsa_sshcom.pub containing the public key. Now you have your key pair.
   2. Launch SSH and pick from the menu bar Edit->Settings.
      Click on GlobalSettings/UserAuthentication/Keys and press the Import button. Point to your public key rsa_sshcom.pub. The private key will be automatically loaded too. That's it. Press OK and quit SSH. We are not quite ready yet. We still have to generate and upload the OpenSSH key to SDCC.
3. Getting keys to work with OpenSSH/Linux:
   1. Copy the private key rsa_openssh to a Linux box (cygwin on Windows works of course too).
   2. Set the permissions such that only you can read the private key file:

      % chomod 600 rsa_openssh

   3. Generate the public key with:
      

      % ssh-keygen -y -f rsa_openssh > rsa_openssh.pub

   4. Now you have the key pair.
   5. To install the key pair on a Linux box copy rsa_openssh and rsa_openssh.pub to your ~/.ssh directory.
      Important: the keys ideally will be named id_rsa and id_rsa.pub, otherwise extra steps/options will be required to work with them.  So, you are recommended to also do

      % mv rsa_openssh ~/.ssh/id_rsa  
      % mv rsa_openssh.pub ~/.ssh/id_rsa.pub

      All done.  Note that there is no need to put your key files on every machine to which you are going to connect.  In fact, you should keep your private key file in as few places as possible -- just the source machine(s) from which you will initiate SSH connections.  Your public key file is indeed safe to share with the public, so you need not be so careful with it and in fact will have to provide it to remote systems (such in the next section) in order to use your keys at all.
       

[B] Uploading the public key to SDCC:

1. https://web.racf.bnl.gov/Facility/SshKeys/UploadSshKey.php
2. Make sure you upload the OpenSSH public key. Everything else won't work.
   You need to provide the key fingerprint which you hopefully saved from the instructions above.  In case of OpenSSH based keys, you can re-generate the fingerprint with
   % ssh-keygen -Emd5 -l -f <key_file>

Note that forcing MD5 hash is important (default hash is SHA256 the RACF interface will not take). All done.
If you followed all instructions you now have 3 key pairs (files). This covers essentially all SSH implementations there are. Where ever you go, whatever machine and system you deal with, one key pair will work. Keep them all in a very save place.

[C] Done.What's next?

Uploading your keys to the SDCC and STAR SSH-key management interfaces

You need to upload your SSH keys only once. But after your first upload, please wait a while (30 mnts) before connecting to the SDCC SSH Gatekeepers. Basic connection instructions, use:

% ssh -AX xxx@sssh.sdcc.bnl.gov
% rterm

The rterm command will open an X-terminal on a valid STAR interactive node. If you do NOT have an X11 server running on your computer, you could use the -i options of rterm for interactive (non X-term based) session.

If you intend to logon to our online enclave, please check the instructions on You do not have access to view this node to request an account on the STAR SSH gateways and Linux pool (and upload your keys to the STAR Key SSH Management system).  Note that you cannot upload your keys anywhere without a Kerberos password (both the SDCC and STAR's interface will require a real account kerberos password to log in). Logging in to the Online enclave involves the following ssh connection:

% ssh -AX xxx@cssh.sdcc.bnl.gov
% ssh -AX xxx@stargw.starp.bnl.gov

A first thing to see is that SDCC gatekeeper is here "cssh" as the network is spearated into a "campus" side (cssh) and a ScienceZone side (sssh). For convenience, we have asked Cyber security to allow connections from "sssh" to our online enclave as well (so if you use sssh all the time, it will work).

For the requested an account online ... note that users do not request access to the individual stargw machines directly.  Instead, a shared user database is kept on onlcs.starp.bnl.gov - approval for access to onlcs grants access to the stargw machines and the Online Linux Pool.  Such access is typically requested on the user's behalf when the user requests access to the online resources following the instructions at You do not have access to view this node, though users may also initiate the request themselves. 

Logging in to the stargw machines is most conveniently done Using the SSH Agent, and is generally done through the SDCC's SSSH gateways.  This additional step of starting an agent would be removed whenever we will be able to directly access the STAR SSH GW (as per 2009, this is not yet possible due to technical details).

See also

To learn more, see:

• SSH connection stability
• Caveats, issues, special cases and possible problems
• SSH Key Management (an overview of the STAR SSH key management system)

Caveats, issues, special cases and possible problems

Shortcut links

• SSH side effects (login in using different user names than on the remote client)
• X11 Forwarding, -X or -Y?? (and all of what you want to know and what you may not want to know)
  • When to use trusted (-Y not-secure) and untrusted (-X secure)
  • Customized setups
• Kerberos hand-shake and the RCF
• Generic group accounts - how does this work?
• Special nodes
• Other caveats related to Kerberos

SSH side effects

Please note that if you remote account name is different from your RCF account name, you will need to use

% ssh -X username@rssh.rhic.bnl.gov

specifying explicitly username rather as the form

% ssh -X rssh.rhic.bnl.gov

will assume a username defaulting to your local machine (remote from the BNL ssh-daemon stand point) user name where you issue the ssh command. This has been a source of confusion for a few users. The first form by the way is preferred as always work and removes all ambiguities.

X11 Forwarding: -X or -Y ??

-X is used to automatically set the display environment to a secure channel (also called untrusted X11 forwarding) . In other words, it enables X11 forwarding without having to grant remote applications the right to manipulate your Xserver parameters. If you want ssh client to always act like with X11 forwarding, have the following line added in your /etc/ssh/ssh_config (or any /etc/ssh*/ssh*_config ).

ForwardX11 yes

-Y enables trusted X11 forwarding. So, what does trusted mean? It means that the X-client will be allowed to gain full access to your Xserver, including changing X11 properties (i.e. attributes and values which alters the look and feel of opened X windows or things such as mouse controls and position info, keyboard input reading and so on).  Starting with OpenSSH 3.8, you will need to set

ForwardX11Trusted yes 

in the client configuration  to allow remote nodes full access to your Xserver as it is NOT enabled by default.

When to use trusted, when to use untrusted

Recent OpenSSH version supports both untrusted (-X) and trusted (-Y) X11 Forwarding. As hinted above, the difference is what level of permissions the client application has on the Xserver running on the client machine.  Untrusted (-X) X11 Forwarding is more secure, but unfortunately several applications (especially older X-based applications) do not support running with less privileges and will eventually die and/or crash your entire Xserver session.

Dilema? A rule of thumb is that while using trusted (-Y) X11 Forwarding will have less applications problems for the near future, try first the most secured untrusted (-X) way and see what happens. If remote X applications fail with a errorssimilar to the below:

X Error of failed request: BadAtom (invalid Atom parameter)
  Major opcode of failed request: 18 (X_ChangeProperty)
  Atom id in failed request: 0x114
  Serial number of failed request: 370
  Current serial number in output stream: 372

you will have to use the trusted (-Y) connection.

Per client / server setup?

Instead of a system global configuration which will require your system administrator's assistance, you may create a config file in your user’s home directory (client side) under the .ssh directory with the following line $HOME/.ssh/config

ForwardX11Trusted yes 

But it gets better as the config file allows per host or per-domain configuration. For example, the below is valid

Host *.edu
	ForwardX11 no
	User jlauret

Host *.starp.bnl.gov
	ForwardX11 yes
    	Cipher blowfish
	User jeromel

Host orion.star.bnl.gov
     ForwardAgent yes
     Cipher 3des
     ForwardX11Trusted yes

Host what.is.this
    User exampleoptions
    ServerAliveInternal=900
    Port 666
    Compression yes
    PasswordAuthentication no
    KeepAlive yes
    ForwardAgent yes
    ForwardX11 yes
    RhostsAuthentication no
    RhostsRSAAuthentication no
    RSAAuthentication yes
    TISAuthentication no
    PasswordAuthentication no
    FallBackToRsh no
    UseRsh no

As a side note, 3des is more secure thank blowfish but also 3x slower. If speed and security is important, use at least aes cypher.

Kerberos hand-shake, How to.

OK, now you are logged to the facility gatekeeper but any sub-sequent login would ask for your password again (and this would defeat security). But you can cure this problem by, on the gatekeeper, issue the following command (we assume $user is your user name)

% kinit -5 -d -l 7d $user

-l 7d is used to provide a long life K5 ticket (7 days long credentials). Note that you should afterward be granted an AFS token automatically upon login to the worker nodes on the facility. From the gatekeeper, the command

% rterm

would open a terminal from the least loaded node on the cluster where you are allowed to log.

Generic (group) accounts

Due to policy regulations, group or generic accounts login cannot be allowed at the facility unless the login is traceable to an individual. The way to log in is therefore to

• Log to the gatekeeper using SSH keys under your PERSONAL account as described at SSH Keys and login to the SDCC
• kinit -5 -4 -l 7d $gaccount
• In case of wide use generic account, one more jump to a "special" node will be necessary. For starreco and starlib for example, this additional gatekeeper node is rcas6003. From there, login to the rest of the facility could be done using rterm as usual (at least in STAR)

Special nodes

This section is about standing on one foot, tapping on to of your head and chanting a mantra unless the moon is full (in such case, the procedure involves parsley and sacrificial offerings). OK, we are in the realm of the very very special tricks for very very special nodes:

• The rmine nodes CANNOT be connected to anymore. However, one can use rsec00.rhic.bnl.gov has a gatekeeper, using your desktop keys and then jump from there to the rmine nodes.
  Scope: Subject to special authorization.
• The test node aplay1.usatlas.bnl.gov cannot be accessed using a Kerberos trick. Since there are two HOPs from your machine to aplay1, you need to use the ssh-agent. See instructions on the Using the SSH Agent help page.

K5 Caveats

• If you log to gatekeeper GK1 for your personal account, you will need to chose another gatekeeepr GK2 for your group account login. This will allow not interference of Kerberos credentials.
• Whenever you log in a gatekeeper and you know you had previously obtained Kerberos credentials on this gatekeeper, you should ensure the destruction of previous credential to avoid premature lifetime expiration. In other words, -l 7d will NOT give you a 7 days lifetime K5 ticket on a gatekeeper where previous credentials exists. To destroy previous credentials, be sure
  1. you do not have (still) opened windows using the credential. Check this by issuing a klist and observe the listing. Valid credentials used in opened session would look like this

     Valid starting     Expires            Service principal
     12/26/06 10:59:28  12/31/06 10:59:28  krbtgt/RHIC.BNL.GOV@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 10:59:30  12/31/06 10:59:28  host/rcas6005.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 11:11:48  12/31/06 10:59:28  host/rplay43.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 17:51:05  12/31/06 10:59:28  host/stargrid02.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 18:34:03  12/31/06 10:59:28  host/stargrid01.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 18:34:22  12/31/06 10:59:28  host/stargrid03.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/28/06 17:53:29  12/31/06 10:59:28  host/rcas6011.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25 
     

  2. If nothing appears to be relevant or existing, it is safe to issue the kdestroy command to wipe out all old credentials and then re-initiate a kinit.

Using the SSH Agent

General

The ssh-agent is a program you may use together with OpenSSH or similar ssh programs. The ssh-agent provides a secure way of storing the passphrase of the private key.

One advantage and common use of the agent is to use the agent forwarding. Agent forwarding allows you to open ssh sessions without having to repeatedly type your passphrase as you make multiple SSH hops. Below, we provide instructions on starting the agent, loading your keys and how to use key forwarding.

Instructions

Starting the agent

The ssh-agent is started as follow.

% ssh-agent

Note however that the agent will immediately display information such as the one below

% ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-fxDmNwelBA/agent.5884; export SSH_AUTH_SOCK;
SSH_AGENT_PID=3520; export SSH_AGENT_PID;
echo Agent pid 3520;

 
It may not be immediately obvious to you but you actually MUST type those commands on the command line for the next steps to be effective.

Here is what I usually do: redirect the message to a file and source it from the shell like this:

% ssh-agent >agent.sh 
% source agent.sh

The commands above will create a script containing the necessary shell commands, then the source command will load the information into your shell.  This assumes you are using sh. For csh, you need use the setenv shell command to define both SSH_AUTH_SOCK and SSH_AGENT_PID. A simpler approach may however be to use

% ssh-agent csh

The command above will start a new shell, in which the necessary environment variables will be defined in the newly started shell (no sourcing needed). 

Yet another method to start an agent and set the environment variables in tcsh or bash (and probably other shells) is this:
 

% eval `ssh-agent`

Now that you've started an agent and set the environment variables to use it, the next step is to load your SSH key.

 

Loading a key

The agent alone is not very useful until you've actually put keys into it. All your agent key management is handled by the ssh-add command. If you run it without arguments, it will add any of the 'standard' keys $HOME/.ssh/identity, $HOME/.ssh/id_rsa, and $HOME/.ssh/id_dsa.

To be sure the agent has not loaded any id yet, you may use the -l option with ssh-add.  Here's what you should see if you have not loaded a key:

% ssh-add -l
The agent has no identities.

To load your key, simply type

% ssh-add
Enter passphrase for /home/jlauret/.ssh/id_rsa:
Identity added: /home/jlauret/.ssh/id_rsa (/home/jlauret/.ssh/id_rsa)

To very if all is fine, you may use again the ssh-add command with the -l option. The result should be different now and similar to the below (if not, something went wrong).

% ssh-add -l
1024 34:a0:3f:56:6d:a2:02:d1:c5:23:2e:a0:27:16:3d:e5 /home/jlauret/.ssh/id_rsa (RSA)

Is so, all is fine.

Agent forwarding

Two conditions need to be present for agent forwarding to function:

• The server need to be set to accept forwards (enabled by default)
• You need to use the ssh client with the -A option

Usage is simply

% ssh -A user@remotehost

And that is all. For every hop, you need to use the -A option to have the key forwarded throughout the chain of ssh logins. Ideally, you may want to use -AX (where "X" enabled X11 agent forwarding).

Agent security concern

The ssh-agent creates a unix domain socket, and then listens for connections from /usr/bin/ssh on this socket. It relies on simple unix permissions to prevent access to this socket, which means that any keys you put into your agent are available to anyone who can connect to this socket. BE AWARE that root especially has acess to any file hence any sockets and as a consequence, may acquire access to your remote system whenever you use an agent.

Manpages indicates you may use the -c of ssh-add and this indeed adds one more level of safety to the agent mechanism (the agent will aks for the passphrase confirmation at each new session). However, if root has its mind on stealing a session, you are set for a lost battle from the start so do not feel over-confident of this option.

Addittional information

Help pages below links to the OpenSSH implementation of the ssh client/server and other ssh related documentation from our site.

• ssh-agent manpage
• ssh-add manpage
• ssh manpage
• How to safely start the ssh-agent from .bashrc
• ssh-add / ssh-agent issue
• SSH connection stability

SSH connection stability

• Your SSH connections are closed from home
• You get disconnected from any nodes without any reasons?
• ... and you are a PuTTY user
• ... or an Uglix SSH client user

PuTTY users

PuTTY to connect to gateway (from a home connection), you have to

• set a session, be sure to enable SSH

• go to the 'Connection' menu and have the following options box checked

• Disable Nagle's algorithm (TCP_NODELAY option)

• Enable TCP keepalives (SO_KEEPALIVE option)

• Furthermore, in 'Connection' -> 'SSH' -> 'Tunnels' enable the option

• Enable X11 forwarding

• Enable MIT-Magic-Cookie-1

• Save the session

Documentation on those features (explanation for the interested) are added at the end of this document.

SSH Users

SSH users and owner of their system could first of all be sure to manipulate the SSH client configuration file and be sure settings are turned on by default. The client configuration is likely located as /etc/ssh_config or /usr/local/etc/ssh_config depending on where you have ssh installed.

But if you do NOT have access to the configuration file, the client can nonetheless pass on options from the command line. Those options would have the same name as they would appear in the config file.

Especially, KEEP_ALIVE is controlled via the SSH configuration option TCPKeepAlive.

% ssh -o TCPKeepAlive=yes

You will note in the next section that a spoofing issue exists with keep alive (I know it works well, but please consider the ServerAliveCountMax mechanism) so, you may use instead

% ssh -o TCPKeepAlive=no -o ServerAliveInterval=15

Note that the value 15 in our example is purely empirical. There are NO magic values and you need to test your connection and detect when (after what time) you get kicked out and disconnected and set the parameters from your client accordingly. Let's explain the default first and come back to this and a rule of thumb.

There are two relevant parameters (in addition of TCPKeepAlive):

ServerAliveInterval

Sets a timeout interval in seconds after which if no data has been received from the server, ssh will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.

This option applies to protocol version 2 only.

ServerAliveCountMax

Sets the number of server alive messages (see above) which may be sent without ssh receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.

The default value is 3. If, for example, ServerAliveInterval (above) is set to 15, and ServerAliveCountMax is left at the default, if the server becomes unresponsive ssh will disconnect after approximately 45 seconds.

In our example

% ssh -o TCPKeepAlive=no -o ServerAliveInterval=15

The recipe should be: if you get disconnected after N seconds, play with the above and be sure to set a

time of ServerAliveInterval*ServerAliveCountMax <= 0.8*N, N being the timeout. Since ServerAliveCountMax is typically not modified, in our example we assume the default value of 3 and therefore a a 3x15 = 45 seconds (and we guessed a disconnect every minute or so). If you set the value too low, the client will send to much "chatting" to the server and there will be a traffic impact.

Appendix

Nagle's algorithm

This was written based on this article.

RPC implementations on TCP should disable Nagle. This reduces average RPC request latency on TCP, and makes network trace tools work a little nicer.

Determines whether Nagle's algorithm is to be used. The Nagle's algorithm tries to conserve bandwidth by minimizing the number of segments that are sent. When applications wish to decrease network latency and increase performance, they can disable Nagle's algorithm (that is enable TCP_NODELAY). Data will be sent earlier, at the cost of an increase in bandwidth consumption.

KeepAlive

The KEEPALIVE option of the TCP/IP Protocol ensures that connections are kept alive even while they are idle. When a connection to a client is inactive for a period of time (the timeout period), the operating system sends KEEPALIVE packets at regular intervals. On most systems, the default timeout period is two hours (7,200,000 ms).

If the network hardware or software drops connections that have been idle for less than the two hour default, the Windows Client session will fail. KEEPALIVE timeouts are configured at the operating system level for all connections that have KEEPALIVE enabled.

If the network hardware or software (including firewalls) have a idle limit of one hour, then the KEEPALIVE timeout must be less than one hour. To rectify this situation TCP/IP KEEPALIVE settings can be lowered to fit inside the firewall limits. The implementation of TCP KEEPALIVE may vary from vendor to vendor. The original definition is quite old and described in RFC 1122.

MIT Magic cookie

To avoid unauthorized connections to your X display, the command xauth for encrypted X connections is widely used. When you login, a .Xauthority file is created in your home directory ($HOME). Even SSH initiate the creation of a magic cookie and without it, no display could be opened. Note that since the .Xauthority file IS the file containing the MIT Magic cookie, if you ever run out of disk quota or the file system is full, this file CANNOT be created or updated (even from the sshd impersonating the user) and consequently, no X connections can be opened.

The .Xauthority file sometimes contains information from older sessions, but this is not important, as a new key is created at every login session. The Xauthority is simple and powerful, and eliminates many of the security problems with X.

Examples and other documentation

A few examples will be left here to guide users and installer.

#use FileCatalog;

#
# All IDs are named after their respective table. This MUST
# remain like this.
#  eventGeneratorID        -> eventGenerator+ID       in 'EventGenerators'
#  detectorConfigurationID ->detectorConfiguration+ID in 'DetectorConfigurations'
#
# etc...
#

DROP TABLE IF EXISTS EventGenerators;
CREATE TABLE EventGenerators
(
  eventGeneratorID      SMALLINT     NOT NULL    AUTO_INCREMENT,
  eventGeneratorName    VARCHAR(30)        NOT NULL,
  eventGeneratorVersion VARCHAR(10)     NOT NULL,
  eventGeneratorParams  VARCHAR(200),

  eventGeneratorIDate   TIMESTAMP       NOT NULL,
  eventGeneratorCreator CHAR(15)        DEFAULT 'unknown' NOT NULL,
  eventGeneratorCount   INT,
  eventGeneratorComment TEXT,
  UNIQUE        EG_EventGeneratorUnique (eventGeneratorName, eventGeneratorVersion, eventGeneratorParams),
  PRIMARY KEY (eventGeneratorID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS DetectorConfigurations; CREATE TABLE DetectorConfigurations
(
  detectorConfigurationID               INT             NOT NULL        AUTO_INCREMENT,
  detectorConfigurationName             VARCHAR(50)        NULL           UNIQUE,
  dTPC                                  TINYINT,
  dSVT                                  TINYINT,
  dTOF                                  TINYINT,
  dEMC                                  TINYINT,
  dEEMC                                 TINYINT,
  dFPD                                  TINYINT,
  dFTPC                                 TINYINT,
  dPMD                                  TINYINT,
  dRICH                                 TINYINT,
  dSSD                                  TINYINT,
  dBBC                                  TINYINT,
  dBSMD                                 TINYINT,
  dESMD                                 TINYINT,
  PRIMARY KEY (detectorConfigurationID)
) TYPE=MyISAM;


# Trigger related tables
DROP TABLE IF EXISTS TriggerSetups; CREATE TABLE TriggerSetups
(
   triggerSetupID               SMALLINT     NOT NULL    AUTO_INCREMENT,
   triggerSetupName             VARCHAR(50)        NOT NULL       UNIQUE,
   triggerSetupComposition      VARCHAR(255) NOT NULL,

   triggerSetupIDate            TIMESTAMP       NOT NULL,
   triggerSetupCreator          CHAR(15)       DEFAULT 'unknown' NOT NULL,
   triggerSetupCount            INT,
   triggerSetupComment          TEXT,
   PRIMARY KEY                  (triggerSetupID)
) TYPE=MyISAM;


DROP TABLE IF EXISTS TriggerCompositions; CREATE TABLE TriggerCompositions
(
  fileDataID                    BIGINT          NOT NULL,
  triggerWordID                 INT             NOT NULL,
  triggerCount                  MEDIUMINT       DEFAULT 0,
  PRIMARY KEY                   (fileDataID, triggerWordID)
) TYPE=MyISAM;



DROP TABLE IF EXISTS TriggerWords;
CREATE TABLE TriggerWords (
  triggerWordID         mediumint(9)   NOT NULL auto_increment,
  triggerWordName       varchar(50)  NOT NULL default '',
  triggerWordVersion    varchar(6)        NOT NULL default 'V0.0',
  triggerWordBits       varchar(6)   NOT NULL default '',
  triggerWordIDate      timestamp(14)       NOT NULL,
  triggerWordCreator    varchar(15)       NOT NULL default 'unknown',
  triggerWordCount      int(11)     default NULL,
  triggerWordComment    text,
  PRIMARY KEY           (triggerWordID),
  UNIQUE KEY TW_TriggerCharacteristic (triggerWordName,triggerWordVersion,triggerWordBits)
) TYPE=MyISAM;




DROP TABLE IF EXISTS CollisionTypes; CREATE TABLE CollisionTypes
(
  collisionTypeID SMALLINT NOT NULL AUTO_INCREMENT,
  firstParticle VARCHAR(10) NOT NULL,
  secondParticle VARCHAR(10) NOT NULL,
  collisionEnergy FLOAT NOT NULL,
  PRIMARY KEY (collisionTypeID)
) TYPE=MyISAM;


#
# A few dictionary tables
#
DROP TABLE IF EXISTS ProductionConditions; CREATE TABLE ProductionConditions
(
  productionConditionID         SMALLINT       NOT NULL      AUTO_INCREMENT,
  productionTag                 VARCHAR(10)   NOT NULL,
  libraryVersion                VARCHAR(10)   NOT NULL,

  productionConditionIDate      TIMESTAMP       NOT NULL,
  productionConditionCreator    CHAR(15)        DEFAULT 'unknown' NOT NULL,
  productionConditionCount      INT,
  productionConditionComments   TEXT,
  PRIMARY KEY                   (productionConditionID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS StorageSites; CREATE TABLE StorageSites
(
  storageSiteID                 SMALLINT      NOT NULL     AUTO_INCREMENT,
  storageSiteName               VARCHAR(30)  NOT NULL,
  storageSiteLocation           VARCHAR(50),

  storageSiteIDate              TIMESTAMP       NOT NULL,
  storageSiteCreator            CHAR(15)       DEFAULT 'unknown' NOT NULL,
  storageSiteCount              INT,
  storageSiteComment            TEXT,
  PRIMARY KEY                   (storageSiteID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS FileTypes; CREATE TABLE FileTypes
(
  fileTypeID                    SMALLINT NOT NULL        AUTO_INCREMENT,
  fileTypeName                  VARCHAR(30)    NOT NULL   UNIQUE,
  fileTypeExtension             VARCHAR(15)        NOT NULL,

  fileTypeIDate                 TIMESTAMP       NOT NULL,
  fileTypeCreator               CHAR(15) DEFAULT 'unknown' NOT NULL,
  fileTypeCount                 INT,
  fileTypeComment               TEXT,
  PRIMARY KEY                   (fileTypeID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS FilePaths; CREATE TABLE FilePaths
(
  filePathID                    BIGINT         NOT NULL         AUTO_INCREMENT,
  filePathName                  VARCHAR(255)   NOT NULL         UNIQUE,

  filePathIDate                 TIMESTAMP       NOT NULL,
  filePathCreator               CHAR(15) DEFAULT 'unknown' NOT NULL,
  filePathCount                 INT,
  filePathComment               TEXT,
  PRIMARY KEY                   (filePathID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS Hosts; CREATE TABLE Hosts
(
  hostID                        SMALLINT       NOT NULL         AUTO_INCREMENT,
  hostName                      VARCHAR(30)    NOT NULL DEFAULT 'localhost' UNIQUE,

  hostIDate                     TIMESTAMP       NOT NULL,
  hostCreator                   CHAR(15)     DEFAULT 'unknown' NOT NULL,
  hostCount                     INT,
  hostComment                   TEXT,
  PRIMARY KEY                   (hostID)
) TYPE=MyISAM;


DROP TABLE IF EXISTS RunTypes; CREATE TABLE RunTypes
(
  runTypeID                     SMALLINT  NOT NULL AUTO_INCREMENT,
  runTypeName                   VARCHAR(255)    NOT NULL   UNIQUE,

  runTypeIDate                  TIMESTAMP       NOT NULL,
  runTypeCreator                CHAR(15)  DEFAULT 'unknown' NOT NULL,
  runTypeCount                  INT,
  runTypeComment                TEXT,
  PRIMARY KEY                   (runTypeID)
) TYPE=MyISAM;


DROP TABLE IF EXISTS StorageTypes; CREATE TABLE StorageTypes
(
  storageTypeID                 MEDIUMINT       NOT NULL    AUTO_INCREMENT,
  storageTypeName               VARCHAR(6)   NOT NULL  UNIQUE,

  storageTypeIDate              TIMESTAMP       NOT NULL,
  storageTypeCreator            CHAR(15)       DEFAULT 'unknown' NOT NULL,
  storageTypeCount              INT,
  storageTypeComment            TEXT,
  PRIMARY KEY                   (storageTypeID)
) TYPE=MyISAM;





DROP TABLE IF EXISTS SimulationParams; CREATE TABLE SimulationParams
(
  simulationParamsID            INT             NOT NULL     AUTO_INCREMENT,
  eventGeneratorID              SMALLINT    NOT NULL,
  detectorConfigurationID       INT             NOT NULL,
  simulationParamComments       TEXT,
  PRIMARY KEY                   (simulationParamsID),
  INDEX         SP_EventGeneratorIndex          (eventGeneratorID),
  INDEX         SP_DetectorConfigurationIndex   (detectorConfigurationID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS RunParams;
CREATE TABLE RunParams
(
  runParamID                  INT        NOT NULL AUTO_INCREMENT,
  runNumber                   BIGINT     NOT NULL UNIQUE,
  dataTakingStart             TIMESTAMP,
  dataTakingEnd               TIMESTAMP,
  simulationParamsID          INT       NULL,
  runTypeID                   SMALLINT     NOT NULL,
  triggerSetupID              SMALLINT      NOT NULL,
  detectorConfigurationID     INT            NOT NULL,
  collisionTypeID             SMALLINT             NOT NULL,
  magFieldScale               VARCHAR(50)    NOT NULL,
  magFieldValue               FLOAT,
  runComments                 TEXT,
  PRIMARY KEY                          (runParamID),
  INDEX RP_RunNumberIndex              (runNumber),
  INDEX RP_DataTakingStartIndex        (dataTakingStart),
  INDEX RP_DataTakingEndIndex          (dataTakingEnd),
  INDEX RP_MagFieldScaleIndex          (magFieldScale),
  INDEX RP_MagFieldValueIndex          (magFieldValue),
  INDEX RP_SimulationParamsIndex       (simulationParamsID),
  INDEX RP_RunTypeIndex                (runTypeID),
  INDEX RP_TriggerSetupIndex           (triggerSetupID),
  INDEX RP_DetectorConfigurationIndex  (detectorConfigurationID),
  INDEX RP_CollisionTypeIndex          (collisionTypeID)
) TYPE=MyISAM;

DROP TABLE IF EXISTS FileData; CREATE TABLE FileData
(
  fileDataID                    BIGINT          NOT NULL AUTO_INCREMENT,
  runParamID                    INT             NOT NULL,
  fileName                      VARCHAR(255)       NOT NULL,
  baseName                      VARCHAR(255)       NOT NULL COMMENT 'Name without extension',
  sName1                        VARCHAR(255) NOT NULL COMMENT 'Will be used for name+runNumber',
  sName2                        VARCHAR(255) NOT NULL COMMENT 'Will be used for name before runNumber',
  productionConditionID         INT             NULL,
  numEntries                    MEDIUMINT,
  md5sum                        CHAR(32)     DEFAULT 0,
  fileTypeID                    SMALLINT NOT NULL,
  fileSeq                       SMALLINT,
  fileStream                    SMALLINT,
  fileDataComments              TEXT,
  PRIMARY KEY                   (fileDataID),
  INDEX         FD_FileNameIndex                (fileName(40)),
  INDEX         FD_BaseNameIndex                (baseName),
  INDEX         FD_SName1Index                  (sName1),
  INDEX         FS_SName2Index                  (sName2),
  INDEX         FD_RunParamsIndex               (runParamID),
  INDEX         FD_ProductionConditionIndex     (productionConditionID),
  INDEX         FD_FileTypeIndex                (fileTypeID),
  INDEX         FD_FileSeqIndex                 (fileSeq),
  UNIQUE        FD_FileDataUnique               (runParamID, fileName, productionConditionID, fileTypeID, fileSeq)
) TYPE=MyISAM;



# FileParents
DROP TABLE IF EXISTS FileParents; CREATE TABLE FileParents
(
  parentFileID                  BIGINT          NOT NULL,
  childFileID                   BIGINT          NOT NULL,
  PRIMARY KEY                   (parentFileID, childFileID)
) TYPE=MyISAM;

# FileLocations
DROP TABLE IF EXISTS FileLocations; CREATE TABLE FileLocations
(
  fileLocationID                BIGINT          NOT NULL      AUTO_INCREMENT,
  fileDataID                    BIGINT          NOT NULL,
  filePathID                    BIGINT          NOT NULL,
  storageTypeID                 MEDIUMINT       NOT NULL,
  createTime                    TIMESTAMP,
  insertTime                    TIMESTAMP       NOT NULL,
  owner                         VARCHAR(30),
  fsize                         BIGINT,
  storageSiteID                 SMALLINT      NOT NULL,
  protection                    VARCHAR(15),
  hostID                        BIGINT          NOT NULL DEFAULT 1,
  availability                  TINYINT         NOT NULL DEFAULT 1,
  persistent                    TINYINT         NOT NULL DEFAULT 0,
  sanity                        TINYINT         NOT NULL DEFAULT 1,
  PRIMARY KEY                   (fileLocationID),
  INDEX         FL_FilePathIndex                (filePathID),
  INDEX         FL_FileDataIndex                (fileDataID),
  INDEX         FL_StorageTypeIndex             (storageTypeID),
  INDEX         FL_StorageSiteIndex             (storageSiteID),
  INDEX         FL_HostIndex                    (hostID),
  UNIQUE        FL_FileLocationUnique           (fileDataID, storageTypeID, filePathID, storageSiteID, hostID)
) TYPE=MyISAM;

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<!DOCTYPE SCATALOG [
   <!ELEMENT SCATALOG (SITE*)>
       <!ATTLIST SCATALOG VERSION CDATA #REQUIRED>
   <!ELEMENT SITE (SERVER+)>
       <!ATTLIST SITE name (BNL | LBL) #REQUIRED>
       <!ATTLIST SITE description CDATA #IMPLIED>
       <!ATTLIST SITE URI CDATA #IMPLIED>
   <!ELEMENT SERVER (HOST+)>
       <!ATTLIST SERVER SCOPE (Master | Admin | User) #REQUIRED>
   <!ELEMENT HOST (ACCESS+)>
       <!ATTLIST HOST NAME CDATA #REQUIRED>
       <!ATTLIST HOST DBTYPE CDATA #IMPLIED>
       <!ATTLIST HOST DBNAME CDATA #REQUIRED>
       <!ATTLIST HOST PORT CDATA #IMPLIED>
   <!ELEMENT ACCESS EMPTY>
       <!ATTLIST ACCESS USER CDATA #IMPLIED>
       <!ATTLIST ACCESS PASS CDATA #IMPLIED>
]>



<SCATALOG VERSION="1.0.1">
        <SITE name="BNL">
                <SERVER SCOPE="Master">
                        <HOST NAME="mafata.wherever.net" DBNAME="Catalog_XXX" PORT="1234">
                                <ACCESS USER="Moi" PASS="HelloWorld"/>
                        </HOST>
                        <HOST NAME="mafata.wherever.net" DBNAME="Catalog_YYY" PORT="1235">
                                <ACCESS USER="Moi" PASS="HelloWorld"/>
                        </HOST>
                        <HOST NAME="duvall.star.bnl.gov" DBNAME="FileCatalog" PORT="">
                                <ACCESS USER="FC_master" PASS="AllAccess"/>
                        </HOST>
                </SERVER>
                <SERVER SCOPE="Admin">
                        <HOST NAME="duvall.star.bnl.gov" DBNAME="FileCatalog_BNL" PORT="">
                                <ACCESS USER="FC_admin" PASS="ExamplePassword"/>
                        </HOST>
                </SERVER>
                <SERVER SCOPE="User">
                        <HOST NAME="duvall.star.bnl.gov" DBNAME="FileCatalog_BNL" PORT="">
                                <ACCESS USER="FC_user" PASS="FCatalog"/>
                        </HOST>
                </SERVER>
        </SITE>
</SCATALOG>

Migration and notes from V01.265 to V01.275

This document is intended for FileCatalog managers only who have previously deployed an earlier version of API and older database table layout. It is NOT intended for users.

Reasoning for this upgrade and core of the upgrade

One of the major problem with the preceding database layout started to show itself when we reached 4 Million entries (for some reason, we seem to have magic numbers). A dire restriction was the presence of the field 'path' and 'nodename' in the FileLocations table. This table became unnecessarily large (of the order of GB) and sorting and queries would become slow and IO demanding (regardless of our careful indexing). The main action was to move both field to separate tables. This change requires a two step modification :

1. reshape of the database (leaving the old field), deployment of the database API in cross mode support
2. run the normalization scripts filling the new table and fields, deployment of the final API and drop of the obsolete columns (+ index rebuild)

The steps are more carefully described below ...

• Step by steps (general)
• Steps in Phase I
• Steps in Phase II
• Notes

Step by step migration instructions

Has to be made in several steps for safety a least interruption of service (although a pain to the manager). Note that you can do that much faster by cutting the Master/slave relationship, disabling all daemons auto-updating the database, proceed with table reshape and normalization script execution, drop and rebuild index, deploy the point-of-no-return API and restore Master/slave relation).

This upgrade is best if you have perl 5.8 or upper. Note that this transition will be the LAST one using perl 5.6 (get ready for a perl upgrade on your cluster).

We will assume you know how to connect to your database from an account able to manipulate and create any tables in the FileCatalog database.

Steps in Phase I

1. (0) Create the following tables

     DROP TABLE IF EXISTS FilePaths; CREATE TABLE FilePaths
     (
       filePathID                    BIGINT         NOT NULL         AUTO_INCREMENT,
       filePathName                  VARCHAR(255)   NOT NULL         UNIQUE,
       filePathCount                 INT,
       PRIMARY KEY                   (filePathID)
     ) TYPE=MyISAM;
   
     DROP TABLE IF EXISTS Hosts; CREATE TABLE Hosts 
    (
       hostID      smallint(6) NOT NULL auto_increment,
       hostName    varchar(30) NOT NULL default 'localhost',
       hostIDate   timestamp(14) NOT NULL,
       hostCreator varchar(15) NOT NULL default 'unknown',
       hostCount   int(11) default NULL,
       hostComment text,
       PRIMARY KEY (hostID),
       UNIQUE KEY  hostName (hostName)
     ) TYPE=MyISAM;
   
   

2. Modify some table and recreate one

        
        ALTER TABLE `FileLocations` ADD `filePathID` bigint(20) NOT NULL default '0' AFTER `fileDataID`;
        ALTER TABLE `FileLocations` ADD `hostID` bigint(20) NOT NULL default '1' AFTER `protection`;
        UPDATE TABLE `FileLocations` SET hostID=0;
   
        # note that I did that one from the Web interface (TBC)
        INSERT INTO Hosts VALUES(0,'localhost',NOW()+0,'',0,'Any unspecified node'); 
   
        ALTER TABLE `FileLocations` ADD INDEX ( `filePathID` )  
   
        ALTER TABLE `FilePaths` ADD `filePathIDate` TIMESTAMP NOT NULL AFTER `filePathName` ;
        ALTER TABLE `FilePaths` ADD `filePathCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `filePathIDate` ;
        ALTER TABLE `FilePaths` ADD `filePathComment` TEXT AFTER `filePathCount`;
   
        ALTER TABLE `StorageSites` ADD  `storageSiteIDate` TIMESTAMP NOT NULL AFTER `storageSiteLocation` ;
        ALTER TABLE `StorageSites` ADD  `storageSiteCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `storageSiteIDate` ;
        ALTER TABLE `StorageSites` DROP `storageComment`;
        ALTER TABLE `StorageSites` ADD  `storageSiteComment` TEXT AFTER `storageSiteCount`;
   
        ALTER TABLE `StorageTypes` ADD `storageTypeIDate` TIMESTAMP NOT NULL AFTER `storageTypeName` ;
        ALTER TABLE `StorageTypes` ADD `storageTypeCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `storageTypeIDate` ;
   
   
        ALTER TABLE `FileTypes` ADD `fileTypeIDate` TIMESTAMP NOT NULL AFTER `fileTypeExtension` ;
        ALTER TABLE `FileTypes` ADD `fileTypeCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `fileTypeIDate` ;
        ALTER TABLE `FileTypes` ADD `fileTypeComment` TEXT AFTER `fileTypeCount`;
   
   
        ALTER TABLE `TriggerSetups` ADD `triggerSetupIDate` TIMESTAMP NOT NULL AFTER `triggerSetupComposition` ;
        ALTER TABLE `TriggerSetups` ADD `triggerSetupCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `triggerSetupIDate`;
        ALTER TABLE `TriggerSetups` ADD `triggerSetupCount`   INT AFTER `triggerSetupCreator`;
        ALTER TABLE `TriggerSetups` ADD `triggerSetupComment` TEXT  AFTER `triggerSetupCount`;
   
        ALTER TABLE `EventGenerators` ADD `eventGeneratorIDate` TIMESTAMP NOT NULL AFTER `eventGeneratorParams` ;
        ALTER TABLE `EventGenerators` ADD `eventGeneratorCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `eventGeneratorIDate` ;
        ALTER TABLE `EventGenerators` ADD `eventGeneratorCount`   INT AFTER `eventGeneratorCreator`;
   
        ALTER TABLE `RunTypes` ADD `runTypeIDate` TIMESTAMP NOT NULL AFTER `runTypeName` ;
        ALTER TABLE `RunTypes` ADD `runTypeCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `runTypeIDate` ;
   
        ALTER TABLE `ProductionConditions` DROP `productionComments`; 
        ALTER TABLE `ProductionConditions` ADD  `productionConditionIDate`   TIMESTAMP NOT NULL AFTER `libraryVersion`;
        ALTER TABLE `ProductionConditions` ADD  `productionConditionCreator` CHAR( 15 ) DEFAULT 'unknown' NOT NULL AFTER `productionConditionIDate`;
        ALTER TABLE `ProductionConditions` ADD  `productionConditionComment` TEXT AFTER `productionConditionCount`;
   
   
   
        #
        # This table was not shaped as a dictionary so needs to be re-created
        # Hopefully, was not filled prior (but will be this year)
        #
        DROP TABLE IF EXISTS TriggerWords; CREATE TABLE TriggerWords
        (
           triggerWordID           MEDIUMINT       NOT NULL        AUTO_INCREMENT,
           triggerWordName         VARCHAR(50)     NOT NULL,
           triggerWordVersion      CHAR(6)         NOT NULL DEFAULT "V0.0",
           triggerWordBits         CHAR(6)         NOT NULL,  
           triggerWordIDate        TIMESTAMP       NOT NULL,
           triggerWordCreator      CHAR(15)        DEFAULT 'unknown' NOT NULL,
           triggerWordCount        INT,
           triggerWordComment      TEXT,
           UNIQUE   TW_TriggerCharacteristic (triggerWordName, triggerWordVersion, triggerWordBits),
           PRIMARY KEY             (triggerWordID)
        ) TYPE=MyISAM;

3. Deploy the new API CVS version 1.62 of FileCatalog.pm

4. Run the following utility scripts

   util/path_convert.pl
   util/host_convert.pl

   Note that those scripts use a new method $fC->connect_as("Admin"); which assumes that the Master Catalog will be accessed using the XML connection description. Also, it should be obvious that

   use lib "/WhereverYourModulAPIisInstalled"; should be replaced by the appropriate path for your site (or test area). Finally, it uses API CVS version 1.62 which supports Xpath and Xnode transitional keywords allowing us to transfer the information from one field to one table.

5. Check that Hosts table was filled properly and automatically with Creator/IDate
6. Paranoia step : Re-run the scripts mentioned 2 steps ago
   
   At this stage and ideally, nothing should happen (as you have already modified the records).
   A few tips prior from doing that
   • % fC_cleanup.pl -modif node=localhost -cond node='' -doit
     would hopefully do nothing but if you have messed something up in the past, hostName would be NULL and the above would be necessary.
   • After a full update, the following queries should return NOTHING
     % get_file_list.pl -keys flid -cond rfpid=0 -all -alls -as Admin
     % get_file_list.pl -keys flid -cond rhid=0 -all -alls -as Admin
     
     Those are equivalent to the SQL statements

     >SELECT FileLocations.fileLocationID FROM FileLocations WHERE FileLocations.filePathID = 0 LIMIT 0, 100
     >SELECT FileLocations.fileLocationID FROM FileLocations WHERE FileLocations.hostID = 0 LIMIT 0, 100 

   If it does return anything, contact me for further investigation and database repairs. As a side note, the -as keyword was introduced recently and you should update your get_file_list.pl script if not available.
7. Make a backup copy of the database for security (optional but safer) Backup can be done by easer a dump of mysql or more trivially, a cp -r of the database directory.
8. Leave it running for a few days (should be fine) for confidence consolidation ;-)

You are ready for phase II. Hang on tight now ...

Steps in Phase II

Those steps are no VERY intrusive and potentially destructive. Be careful from here on ...

1. Stop all daemons, be sure that during the rest of the operations, NO command attempts to manipulate the database. If you want to shield your users from the upgrade, stop all Master/slave relations.
2. Connect to the master FileCatalog as administrator for that database and execute the following SQL commands

     > ALTER TABLE `FileLocations` ADD INDEX FL_HostIndex (hostID);
     > ALTER TABLE `FileLocations` DROP INDEX `FL_FileLocationUnique`, ADD UNIQUE (fileDataID, storageTypeID, filePathID, storageSiteID, hostID);
   
     # drop the columns not in use anymore / should also get rid of the associated
     # indexes.
     > ALTER TABLE `FileLocations` DROP COLUMN nodeName;
     > ALTER TABLE `FileLocations` DROP COLUMN filePath;
   
     # "rename" index / was created with a name difference to avoid clash for transition
     # now renamed for consistency
     > ALTER TABLE `FileLocations` DROP INDEX `filePathID`, ADD INDEX  FL_FilePathIndex (filePathID);

3. OK, you should be done. Deploy either CVS version 1.63 which correspond to the FileCatalog API version V01.275 and above ... (by the way, get_file_list.pl -V gives the API version).

A few notes

• The new API is XML connection aware via a non-mandatory module named XML::Simple . You should install that module but there are some limitations if you are using perl 5.6 i.e., you MUST use the schema with ONLY one choice per category (Admin, Master or User).
• Your scripts will likely need to change if your Database Master and Slave are not on the same node (i.e. the administration account for the FileCatalog can be used only on the database Master and the regular user account on the Slave). There are a few forms of this such as the one below

  # Get connection fills the blanks while reading from XML
  # However, USER/PASSWORD presence are re-checked
  #$fC->debug_on();
  ($USER,$PASSWD,$PORT,$HOST,$DB) = $fC->get_connection("Admin");
  $port = $PORT if ( defined($PORT) );
  $host = $HOST if ( defined($HOST) );
  $db   = $DB   if ( defined($DB) );
  
  
  if ( defined($USER) ){   $user = $USER;}
  else {                   $user = "FC_admin";}
  
  if ( defined($PASSWD) ){ $passwd = $PASSWD;}
  else {                   print "Password for $user : ";
                           chomp($passwd = );}
  
  #
  # Now connect using a fully specified user/passwd/port/host/db
  #
  $fC->connect($user,$passwd,$port,$host,$db);

  or counting on the full definition in the XML file

  $fC    = FileCatalog->new();
  $fC->connect_as("Admin");

• Note a small future convenience when XML is ON. connect_as() does not only select as who you want to connect to but where as well. In fact, the proper syntax is intent=SITE::User (for example BNL::Admin is valid as well as LBL::User). This is only partly supported however.
• The new version of the API automatically add information in dictionary tables. Especially, the account under which a new dictionary value was inserted (Creator) and the insertion date (IDate) are filled automatically. A side effect being that the new API is NOT compatible with previous database table layout (no backward support will be attempted).

Migration and notes from V01.275 to V01.280

This document is intended for FileCatalog managers only who have previously deployed an earlier version of API and older database table layout. It is NOT intended for users.

Reasoning for this upgrade and core of the upgrade

This upgrade is a minor one, making support for two more detector sub-systems. The new API supports this modification. You need to alter the table DetectorConfigurations and add two columns. API are always forward compatible in that regard so it is completely safe to alter the table and deploy the API later.

ALTER TABLE `DetectorConfigurations` ADD dBSMD TINYINT;
ALTER TABLE `DetectorConfigurations` ADD dESMD TINYINT;
UPDATE `DetectorConfigurations` SET dBSMD=0;
UPDATE `DetectorConfigurations` SET dESMD=0;

And deploy the API V01.280 or later. You are done.

HPSS Performance study

Introduction

• Access pattern optimization (request ordering, ...)
• Optimization based on tape drive and technology capabilities
• Miscellaneous technology considerations
  (cards, interface, firmware and driver, RAID, disks, ...)
  
• HPSS disk cache optimizations
• COS and/or PVR optimization

Tape drive and technology capabilities

• the 994B drives
• the LTO-3 drives

Access pattern optimization (request ordering, ...)

A first simple and immediate consideration is to minimize tape mount and dismount operations, causing latencies and therefore performance drops. Since we use the DataCarousel for most restore operations, let's summarize its features.

The DataCarousel

The DataCarousel (DC) is an HPSS front end which main purpose is to coordinate requests from many un-correlated client's requests. Its main assumption is that all requests are asynchronous that is, you make a request from one client and it is satisfied “later” (as soon as possible). In other words, the DC aggregates all requests from all clients (many users could be considered as separate clients) and re-order them according policies, and possibly aggregating multiple requests for the same source into one request to the mass storage. The DC system itself is composed of a light client program (script), a plug-and-play policy based server architecture component (script) and a permanent process (compiled code) interfacing with the mass storage using HPSS API calls (this components is known as the “Oakridge Batch” although it current code content has little to do with the original idea from the Oakridge National Laboratory). Client and server interacts via a database component isolating client and server completely from each other (but sharing the same API , a perl module).

Policies may throttle the amount of data by group (quota, bandwidth percentage per user, etc ... i.e. queue request fairshare) but also perform tape access optimization such as grouping requests by tape ID (for equivalent share, all requests from the same tape are grouped together regardless of the time at which this request was performed or position in the request queue). The policy could be anything one can come up with based on the information either historical usage or current pending requests and characteristics of those requests (this could include file type, user, class of service, file size, ...). The DC then submits bundle of requests to the daemon component ; each request is a bundle of N file and known as a “job”. The DC submits K of those jobs before stopping and observing the mass storage behavior: if the jobs go through, more are submitted otherwise, either the server stops or proceed with a recovery procedure and consistency checks (as it will assume that no reaction and no unit of work being performed is a sign of MSS failure). In other words, the DC will also be error resilient and recover from intrinsic HPSS failures (being monitored). Whenever the files are moved from tape to cache in the MSS, a call back to the DC server is made and captive account connection is initiated to pull the file out of mass storage cache to more permanent storage.

Optimizations

While the policy is clearly a source of optimization (as far as the user is concerned), from a DataCarousel “post policy” perspective, N*K files are being requested at minimum at every point in time. In reality, more jobs are being submitted so the consumption of the “overflow”of job are used to monitor if the MSS is alive. The N*K files represents a total amount of files which should match the number of threads allowed by the daemon. The current setting are K=50, N=15 with an overflow allowed up to 25). The daemon itself has the possibility to treat requests simultaneously according to a“depth”. Those calls to HPSS are however only advisory. The depth is set at being 30 deep for the DST COS and 20 deep for the Raw COS. The deepest the request queue will be, more files will be requested simultaneously but this means that the daemon will also have to start more threads as previously noted. Those parameters have been showed to influence the performance to some extent (within 10%) with however a large impact on response time: the larger the request stack, the “less instantaneous” the response from a user's perspective (since the request queue length is longer).

The daemon has the ability of organizing X requests into a sorted list of tape ID and number of requests per tape. There are a few strategies allowing to alter the performance. We chose to enable “start with the tape with the largest number of files requested”. In addition, and since our queue depth is rather small comparing to the ideal number of files (K) per job, we order the files requested by the user by tape ID. Both optimizations are in place and lead to a 20% improvement within a realistic usage (bulk restore, Xrootd, other user activities).

Remaining optimizations

• Optimization based on tapeID would need to be better quantified (graph, average restore rate) for several class of files and usage. TBD.

• The tape ID program is a first implementation returning partial information. Especially, the MSS failures are not currently handled, leading to setting the tape ID to -1 (since there are now ways to recognize whether or not it is an error or a file missing in HPSS or even a file in the MSS MetaData server but located on a bad tape). Work in progress.

• The queue depth parameters should be studied and adjusted according to the K and N values. However, this would need to respect the machine / hardware capabilities. The beefier the machine would be, the better but this is likely a fine tuning. This needs to be done with great care as the hardware is also shared by multiple experiments. Ideally the compiled daemon should auto-adjust to the DC API settings (and respect command line parameters for queue depth). TBD.

• Currently, the daemon number of threads used for handling the HPSS API calls and to handle the call backs are sharing the same pool. This diminishes the number of threads available to communication with the Mass Storage and therefore, causes performance fluctuations (call back threads could get “stuck” or come in “waves” - we observed cosine behavior perhaps related to this issue). TBD.

Optimizations based on drive and technology capabilities

File size effects on IO performance

MSS failures and cascading effects

Miscellaneous technology considerations

HPSS disk cache optimizations

This section seems rather academic considering the previous sections improvement perspectives.

COS and PVR optimizations

In this section, we will discuss optimizing based on file size, perhaps isolated by PVR or COS. This will be possible in future run but would lead to a massive repackaging of files and data for the past years.

Appendix

Further reading:

• Data services tape use optimizations .
  
• Improving the Access Time of serpentine tape drives.
• Calculations for the sizing of tape performance .
• Ultrium LTO-2 tape drive testing .
• Performance to/from tape and recommended file size in Enstore .
  

HSI

• Kerberos (the preferred method)
• DCE keytab (using a keytab file generated for you by the HPSS system administrators)

• Familiar Unix-style command interface, with commands such as "LS", "CD", etc.
• Interactive, batch, or "one-liner" execution modes
• Ability to interactively pipe data into or out of HPSS, using filters such as "TAR"
• Recursive option is available for most commands; including the ability to copy an entire directory tree to or from HPSS with a single simple command
• Conditional put and get operations, including ability to update based on file timestamps
• Automatically uses HPSS parallel I/O features for file transfer operations
• Multi-threaded I/O within a single process space
• Command aliases and abbreviations
• 10 working directories
• Ability to read command input from a file, and write log or command output to a file.
• Non-DCE version runs on most major Unix-based platforms
• Non-DCE version provides the ability to connect to multiple HPSS systems and perform 3rd-party copies between the systems, using a "virtual drive" path notation.

HTAR

The following is the man page of how to use htar.

   
                     NAME
                          htar - HPSS tar utility
     
   
                     PURPOSE
                          Manipulates HPSS-resident tar-format archives.


                     SYNOPSIS
                          htar  -{c|t|x|X}  -f Archive [-?]  [-B] [-E]  [-L  inputlist] [-h]  [-m] [-o]
                                 [-d  debuglevel] [-p] [-v]  [-V] [-w]
                                 [-I  {IndexFile | .suffix}] [-Y  [Archive COS ID][:Index File COS ID]]
                                 [-S  Bufsize] [-T  Max Threads] [Filespec | Directory ...]

                     DESCRIPTION
                          htar  is a utility which manipulates HPSS-resident archives
                          by writing files to,  or retrieving files from the High
                          Performance Storage System (HPSS).  Files written to HPSS
                          are in the POSIX 1003.1 "tar" format, and may be retrieved
                          from HPSS, or read by native tar programs.

                          For those unfamiliar with HPSS, an introduction can be found
                          on the web at
                                 http://www.sdsc.edu/hpss

                          The local files used by the htar command are represented by
                          the Filespec parameter. If the Filespec parameter refers to
                          a directory, then that directory, and, recursively, all
                          files and directories within it, are referenced as well.

                          Unlike the standard Unix "tar" command, there is no default
                          archive device; the "-f Archive" flag is required.

                     Archive and Member files
                          Throughout the htar documentation, the term "archive file"
                          is used to refer to the tar-format  file, which is named by
                          the "-f filename" command line option. The term "member
                          file" is used to refer to individual files contained within
                          the archive file.

                     WHY USE HTAR
                          htar has been optimized for creation of archive files
                          directly in HPSS, without having to go through the
                          intermediate step of first creating the archive file on
                          local disk storage, and then copying the archive file to
                          HPSS via some other process such as ftp or hsi. The program
                          uses multiple threads and a sophisticated buffering scheme
                          in order to package member files into in-memory buffers,
                          while making use of the high-speed network striping
                          capabilities of HPSS.

                          In most cases, it will be signficantly  faster to use htar
                          to create a tar file in HPSS than to either create a local
                          tar file and then copy it to HPSS, or to use tar piped into
                          ftp (or hsi) to create the tar file directly in HPSS.

                          In addition, htar creates a separate index file, (see next
                          section) which contains the names and locations of all of
                          the  member files in the archive (tar) file.  Individual
                          files and directories in the archive can be randomly
                          retrieved without having to read through the archive file.
                          Because the index file is usually smaller than the archive
                          file, it is possible that the index file may reside in HPSS
                          disk cache  even though the archive file has been moved
                          offline to tape; since htar uses the index file for listing
                          operations, it may be possible to list the contents of the
                          archive file without having to incur the time delays of
                          reading the archive file back onto disk cache from tape.

                          It is also possible to create an index file for a tar file
                          that was not originally created by htar.

                     HTAR Index File
                          As part of the process of creating an archive file on HPSS,
                          htar also creates an index file, which is a directory of the
                          files contained in the archive. The Index File includes the
                          position of member files within the archive, so that files
                          and/or directories can be randomly retrieved from the
                          archive without having to read through it sequentially.  The
                          index file is usually significantly smaller in size than the
                          archive file, and may often reside in HPSS disk cache even
                          though the archive file resides on tape. All htar operations
                          make use of an index file.

                          It is also possible to create an index file for an archive
                          file that was not created by htar, by using the "Build
                          Index" [-X] function (see below).

                          By default, the index filename is created by adding ".idx"
                          as a suffix to the Archive name specified by the -f
                          parameter.  A different suffix or index filename may be
                          specified by the "-I " option, as described below.

                          By default, the Index File is assumed to reside in the same
                          directory as the Archive File.  This can be changed by
                          specifying a relative or absolute pathname via the -I
                          option. The Index file's relative pathname is relative to
                          the Archive File directory unless an absolute pathname is
                          specified.

                     HTAR Consistency File
                          HTAR writes an extra file as the last member file of each
                          Archive, with a name similar to:

                                  /tmp/HTAR_CF_CHK_64474_982644481

                          This file is used to verify the consistency of the Archive
                          File and the Index File.  Unless the file is explicitly
                          specified, HTAR does not extract this file from the Archive
                          when the -x action is selected.  The file is listed,
                          however, when the -t action is selected.

                     Tar File Restrictions
                          When specifying path names that are greater than 100
                          characters for a file (POSIX 1003.1 USTAR) format, remember
                          that the path name is composed of a prefix bufferFR, a /
                          (slash), and a name buffer.

                          The prefix buffer can be a maximum of 155 bytes and the name
                          buffer can hold a maximum of 100 bytes. Since some
                          implementations of TAR require the prefix and name buffers
                          to terminate with a null (' ') character, htar enforces the
                          restriction that the effective prefix buffer length is 154
                          characters (+ trailing zero byte), and the name buffer
                          length is 99 bytes (+ trailing zero byte). If the path name
                          cannot be split into these two parts by a slash, it cannot
                          be archived. This limitation is due to the structure of the
                          tar archive headers, and must be maintained for compliance
                          with standards and backwards compatibility. In addition, the
                          length of a destination for a hard or symbolic link ( the
                          'link name') cannot exceed 100 bytes (99 characters + zero-
                          byte terminator).

                     HPSS Default Directories
                          The default directory for the Archive file is the HPSS home
                          directory for the DCE user.  An absolute or relative HPSS
                          path can optionally be specified for either the Archive file
                          or the Index file. By default, the Index file is created in
                          the same HPSS directory as the Archive file.

                     Use of Absolute Pathnames
                          Although htar does not restrict the use of absolute
                          pathnames (pathnames that begin with a leading "/") when the
                          archive is created, it will remove the leading / when files
                          are extracted from the archive.  All extracted files use
                          pathnames that are relative to the current working
                          directory.

                     HTAR USAGE
                          Two groups of flags exist for the htar command; "action"
                          flags and "optional" flags. Action flags specify the
                          operation to be performed by the htar command, and are
                          specified by one of the following:

                          -c, -t, -x, -X

                          One action flag must be selected in order for the htar
                          command to perform any useful function.

                     File specification (Filespec)
                          A file specification has one of the following forms:

                                  WildcardPath
                                     or
                                  Pathname
                                     or
                                  Filename

                          WildcardPath is a path specification that includes standard
                          filename pattern-matching characters, as specified for the
                          shell that is being used to invoke htar.  The pattern-
                          matching characters are expanded by the shell and passed to
                          htar as command line arguments.

                     Action Flags
                          Action flags defined for htar are as follows:

                          -c   Creates a new HPSS-resident archive, and writes the
                               local files specified by one or more File parameters
                               into the archive. Warning: any pre-existing archive file
                               will be overwritten without prompting. This behavior
                               mimics that of the AIX tar utility.

                          -t   Lists the files in the order in which they appear in
                               the HPSS- resident archive.   Listable output is
                               written to standard output; all other output is written
                               to standard error.

                          -x   Extracts the files specified by one or more File
                               parameters from the HPSS-resident archive. If the File
                               parameter refers to a directory, the htar command
                               recursively extracts that directory and all of its
                               subdirectories from the archive.

                               If the File parameter is not specified, htar extracts
                               all of the files from the archive. If an archive
                               contains  multiple copies of the same file, the last
                               copy extracted overwrites  all previously extracted
                               copies. If the file being extracted does not already
                               exist on the system, it is created. If you have the
                               proper permissions, then htar command restores all
                               files and directories with the same owner and group IDs
                               as they have on the HPSS tar file. If you  do not have
                               the proper permissions, then files and directories are
                               restored with your owner and group IDs.

                          -X   builds a new index file by reading the entire tar file.
                               This operation is used either to reconstruct an index
                               for tar files whose Index File is unavailable (e.g.,
                               accidentally deleted), or for tar files that were not
                               originally created by htar.

                     Options
                          -?   Displays htar's verbose help

                          -B   Displays block numbers as part of the listing (-t
                               option). This is normally used only for debugging.

                          -d debuglevel
                               Sets debug level (0 - N) for htar. 0 disables debug, 1
                               - n enable progressively higher levels of debug output.
                               5 is the highest level; anything > 5 is silently mapped
                               to 5.  0 is the default debug level.

                          -E   If present, specifies that a local file should be used
                               for the file specified by the "-f Archive" option.  If
                               not specified, then the archive file will reside in
                               HPSS.

                          -f Archive
                               Uses Archive as the name of archive to be read or
                               written. Note: This is a required parameter for htar,
                               unlike the standard tar utility, which uses a built-in
                               default name.

                               If the Archive variable specified is - (minus sign),
                               the tar command writes to standard output or reads from
                               standard input. If you write to standard output, the -I
                               option is mandatory, in order to specify an Index File,
                               which is copied to HPSS if the Archive file is
                               successfully written to standard output. [Note: this
                               behavior is deferred - reading from or writing to pipes
                               is not supported in the initial version of htar].

                          -h   Forces the htar command to follow symbolic links as if
                               they were normal files or directories. Normally, the
                               tar command does not follow symbolic links.

                          -I index_name
                               Specifies the index file name or suffix.  If the first
                               character of the index_name is a period, then
                               index_name is appended to the Archive name, e.g. "-f
                               the_htar -I .xdnx" would create an index file called
                               "the_htar.xndx".  If the first character is not a
                               period, then index_name is treated as a relative
                               pathname for the index file (relative to the Archive
                               file directory) if the pathname does not start with
                               "/", or an absolute pathname otherwise.

                               The default directory for the Index file is the same as
                               for the Archive file.  If a relative Index file
                               pathname is specifed, then it is appended to the
                               directory path for the Archive file.  For example, if
                               the Archive file resides in HPSS in the directory
                               "projects/prj/files.tar", then an Index file
                               specification of "-I projects/prj/files.old.idx" would
                               fail, because htar would look for the file in the
                               directory "projects/prj/projects/prj".  The correct
                               specification in this case is "-I files.old.idx".

                          -L InputList
                               Writes the files and directories listed in the
                               "InputList" file to the archive. Directories named in
                               the InputList file are not treated recursively. For
                               directory names contained in the InputList file, the
                               tar command writes only the directory entry to the
                               archive, not the files and subdirectories rooted in the
                               directory.  Note that "home directory" notation ("~")
                               is not expanded for pathnames contained in the
                               InputList file, nor are wildcard characters, such as
                               "*" and "?".

                          -m   Uses the time of extraction as the modification time.
                               The default is to preserve the modification time of the
                               files. Note that the modification time of directories
                               is not guaranteed to be preserved, since the operating
                               system may change the timestamp as the directory
                               contents are changed by extracting other files and/or
                               directories.  htar will explicitly set the timestamp on
                               directories that it extracts from the Archive, but not
                               on intermediate directories that are created during the
                               process of extracting files.

                          -o   Provides backwards compatibility with older versions
                               (non-AIX) of the tar command. When this flag is used
                               for reading, it causes the extracted file to take on
                               the User and Group ID (UID and GID) of the user running
                               the program, rather than those on the archive.  This is
                               the default behavior for the ordinary user. If htar is
                               being run as root, use of this option causes files to
                               be owned by root rather than the original user.

                          -p   Says to restore fields to their original modes,
                               ignoring the present umask. The setuid, setgid, and
                               tacky bit permissions are also restored to the user
                               with root user authority.

                          -S bufsize
                               Specifies the buffer size to use when reading or
                               writing the HPSS tar file.  The buffer size can be
                               specified as a value, or as kilobytes by appending any
                               of  "k","K","kb", or "KB" to the value.  It can also be
                               specified as megabytes by appending any of  "m" or "M"
                               or "mb" or "MB" to the value, for example, 23mb.

                          -T max_threads
                               Specifies the maximum number of threads to use when
                               copying local member files to the Archive file.  The
                               default is defined when htar is built; the release
                               value is 20.  The maximum number of threads actually
                               used is dependent upon the local file sizes, and the
                               size of the I/O buffers.  A good approximation is
                               usually

                                  buffer size/average file size

                               If the -v or -V option is specified, then the maximum
                               number of local file threads  used while writing the
                               Archive file to HPSS is displayed when the transfer is
                               complete.

                          -V   "Slightly verbose" mode. If selected, file transfer
                               progress will be displayed in interactive mode. This
                               option should normally not be selected if verbose (-v)
                               mode is enabled, as the outputs for the two different
                               options are generated by separate threads, and may be
                               intermixed on the output.

                          -v   "Verbose" mode. For each file processed, displays a
                               one-character operation flag, and lists the name of
                               each file. The flag values displayed are:
                                   "a"  - file was added to the archive
                                   "x"  - file was extracted from the archive
                                   "i"  - index file entry was created (Build Index
                               operation)

                          -w   Displays the action to be taken, followed by the file
                               name, and then waits for user confirmation. If the
                               response is affirmative, the action is performed. If
                               the response is not affirmative, the file is ignored.

                          -Y auto | [Archive CosID][:IndexCosID]
                               Specifies the HPSS Class of Service ID to use when
                               creating a new Archive and/or Index file. If the
                               keyword auto is specified, then the HPSS hints
                               mechanism is used to select the archive COS, based upon
                               the file size.  If -Y cosID  is specified, then cosID
                               is the numeric COS ID to be used for the Archive File.

                               If -Y :IndexCosID is specified, then IndexCosID is the
                               numeric COS ID to be  used for the Index File.  If both
                               COS IDs are specified, the entire parameter must be
                               specified as a single string with no embedded spaces,
                               e.g. "-Y 40:30".

                     HTAR Memory Restrictions
                          When writing to an HPSS archive, the htar command uses a
                          temporary file (normally in /tmp) and maintains in memory a
                          table of files; you receive an error message if htar cannot
                          create the temporary file, or if there is not enough memory
                          available to hold the internal tables.

                     HTAR Environment
                          HTAR should be compiled and run within a non-DCE HPSS environment.

                     Miscellaneous Notes:
                          1. The maximum size of a single Member file within the
                          Archive is approximately 8 GB, due to restrictions in the
                          format of the tar header.  HTAR does not impose any
                          restriction on the total size of the Archive File when it is
                          written to HPSS; however, space quotas or other system
                          restrictions may limit the size of the Archive File when it
                          is written to a local file (-E option).

                          2.  HTAR will optionally write to a local file; however, it
                          will not write to any file type except "regular files".  In
                          particular, it is not suitable for writing to magnetic tape.
                          To write to a magnetic tape device, use the "tar" or "cpio"
                          utility.

                     Exit Status
                          This command returns the following exit values:

                          0       Successful completion.

                          >0      An error occurred.

                     Examples
                          1.   To write the file1 and file2 files to a new archive
                               called "files.tar" in the current HPSS home directory,
                               enter:

                                      htar -cf files.tar file1 file2

                          2.   To extract all files from the project1/src directory in
                               the Archive file called proj1.tar, and use the time of
                               extraction as the modification time,  enter:

                                     htar -xm -f proj1.tar project1/src

                          3.   To display the names of the files in the out.tar
                               archive file within the HPSS home directory, enter:

                                     htar -tvf out.tar

                     Related Information
                          For file archivers: the cat command, dd command, pax
                          command.  For HPSS file transfer programs: pftp, nft, hsi

                          File Systems Overview for System Management in AIX Version 4
                          System Management Guide: Operating System and Devices
                          explains file system types, management, structure, and
                          maintenance.

                          Directory Overview in AIX Version 4 Files Reference explains
                          working with directories and path names.

                          Files Overview in AIX Version 4 System User's Guide:
                          Operating System and Devices provides information on working
                          with files.

                          HPSS web site at http://www.sdsc.edu/hpss

                     Bugs and Limitations:
                          - There is no way to specify relative Index file pathnames
                          that are not rooted in the Archive file directory without
                          specifying an absolute path.

                          - The initial implementation of HTAR does not provide the
                          ability to append, update or remove files.  These features,
                          and others, are planned enhancements for future versions.

Group login scripts

Installing

The STAR general group login scripts are necessary to define the STAR environment. They reside in $CVSROOT within the group/ sub-tree. Template files for users .cshrc and .login support also exists within this tree in a sub-directory group/templates. To install properly on a local cluster, there are two possibilities:

• if you have access to AFS, you should simply
  

      % mkdir  /usr/local/star # this is only an example
      % cd /usr/local/star     # this directory needs to be readable by a STAR group
      % cvs checkout group     # this assumes CVSROOT is defined 

  This will bring a copy of all you need locally in /usr/local/star/group
• If you do not have access to AFS from your remote site, get a copy of the entire BNL $GROUP_DIR tree and unpack in a common place (like /usr/local/star above). A copy resides in the AFS tree mentioned Additional software components.

Note that wherever you install the login scripts, they need to be readable by a STAR members (you can do this by allowing a Unix group all STAR users will belong to read access to the tree or by making sure the scripts are all users accessible).

Also, as soon as you get a local copy of the group/templates/ files, EDIT BOTH the cshrc and login files and change on top the definition of GROUP_DIR to it matches your site GROUP script location (/usr/local/group in our example).

To enable a user to use the STAR environment, simply copy the template cshrc and login scripts as indicated in Setting up your computing environment.

Special scripts

Part of our login is optional and the scripts mentioned here will NOT be part of our CVS repository but, if exists, will be executed.

• site_pre_setup.csh - this script, if exists in $GROUP_DIR, will be executed before the execution of the STAR standard login. Its purpose is to define variables indicating non-standard location for your packages. For those variables which may be redefined, please consult Setting up your computing environment for all the variables (in blue) which may be redefined prior to login.
• site_post_setup.csh - this script, if exists in $GROUP_DIR, will be executed after the STAR standard login. Its purpose is to define local variables nor related to STAR's environment. Such variables may be for example the definition of a proxy (http_proxy, ftp_proxy, https_proxy), a NNTP server or a default WWW home directory (WW_HOME). Do not try to redefine STAR login's defined variables using this script.

Testing this phase

Testing this phase is as simple as creating a test account and verifying that the login does succeed. Whenever you start with a blank site, the login MUST succeed and lead to viable environment ($PATH especially should be minimally correct). A typical login example would be at this stage something like

Setting up WWW_HOME  = http://www.star.bnl.gov/

         ----- STAR Group Login from /usr/local/star/group/ -----

Setting up STAR_ROOT = /usr/local/star
Setting up STAR_PATH = /usr/local/star/packages
Setting up OPTSTAR   = /usr/local/star/opt/star
WARNING : XOPTSTAR points to /dev/null (no AFS area for it)
Setting up STAF      = /usr/local/star/packages/StAF/pro
Setting up STAF_LIB  = /usr/local/star/packages/StAF/pro/.cos46_gcc346/lib
Setting up STAF_BIN  = /usr/local/star/packages/StAF/pro/.cos46_gcc346/bin
Setting up STAR      = /usr/local/star/packages/pro
Setting up STAR_LIB  = /usr/local/star/packages/pro/.cos46_gcc346/lib
Setting up STAR_BIN  = /usr/local/star/packages/pro/.cos46_gcc346/bin
Setting up STAR_PAMS = /usr/local/star/packages/pro/pams
Setting up STAR_DATA = /usr/local/star/data
Setting up CVSROOT   = /usr/local/star/packages/repository
Setting up ROOT_LEVEL= 5.12.00
Setting up SCRATCH   = /tmp/jeromel
CERNLIB version pro has been initiated with CERN_ROOT=/cernlib/pro
STAR setup on star.phys.pusan.ac.kr by Tue Mar 12 06:43:47 KST 2002  has been completed
LD_LIBRARY_PATH = .cos46_gcc346/lib:/usr/local/star/ROOT/5.12.00/.cos46_gcc346/rootdeb/lib:ROOT:/usr/lib/qt-3.3/lib

Suggestions

STAR group

You may want to to create a rhstar group on your local cluster matching GID 31012. This will make AFS integration easier as the group names in AFS will then translate to rhstar (it will however not grant you any special access obviously since AFS is Kerberos authentication based and not Unix UID based).
To do this, and after checking that /etc/group do not contain any mapping for gid 31012, you could (Linux):

% groupadd -g 31012 rhstar

Test account

It may be practical for testing the STAR environment to create a test account on your local cluster. The starlib account is an account  used in STAR for software installation. You may want to create such account as follow (Linux):

% useradd -d /home/starlib -g rhstar -s /bin/tcsh  starlib

 This will allow for easier integration. Any account name will do (but testing is important and we will have a section on this later).

Additional software components

Scope & audience

Described in Setting up your computing environment, OPTSTAR is the environment variable pointing to an area which will supplement the operating system installation of libraries and program. This area is fundamental to the STAR software installation as it will contain needed libraries, approved software component version, shared files, configuration and so on.

The following path should contain all software components as sources for you to install a fresh copy on your cluster:
    /afs/rhic.bnl.gov/star/common

Note that this path should allow anyuser to read so there is no need for an AFS token. The note below are sparse and ONLY indicate special instructions you need to follow if any. In the absence of special instructions, the "standard" instructions are to be followed. None of the explanations below are aimed to target a regular user but aimed to target system administrator or software infrastructure personnel.

System wide RPMs

Some RPMs from your OS distribution may be found at BNL under the directory /afs/rhic.bnl.gov/rcfsl/X.Y/*/ where X.Y is the major and minor version for your Scientific Linux version respectively. You should have a look and install. If you do not have AFS, you should log to the RCF and transfer whatever is appropriate.

In other words, we may have re-packaged some packages and/or created additional ones for compatibility purposes. An example of this for SL5.3 is flex32libs-2.5.4a-41.fc6.i386.rpm located in /afs/rhic.bnl.gov/rcfsl/5.3/rcf-custom/ which supports the 32 bits compatbility package for flex on a kernel with dual 32/64 bits support.

STAR Specific

The directory tree /afs/rhic.bnl.gov/star/common contains packages installed on our farm in addition of the default distribution software packages coming with the operating system. At BNL, all packages referred here are installed in the AFS tree

	/opt/star -> /afs/rhic.bnl.gov/@sys/opt/star/

Be aware of the intent explained in Setting up your computing environment as per the difference between $XOPTSTAR and OPTSTAR.

OPTSTAR will either

• at BNL or to a remote site: be used to indicate and access the local software BUT may be supported through a soft-link to the same AFS area as showed above whereas @sys will expand to the operating system of interest (see Setting up your computing environment as well for a support matrix)
• at a remote site, will point to a LOCAL (that is, non-networked) installation of the software components. This space could be anywhere on your local cluster but obviously, will have to be shared and visible from all nodes in your cluster.

XOPTSTAR

The emergence of $XOPTSTAR started from 2003 to provide better support for software installation to remote institutions. Many packages add path information to their configuration (like the infamous .la files) and previously installed in $OPTSTAR, remote sites had problems loading libraries for a path reason. Hence, and unless specified otherwise below, $XOPTSTAR will be used preferably at BNL for installation the software so remote access to the AFS repository and copy will be made maximally transparent.

In 2005, we added an additional tree level reflecting the possibility of multiple compilers and possible mismatch between fs sysname setups and operating system versions. Hence, you may see path like OPTSTAR=/opt/star/sl44_gcc346 but this structure is a detail and if the additional layer does not exist for your site, later login will nonetheless succeed. This additional level is defined by the STAR login environment $STAR_HOST_SYS. In the next section, we explained how to set this up from a "blank" site (i.e. a site without the STAR environment and software installed).

On remote sites where you decide to install the software components locally, you should use $OPTSTAR in the configure or make statements.

Basic starting point

From a blank node on remote site, be sure to have $OPTSTAR defined. You can do this by hand for example like this

% setenv OPTSTAR /usr/local

or

% mkdir -p /opt/star
% setenv OPTSTAR /opt/star

are two possibilities. The second, being the default location of the software layer, will be automatically recognized by the STAR group login scripts. From this point, a few pre-requisites are

• you have to have a system with "a" compiler - we support gcc but also icc on Linux
• you should have the STAR group login scripts at hand (it could be from AFS). The STAR login scripts will NOT redefine $OPTSTAR if already defined.

Execute the STAR login. This will define $STAR_HOST_SYS appropriately. Then

% cd $OPTSTAR
% mkdir $STAR_HOST_SYS
% stardev
 

the definition of $OPTSTAR will change to the version dependent structure, adding $STAR_HOST_SYS to the path definition (the simple presence of the layer made the login script redefine it).

Changing platform or compiler

32 bits versus 64 bits

If you want to support native 64 bits on 64 bits, do not forget to pass/force -m64 -fPIC to the compiler and -m64 to the linker. If however you want to build a cross platform (64 bit/32 bit kernels compatible) executables and libraries, you will on the contrary need to force -m32 (use -fPIC). Even if you build the packages from a 32 bit kernel node, be aware that many applications and package save a setup including compilation flags (which will have to be using -m32 if you want a cross platform package). There are many places below were I do not specify this.

Often, using CFLAGS="-m32 -fPIC" CXXFLAGS="-m32 -fPIC" LDFLAGS="-m32" would do the trick for a force to 32 bits mode (similarly for -m64). You need to use such option for libraries and packages even if you assemble on a 32 bits kernel node as otherwise, the package may build extension later not compatible as cross-platform support.

Other GCC versions

As for the 32 bits versus 64 bits, often adding something like CC=`which gcc` and CXX=`which g++` to either the configure or make command would do the trick. If not, you will need to modify the Makefile accordingly. You may also define the environment variable CC for consistency.

Summary

If ylu do have a 64 bits kernel and intend to compile both 32 bits and 64 bits, you should define the envrionment variable as shown below.  The variables will make configure (and some Makefile) pick the proper flags and make your life much easier - follow the specific instructions for the packages noted in those instructions for specific tricks. Note as well that even if you do have a 32 bits kernel only, you are encouraged to use the -m32 compilation option (this will make further integration with dual 32/64 bits support smoother as some of the packages configurations include compiler path and options).

32 bits

% setenv CFLAGS   "-m32 -fPIC"
% setenv CXXFLAGS "-m32 -fPIC"
% setenv FFLAGS   "-m32 -fPIC"
% setenv FCFLAGS  "-m32 -fPIC"
% setenv LDFLAGS  "-m32"
% setenv CC  `which gcc`     # only if you use a different compiler than the system default
% setenv CXX `which g++`     # only if you use a different compiler than the system default

and/or pass to Makefile and/or configure the arguments CFLAGS="-m32 -fPIC" CXXFLAGS="-m32 -fPIC" LDFLAGS="-m32" CC=`which gcc` CXX=`which g++` (will not hurt to use it in addition of the environment variables)

64 bits

% setenv CFLAGS   "-m64 -fPIC"
% setenv CXXFLAGS "-m64 -fPIC"
% setenv FFLAGS   "-m64 -fPIC"
% setenv FCFLAGS  "-m64 -fPIC"
% setenv LDFLAGS  "-m64"
% setenv CC  `which gcc`     # only if you use a different compiler than the system default
% setenv CXX `which g++`     # only if you use a different compiler than the system default

and/or pass to Makefile and/or configure the arguments CFLAGS="-m64 -fPIC" CXXFLAGS="-m64 -fPIC" LDFLAGS="-m64" CC=`which gcc` CXX=`which g++` (will not hurt to use it in addition of the environment variables)

Software repository directory - starting a build

In the instructions below, greyed instructions are historical instructions and/or package version which no longer reflects the current official STAR supported platform. However, if you try to install the STAR software under older OS, refer carefully to the instructions and package versions.

perl

The STAR envrionment and login scripts heavily rely on perl for string manipulation, compilation management and a bunch of utility scripts. Assembling it from the start is essential. You may rely on your system-wide installed perl version BUT if so, note that the minimum version indicated below IS required.

In our software repository path, you will find a perl/ sub-directory containing all packages and modules.

The package and minimal version are below
		perl-5.6.1.tar.gz      -- Moved to 5.8.0 starting from RH8
		perl-5.8.0.tar.gz      -- Solaris and True64 upgraded 2003
		perl-5.8.4.tar.gz      -- 2004, Scientific Linux
		perl-5.8.9.tar.gz      -- SL5+
                perl-5.10.1.tar.gz     -- SL6+

When building perl

• Use all default arguments BUT when you are asked for compilation / linker args, add -m32 or -m64 depending on the platform support you are building. Those questions are (example for the 32 bits version):
  • Any additional cc flags? []  -fPIC -m32
  • Any additional ld flags (NOT including libraries)? [] -m32
  • Any special flags to pass to cc -c to compile shared library modules? []  -fPIC -m32
  • Any special flags to pass to cc to create a dynamically loaded library? [-shared -O2] -shared -O2 -m32
  • If you build a 32 bits support on a 64 bit node, you may also answer "no" below but the defalt naswer SHOULD appear as no if you properly passed -m32 as indicated above.
    Try to use maximal 64-bit support, if available? [y] n  <--- you probably did ot pass -m32
    Try to use maximal 64-bit support, if available? [n]    <--- just press return, all is fine
• when asked for the default prefix for the installation, give the value of $OPTSTAR as answer (or a base path starting with the value of $OPTSTAR wherever appropriate). Questions include
  
  • Installation prefix to use? (~name ok) [/usr/local]
• If the build warn you at first that the directory does not exists but proceed - to questions like "Use that name anyway?" answer Yes

After installing perl itself, you will need to install the STAR required module.

The modules are installed using a bundle script (install_perlmods located in /afs/rhic.bnl.gov/star/common/bin/). It needs some work to get it generalized but the idea is that it contains the dependencies and installation order . To install, you can do the following (we assume install_perlmods is in the path for simplicity and clarity):
 

1. first chose a work place where you would unpack the needed modules. Let's assume this is /home/xxx/myworkplace
2. Check things out by running install_perlmods with arguments 0 as follow
   % install_perlmods 0 /home/xxx/myworkplace
   It will tell you the list of modules you need to unpack. If they are already unpacked and /home/xxx/myworkplace contains all needed package directories, skip to step 4.
3. You can unpack manually OR use the command
   % install_perlmods 1 /home/xxx/myworkplace
   to do this automatically. Note that you could have skipped step 2 and do that from the start (if confident enough).
4. The steps above should have created a file named /home/xxx/myworkplace/perlm-install-XXX.csh where XXX is the OS you are working on. Note that the same install directory may therefore be used for ALL platform on your cluster. However, versionning is not (yet) supported.
   Execute this script after checking its content. It will run (hopefully smoothly) the perl Makerfile.PL and make / make install commands. Note that you could have also used
   % install_perlmods 2 /home/xxx/myworkplace
   and skip step 2 & 3. In this mode, it unpacks and proceeds with compilation. To do only if you have absolute blind faith in the process (I don't and have written those scripts ;-)   ).

Very old note [this used to happen with older perl version]: if typing make, you get the following message

make: *** No rule to make target `<command-line>', needed by `miniperlmain.o'.  Stop.

then you bumped into an old gcc/perl build issue (tending to come back periodically depending on message formats of gcc) and can resolve this by a using any perl version available and running the commands:

% make depend
% perl -i~ -nle 'print unless /<(built-in|command.line)>/' makefile x2p/makefile

This will suppress from the makefile the offending lines and will get you back on your feet.
 

After you install perl, and your setup is local (in /opt/star) you may want to do the following

% cd /opt/star
% ln -s $STAR_HOST_SYS/* .
%
% # prepare for later directories packages will create
% ln -s $STAR_HOST_SYS/share .
% ln -s $STAR_HOST_SYS/include .
% ln -s $STAR_HOST_SYS/info .
% ln -s $STAR_HOST_SYS/etc .
% ln -s $STAR_HOST_SYS/libexec .
% ln -s $STAR_HOST_SYS/qt .
% ln -s $STAR_HOST_SYS/jed .
%

While some of those directories will not yet exist, this will create a base set of directories (without the additional compiler / OS version) supporting future upgrades via the "default" set of directories. In other words, any future upgrade of compilers for example leading to a different  $STAR_HOST_SYS will still lead as well to a functional environment as far as compatibility exists. Whenever compatibility will be broken, you will need of course to re-create a new $STAR_HOST_SYS tree.
At this stage, you should install as much of the libraries in $OPTSTAR and re-address the perl modules later as some depends on installed libraries necessary for the STAR environment to be functional.

Others/ [PLEASE READ, SOME PACKAGE MAY HAVE EXCEPTION NOTES]

        Needed on Other platform (but there on Linux). Unless specified 
        otherwise, the packages were build with the default values.
                make-3.80
                tar-1.13
                flex-2.5.4   
                xpm-3.4k
                libpng-1.0.9

                mysql-3.23.43 on Solaris
                mysql-3.23.55 starting from True64 days (should be tried as
                              an upgraded version of teh API)
                              BEWARE mysql-4.0.17 was tried and is flawed.
                              We also use native distribution MySQL
                mysql-4.1.22  *** IMPORTANT *** Actually this was an upgrade 
                              on SL4.4 (not necessary but the default 4.1.20 
                              has some bugs) 

                <gcc-2.95.2>
                dejagnu-1.4.1	 
                gdb-5.2
                texinfo-4.3
                emacs-20.7 

                findutils-4.1
                fileutils-4.1
                cvs-1.11       -- perl is needed before hand as it folds
                               it in generated scripts
                grep-2.5.1a    Started on Solaris 5.9 in 2005 as ROOT would complain 
                               about too old version of egrep 


This may be needed if not installed on your system. It is part of a needed
autoconf/automake deployment.
                m4-1.4.1		
                autoconf-2.53  
                automake-1.6.3
		
Linux only
                valgrind-2.2.0
                valgrind-3.2.3                (was for SL 4.4 until 2009)
                valgrind-3.4.1                SL4.4
         
General/   The installed packages/sources for diverse 
           software layers. The order of installation was
                ImageMagick-5.4.3-9           On RedHat 8+, not needed for SL/RHE but see below
                ImageMagick-6.5.3-10          Used on SL5 as default revision is "old" (6.2.8) - TBC

                slang-1.4.5                   On RedHat 8+, ATTENTION: not needed for SL/RHE, install RPM
                lynx2-8-2 
                lynx2-8-5                     Starting from SL/RHE
                xv-3.10a-STAR                 Note the post-fix STAR (includes patch and 32/64 bits 
                                              support Makefile)
                nedit-5.2-src                 ATTENTION: No need on SL/RHE (installed by default) [+]
                pythia5
                pythia6
                text2c
                icalc

                dejagnu-1.4.1                 Optional / Dropped from SL3.0.5
                gdb-5.1.91                    For RH lower versions - Not RedHat , 8+
                gdb-6.2 (patched)             Done for SL3 only (do not install on others) 

                gsl-1.13                      Started from SL5 and back ported to SL4
                gsl-1.16                      Update for SL6                 

                chtext
                jed-0.99-16
                jed-0.99.18                   Used from SL5+
                jed-0.99.19                   Used in SL6/gcc 4.8.2 (no change in instructions) 
		
                qt-x11-free-3.1.0
                qt-x11-free-3.3.1             Starting with SL/RHE [+]
                qt-x11-opensource-4.4.3       Deployed from i386_sl4 and i386_sl305 (after dropping SL3.0.2), SL5
                qt-everywhere-opensource-src-4.8.5       
                                              Deployed from SL6 onward
                qt-everywhere-opensource-src-4.8.7
                                              Deployed on SL6/gcc 4.8.2 (latest 4.8.x release)

                doxygen-1.3.5
                doxygen-1.3.7                 Starting with SL/RHE
                doxygen-1.5.9                 Use this for SL5+ - this package has a dependence in qt
                                              Installed native on SL6

                Python 2.7.1                  Started from SL4.4 and onward, provides pyROOT support
                Python 2.7.5                  Started from SL6 onward, provides pyROOT support
                        pyparsing V1.5.5      SL5 Note: "python setup.py install" to install
                        pyparsing V1.5.7      SL6 Note: "python setup.py install" to install
                        setuptools 0.6c11     SL5 Note: sh the .egg file to install
                        setuptools 0.9.8      SL6 Note: "python setup.py install" to install
                        MySQL-python-1.2.3    MySQL 14.x client libs compatible
                        virtualenv 1.9        SL6 Note: "python setup.py install" to install
                        Cython-0.24           SL6 Note: "python setup.py build ; python setup.py install" 
                        pyflakes / pygments   {TODO}

                libxml2                       Was used only for RH8.0, installed as part of SL later [+] 

                libtool-1.5.8                 This was used for OS not having libtool, Use latest version. 
                libtool-2.4                   Needed under SL5 64 bits kernel (32 bits code will not assemble
                                              otherwise). This was re-packaged with a patch. 
   
                Coin-3.1.1                    Coin 3D and related packages
                Coin-3.1.3                    ... was used for SL6/gcc 4.8.2 + patch 
                                              (use the package named Coin-3.1.3-star)
                simage-1.7a                   
                SmallChange-1.0.0a        
                SoQt-1.5.0a


                astyle_1.15.3                 Started from SL3/RHE upon user request
                astyle_1.19                   SL4.4 and above
                astyle_1.23                   SL5 and above
                astyle_2.03                   SL6 and above
		
                unixODBC-2.2.6                (depend on Qt) Was experimental Linux only for now.
                unixODBC-2.3.0                SL5+, needed if you intend to use DataManagement tools

                MyODBC-3.51.06                Was Experimental on Linux at first, ignore this version
                MyODBC-3.51.12                Version for SL4.4 (needed for mysql 4.1 and above)
                mysql-connector-odbc-3.51.12  <-- Experimental package change - new name starting from 51.12. BEWARE.
                mysql-connector-odbc-5.x      SL5+. As above, only if you intend to use Data Management tools

                boost                         Experimental and introduced in 2010 but not used then
                boost_1_54_0                  SL6+ needed

                log4cxx 0.9.7                 This should be general, initial version
                log4cxx 0.10.0                Started at SL5 - this is now from Apache
			apr-1.3.5             and depend on the Apache Portable Runtime (apr) package
			apr-util-1.3.7        which need to be installed BEFORE log4cxx and in the order
			expat-1.95.7          showed

                valkyrie-1.4.0                Added to SL3 as a GUI companion to valgrind (requires Qt3)
                                              Not installed in SL5 for now (Qt4 only) so ignore  

               fastjet-2.4.4                  Started from STAR Library version SL11e, essentially for analysis
               fastjet-3.0.6                  SL6 onward

               unuran-1.8.1                   Requested and installed from SL6+
               LHAPDF-6.1.6                   Added after SL6.4, gcc 4.8.2
 
In case you have problems
               emacs-24.3                     Installed under SL6 as the default version had font troubles
               vim-7.4                        Update made under SL6.4, please prefer RPM if possible

Not necessary (installed anyway)
                chksum  
                pine4.64                      Added at SL4.4 as removed from base install at BNL

Retired
                xemacs-21.5.15                << Linux only  -- This was temporary and removed 

Other directories are

WorkStation/   contains packages such as OpenAFS or OpenOffice Linux
WebServer/     mostly perl modules needed for our WebServer
Linux/         Linux specific utilities (does not fit in General) or
               packages tested under Linux only.






Some notes about packages :
 Most of them are pretty straight forward to install (like ./configure
 make ; make install (changing the base path /usr/local to $OPTSTAR).
 With configure, this is done using either
   ./configure --prefix=$OPTSTAR
   ./configure --prefix=$XOPTSTAR

 Specific notes follows and include packages which are NOT yet official
 but tried out.


- Beware that the Msql-Mysql-modules perl module requires a hack
  I have not quite understood yet how to make automatic (the 
  advertized --blabla do not seem to work) on platforms supporting
  the client in OPTSTAR
   INC = ... -I$(XOPTSTAR)/include/mysql ...     
   H_FILES = $(XOPTSTAR)/include/mysql/mysql.h
   OTHERLDFLAGS = -L$(XOPTSTAR)/lib/mysql 
   LDLOADLIBS = -lmysqlclient -lm -lz

- GD-2+
  Do NOT select support for animated GIF. This will fail on standard
  SL distributions (default gd lib has no support for that).

ImageMagick

Really easy to install (usual configure / make / make install) but however, the PerlMagick part should be installed separatly (the usual perl module way i.e. cd to the directory, perl Makefile.PL and make / make install). I used the distribution's module. Therefore, that perl-module is not in perl/Installed/ as the other perl-modules. The copy of PerlMagick to /bin/ by default will fail so you may want to additionally do

% make install-info
% make install-data-html

which comes later depending on version.
 

lynx

- lynx2-8-2 / lynx2-8-5 
  Note: First, I tried lynx2-8-4 and the make file / configure
        is a real disaster. For 2-8-2/2-8-5, follow the notes 
        below

  General :
  %  ./configure --prefix=$XOPTSTAR {--with-screen=slang}

  Do not forget to
  % make install-help
  % make install-doc

 caveat 1 -- Linux (lynx 2-8-2 only, fixed at 2-8-5)

  $OPTSTAR/lib/lynx.cfg was modified as follow
96,97c96,97
< #HELPFILE:http://www.crl.com/~subir/lynx/lynx_help/lynx_help_main.html
< HELPFILE:file://localhost/opt/star/lib/lynx_help/lynx_help_main.html
---
> HELPFILE:http://www.crl.com/~subir/lynx/lynx_help/lynx_help_main.html
> #HELPFILE:file://localhost/PATH_TO/lynx_help/lynx_help_main.html 

   For using curses (needed under Linux, otherwise, the screen looks funny), 
   one has to do a few manipulation by hand i.e. 
   . start with ./configure --prefix=$XOPTSTAR --with-screen=slang
   . edit the makefile and add -DUSE_SLANG to SITE_DEFS
   . change CPPFLAGS from /usr/local/slang to $OPTSTAR/include [when slang is local]
     Version 2-8-5 has this issue fixed.
   . Change LIBS -lslang to -L$OPTSTAR/lib -lslang
   . You are ready now
   There is probably an easier way but as usual, I gave up after close
   to 15mnts reading, as much struggle and complete flop at the end ..

 caveta 2 -- Solaris/True64 : 
   We did not build with slang but native (slang screws colors up)

text2c, chksum, chtext, icalc

Those packages can be assembled simply by using the following command:

% make clean && make install PREFIX=$OPTSTAR

To build a 32 bits versions of the executable under a 64 bits kernel, use

• text2c:             % make CC=`which gcc` CFLAGS="-lfl -m32"
• icalc:                % make CC=`which gcc` CFLAGS="-lm -m32"
• chksum:           % make CC=`which gcc` CFLAGS="-m32 -trigraphs"
• chtext:             % make CC=`which gcc` CFLAGS="-lfl -m32"

xv-3.10a

This package distributed already patched and in principle, only a few 'make' commands should suffice. Note

• xv is licensed so the usage as to remain stricly for your users' amusement only. If you use this package for doing any work, you are violating the law. Please, read the license agreement at http://www.trilon.com/xv/pricing.html 

Normal build

Now,  you should be ready to build the main program (I am not sure why some depencies fail on some platform and did not bother to fix).

% cd tiff/
% make clean && make
% cd ../jpeg
% make clean && make
% cd ..
% rm -f *.o  && make
% make -f Makefile.gcc64 install BINDIR=$OPTSTAR/bin

For 32 bits compilation under a 64 bits kernel

% cd tiff/
% make clean && make CC=`which gcc` COPTS="-O -m32"
% cd ../jpeg
% make clean && make CC=`which gcc` CFLAGS="-O -I. -m32" LDFLAGS="-m32"
% cd ..
% rm -f *.o   && make -f Makefile.gcc32
% make -f Makefile.gcc32 install BINDIR=$OPTSTAR/bin

Makefile.gcc32 and Makefile.gcc64 are both provided for commodity.

Building from scratch (good luck)

However, if you need to re-generate the makefile (may be needed for new architectures), use

% xmkmf 

Then, the patches is as follow

% sed "s|/usr/local|$OPTSTAR|" MakeFile >Makefile.new
% mv Makefile.new Makefile

and in xv.h, line 119 becomes

# if !defined(__NetBSD__) && ! defined(__USE_BSD) 

After xmkmf, you will need to

% make depend

before typing make. This may generate some warnings. Ignore then.

However, I had to fix diverse caveats depending on situations ...

Caveat 1 - no tiff library found

Go into the tiff/ directory and do

% cd tiff  % make -f Makefile.gcc   % cd ..

to generate the mkg3states (itself creating the g3states.h file) as it did not work.

Caveat 2 - tiff and gcc 4.3.2 in tiff/

With gcc 4.3.2 I created an additional .h file named local_types.h and force the definition of a few of the u_* types but using define statements (I know, it is bad). The content of that file is as follows

#ifndef _LOCAL_TYPES_
#define _LOCAL_TYPES_

#if !defined(u_long)
# define u_long unsigned long
#endif
#if !defined(u_char)
# define u_char unsigned char
#endif
#if !defined(u_short)
# define u_short unsigned short
#endif
#if !defined(u_int)
# define u_int unsigned int
#endif

#endif

and it needs to be included in tiff/tif_fax3.h and tiff/tiffiop.h .

Caveat 3 -- no jpeg library?

 In case you have a warning about jpeg such as No rule to make target `libjpeg.a', do the following as well:

% cd jpeg
% ./configure
% make
% cd ..

Nedit

There is no install provided. I did

% make linux
% cp source/nc source/nedit $OPTSTAR/bin/
% cp doc/nc.man $OPTSTAR/man/man1/nc.1
% cp doc/nedit.man $OPTSTAR/man/man1/nedit.1

Other targets

% make dec
% make solaris

If you need to build for another compiler or another platform, you may want to copy one of the provided makefile and modify them to create a new target. For example, if you have a 64 bits kernel but want to build a 32 bits nedit (for consistency or otherwise), you could do this:

% cp makefiles/Makefile.linux makefiles/Makefile.linux32

then edit and add -m32 to bothe CFLAGS and LIBS. This will add a target "platform" linux32 for a make linux32 command (tested this and worked fine). The STAR provided package added (in case) both a linux64 and a linux32 reshaped makefile to ensure easy install for all kernels (gcc compiler should be recent and accept the -m flag).

Pythia libraries

The unpacking is "raw". So, go in a working directory where the .tar.gz are, and do the following (for linux)

% test -d Pythia && rm -fr Pythia ; mkdir Pythia && cd Pythia && tar -xzf ../pythia5.tar.gz 
% ./makePythia.linux 
% mv libPythia.so $OPTSTAR/lib/ 
% cd .. 
% 
% test -d Pythia6 && rm -fr Pythia6 ; mkdir Pythia6 && cd Pythia6 && tar -xzf ../pythia6.tar.gz 
% test -e main.c && rm -f main.c 
% ./makePythia6.linux 
% mv libPythia6.so $OPTSTAR/lib 
% 

Substitute linux with solaris for Solaris platform. On Solaris, Pythia6 requires gcc to build/link.

On SL5, 64 bits note

Depending on whether you compile a native 64 bit library support or a cross-platform 32/64, you will need to handle it differently.

For a 64 bits platform, I had to edit the makePythia.linux and  -fPIC to the options for a so the binaries main.c . I did not provide a patched package mainly because v5 is not really needed in STAR. For pythia6 caveat: On SL5, 64 bits, use makePythia6.linuxx8664 file. You will need to chmod +x first as it was not executable in my version.

On 64 bit platform to actually build a cross-platform version, I had instead to use the normal build but make sure to add -m32 to compilation and linker options and -fPIC to compilation option.

True64

% chmod +x ./makePythia.alpha && ./makePythia.alpha Pythia6

% chmod +x ./makePythia6.alpha && ./makePythia6.alpha 

The following script was used to split the code which was too big

 #!/usr/bin/env perl
 $filin = $ARGV[0];
 open(FI,$filin);
 $idx = $i = 0;
 while( defined($line = <FI>) ){
    chomp($line); $i++;

    if ($i >= 500 && $line =~ /subroutine/){
	$i = 0;
	$idx++;
    }

    if ($i == 0){
	close(FO);
	open(FO,">$idx.$filin");
	print "Opening $idx.$filin\n";
    }
    print FO "$line\n";
 }
 close(FO);
 close(FI);

Qt 4

Starts the same than Qt3 i.e. assuming that SRC=/afs/rhic.bnl.gov/star/common/General/Sources/ and $x and $y stands for major and minor versions of Qt. There are multiple flavors of the package name (it was called qt-x11-free* then qt-x11-opensource* and with more recent package qt-everywhere-opensource-src*). For the sake of instructions, I provide a generic example with the most recent naming (please adapt as your case requires). WHEREVER is a location of your choice (not the final target directory).

% cd $WHEREVER
% tar -xzf $SRC/qt-everywhere-opensource-src-4.$x.$y.tar.gz
% cd qt-everywhere-opensource-src-4.$x.$y
% ./configure --prefix=$XOPTSTAR/qt4.$x -qt-sql-mysql -no-exceptions -no-glib -no-rpath 

To build a 32/64 bits version on a 64 bits OS or forcing a 32 bits exec (shared mode) on a 32 bits OS, use a configure target like the below

% ./configure  -platform linux-g++-32 -mysql_config $OPTSTAR/bin/mysql_config [...] 
% ./configure  -platform linux-g++-64 [...]

Note that the above assumes you have a proper $OPTSTAR/bin/mysql_config. ON a mixed 64/32 bits node, the default in /usr/bin/mysql_config will return the linked library as the /usr/lib64/mysql path and not the /usr/lib/mysql and hence, Qt make will fail finding the dependencies necessary to link with -m32. The trick we had was to copy mysql_config and replace lib64 by lib and voila!.

Compiling

  % make
  % make install

  % cd $OPTSTAR
  % ln -s qt4.$x ./qt4

For compiling with a different compiler, note that the variables referenced in this section will be respected by configure. You HAVE TO do this as the project files and other configuration files from Qt will include information on the compiler (inconsistency may arrise otherwise).

Misc notes

• If you use the same directory tree for compiling the 64 bits and the 32 bits version, please note that 'make clean' will not do the proper job. You will need to use a more systematic % find . -name '*.o' -exec rm -f {} \;  command before running  ./configure again.
• We added mysql support in Qt4 and Qt can now be compiled in a separate directory and installed properly (at last!). If the mysql support gives you trouble on a 64 bit OS attempting to build a 32 bit image, be sure you have used as indicated above the -mysql_config /usr/lib/mysql/mysql_config option as otherwise, the default mysql_config will be picked from /usr/bin and that version will refer to the 64 bits  libraries (the link will then fail). 
• For SL44, we created a qt3 distribution as then, both 3 and 4 existed. Otherwise, the ./qt4 link as indicated above is sufficient for SL5 and above.
• On some systems (SL3.0.2 for sure), I also used
  • -no-openssl   as there were include problems with ssl.h and krb5.h
  • -qt-libtiff   as the default system included header did not agree with Qt code
  • -platform linux-icc  could be used for icc based compiler

Qt 3

Horribly packaged, the easiest is to unpack in $OPTSTAR, cd to qt-x11-free-3.X.X (where X.X stands for the current sub-version deployed on our node), run the configure script, make the package, then make clean. Then, link

  % cd $OPTSTAR && ln -s qt-x11-free-3.X.X qt

Later release can be build that way with changing the soft-link without removing the preceeding version entirely. Before building, do the following (if you had previous version of Qt installed). This is not necessary if you install the package the first time around. Please, close windows after compilation to ensure STAR path sanity.

  % cd $OPTSTAR/qt
  % setenv QTDIR `pwd`
  % setenv LD_LIBRARY_PATH `pwd`/lib:$LD_LIBRARY_PATH 
  % setenv PATH `pwd`/bin:$PATH

To configure the package, then use one of:

• Linux gcc: ./configure --prefix=`pwd` -no-xft -thread
• Linux icc:  ./configure --prefix=`pwd` -no-xft -thread -platform linux-icc
• True64 :   ./configure --prefix=`pwd` -no-xft -thread
• Solaris:     ./configure --prefix=`pwd` -no-xft

In case of thread, the regular version is build first then the threaded version (so far, they have different names and no Soft links).

You may also want to edit  $QTDIR/mkspecs/default/qmake.conf and replace the line

QMAKE_RPATH		= -Wl,-rpath,

by

QMAKE_RPATH		= 

By doing so, you would disable the rpath shared library loading and rely on LD_LIBRARY_PATH only for loading your Qt related libraries. This has the advantages that you may copy the Qt3 libraries along your project and isolate onto a specific machine without the need to see the original installation directory.

unixODBC

% ./configure --prefix=$XOPTSTAR [CC=icc CXX=icc]
% make clean       # in case you are re-using the same directory for multiple platform 
% make 
% make install

Use the environment variables noted in this section and all will go well.

Note on versions earlier than 2.3.0 (including 2.2.14 previously suggested)

The problem desribed below DOES NOT exist if you use 32 bits kernel OS and is specific to 64 bits kernel with 32 bits support.

For a 32 bits compilation under a 64 bits kernel, please use % cp -f $OPTSTAR/bin/libtool .  after the ./configure and before the make (see this section for an explaination of why). unixODBC versions 2.3.0 does not have this problem.

MyODBC

Older version

Came with sources and one could compile "easily" (and register manually).

- MyODBC
  Linux  % ./configure --prefix=$XOPTSTAR --with-unixODBC=$XOPTSTAR [CC=icc CXX=icc]
  Others % ./configure --prefix=$XOPTSTAR --with-unixODBC=$XOPTSTAR --with-mysql-libs=$XOPTSTAR/lib/mysql 
           --with-mysql-includes=$XOPTSTAR/include/mysql --with-mysql-path=$XOPTSTAR

  Note : Because of an unknown issue, I had to use --disable-gui on True64
         as it would complain about not finding the X include ... GUI is
         not important for ODBC client anyway but whenever time allows ... 

  Deploy instructions at http://www.mysql.com/products/myodbc/faq_toc.html

Version 5.x of the connector

Get the proper package, currently named mysql-connector-odbc-5.x.y-linux-glibc2.3-x86-32bit or  mysql-connector-odbc-5.x.y-linux-glibc2.3-x86-64bit. the package are available from the MySQL Web site. The install will need to be manual i.e.

% cp -p bin/myodbc-installer $OPTSTAR/bin/
% cp -p lib/*  $OPTSTAR/lib/
% rehash

To register the driver, use the folowing command

% myodbc-installer -d -a -n "MySQL ODBC 5.1 Driver" -t "DRIVER=$OPTSTAR/lib/libmyodbc5.so;SETUP=$OPTSTAR/lib/libmyodbc3S.so"
% myodbc-installer -d -a -n "MySQL" -t "DRIVER=$OPTSTAR/lib/libmyodbc5.so;SETUP=$OPTSTAR/lib/libmyodbc3S.so"

this will add a few lines in $OPTSTAR/etc/odbcinst.ini . The  myodbc-installer -d -l does not seem to be listing what you installed though (but the proper lines will be added to the configuration).

doxygen

Installation would benefit from some smoothing + note the space between the --prefix and OPTSTAR (non standard option set for configure).

Use one of

% ./configure --prefix $OPTSTAR                       # for general compilation
% ./configure --platform linux-32 --prefix $OPTSTAR   # Linux, gcc 32 bits - this option was added in the STAR package
% ./configure --platform linux-64 --prefix $OPTSTAR   # Linux, gcc 64 bits - this option was fixed in the STAR package

then

% make
% make install

as usual but also

% make docs

which will fail d ue to missing eps2pdf program. Will create however the HTML files you will need to copy somewhere.

% cp -r html $WhereverTheyShouldGo

and as example

% cp -r html /afs/rhic.bnl.gov/star/doc/www/comp/sofi/doxygen

Note: The linux-32 and linux-64 platform were packaged in the archive provided for STAR (linux-32 does not exists in the original doxygen distribution while linux-64 is not consistent with -m64 compilation option).

Additional Graphics libraries

Starting from SL5, we also deployed the following: coin, simage, SmallChange, SoQt. Those needs to be installed before Qt4 but after doxygen. All options are specified below to install those packages. Please, substitute -m32 by -m64 for a 64 bits native support. After the configure, the usual make and make install is expected.

The problem desribed below DOES NOT exist if you use 32 bits kernel OS and is specific to 64 bits kernel with 32 bits support.

For the 32 bits version compilation under a 64 bits kernel and for ALL sub-packages below, please be sure you have the STAR version of libtool installed and use the command
% cp -f $OPTSTAR/bin/libtool .
after the ./configure to replace the generated local libtool script. This will correct a link problem which will occur at link time (see the libtool help for more information).

Coin:

% ./configure --enable-debug --disable-dependency-tracking --enable-optimization=yes \
--prefix=$XOPTSTAR CFLAGS="-m32 -fPIC -fpermissive" CXXFLAGS="-m32 -fPIC -fpermissive" LDFLAGS="-m32 -L/usr/lib" \
--x-libraries=/usr/lib

or, for or the 64 bits version

% ./configure --enable-debug --disable-dependency-tracking --enable-optimization=yes \
--prefix=$XOPTSTAR CFLAGS="-m64 -fPIC -fpermissive" CXXFLAGS="-m64 -fPIC -fpermissive" LDFLAGS="-m64" 

simage (needs Qt installed and QTDIR defined prior):

% ./configure --prefix=$XOPTSTAR --enable-threadsafe --enable-debug --disable-dependency-tracking \ 
--enable-optimization=yes --enable-qimage CFLAGS="-m32 -fPIC" CXXFLAGS="-m32 -fPIC" \ 
LDFLAGS="-m32" FFLAGS="-m32 -fPIC" --x-libraries=/usr/lib

or, for the 64 bits version

% ./configure --prefix=$XOPTSTAR --enable-threadsafe --enable-debug --disable-dependency-tracking \ 
--enable-optimization=yes --enable-qimage CFLAGS="-m64 -fPIC" CXXFLAGS="-m64 -fPIC" \ 
LDFLAGS="-m64" FFLAGS="-m64 -fPIC" 

SmallChange:

% ./configure --prefix=$XOPTSTAR --enable-threadsafe --enable-debug --disable-dependency-tracking \
--enable-optimization=yes CFLAGS="-m32 -fPIC -fpermissive" CXXFLAGS="-m32 -fPIC -fpermissive" \
LDFLAGS="-m32" FFLAGS="-m32 -fPIC"

or, for the 64 bits version

% ./configure --prefix=$XOPTSTAR --enable-threadsafe --enable-debug --disable-dependency-tracking \
--enable-optimization=yes CFLAGS="-m64 -fPIC -fpermissive" CXXFLAGS="-m64 -fPIC -fpermissive" \
LDFLAGS="-m64" FFLAGS="-m64 -fPIC"

SoQt:

./configure --prefix=$XOPTSTAR --enable-threadsafe --enable-debug --disable-dependency-tracking \ 
--enable-optimization=yes --with-qt=true --with-coin CFLAGS="-m32 -fPIC  -fpermissive" CXXFLAGS="-m32 -fPIC  -fpermissive" \
LDFLAGS="-m32" FFLAGS="-m32 -fPIC"

or, for the 64 bits version

./configure --prefix=$XOPTSTAR --enable-threadsafe --enable-debug --disable-dependency-tracking \ 
--enable-optimization=yes --with-qt=true --with-coin CFLAGS="-m64 -fPIC  -fpermissive" CXXFLAGS="-m64 -fPIC -fpermissive" \
LDFLAGS="-m64" FFLAGS="-m64 -fPIC"

flex

Flex is usually not needed but some OS have pre-GNU flex not adequate so I would recommend to deploy flex-2.5.4  anyway (the latest version since Linux 2001). Do not install under Linux if you have flex already on your system as rpm.

Attention: Under SL5 64 bits, be sure you have flex32libs-2.5.4a-41.fc6 installed as documented on Scientific Linux 5.3 from 4.4. Linkage of 32 bits executable would otherwise dramatically fail.

- Xpm (Solaris)
  % xmkmf
  % make Makefiles
  % make includes
  % make 
  I ran the install command by hand changing the path (cut and paste)
  Had to 


  % cd lib
  % installbsd -c -m 0644 libXpm.so $OPTSTAR/lib
  % installbsd -c -m 0644 libXpm.a $OPTSTAR/lib
  % cd ..
  % cd sxpm/
  % installbsd -c sxpm $OPTSTAR/bin
  % cd ../cxpm/
  % installbsd -c cxpm $OPTSTAR/bin
  %
  
  Onsolaris, the .a was not there, add to
  % cd lib && ar -q libXpm.a *.o && cp libXpm.a $OPTSTAR/lib
  % cd ..

  Additionally needed
  % if ( ! -e $OPTSTAR/include) mkdir $OPTSTAR/include 
  % cp lib/xpm.h $OPTSTAR/include/

  

- libpng 
  ** Solaris **
  % cat scripts/makefile.solaris | sed "s/-Wall //" > scripts/makefile.solaris2
  % cat scripts/makefile.solaris2 | sed "s/gcc/cc/" > scripts/makefile.solaris3
  % cat scripts/makefile.solaris3 | sed "s/-O3/-O/" > scripts/makefile.solaris2
  % cat scripts/makefile.solaris2 | sed "s/-fPIC/-KPIC/" > scripts/makefile.solaris3
  % 
  % make -f scripts/makefile.solaris3

  will eventually fail related to libucb. No worry, this can be sorted
  out (http://www.unixguide.net/sun/solaris2faq.shtml) by including
  /usr/ucblib in the -L list
  % cc -o pngtest -I/usr/local/include -O pngtest.o -L. -R. -L/usr/local/lib \
    -L/usr/ucblib -R/usr/local/lib -lpng -lz -lm
  % make -f scripts/makefile.solaris3 install prefix=$OPTSTAR


  ** True64 **
  Copy the make file but most likely, a change like
ZLIBINC = $(OPTSTAR)/include
ZLIBLIB = $(OPTSTAR)/lib

  in the makefile is neeed.
 
  pngconf.h and png.h needed for installation and either .a or .a + .so

cp pngconf.h png.h $OPTSTAR/include/
cp libpng.* $OPTSTAR/lib



- mysql client (Solaris)
 % ./configure --prefix=$XOPTSTAR --without-server {--enable-thread-safe-client}
 (very smooth)
 The latest option is needed to create the libmysqlclient_r needed by some
 applications. While this so is build with early version of MySQL, version
 4.1+ requires the configure option explicitly.


- dejagnu-1.4.1	[Solaris specific]
  the install program was not found.
  % cd doc/ && cp ./runtest.1 $OPTSTAR/man/man1/runtest.1
  % chmod 644 $OPTSTAR/man/man1/runtest.1

Jed

The basic principles is as usual

% ./configure --prefix=$OPTSTAR
% make
% make xjed
% make install

However, on some platform (but this was not seen as a problem on SL/RHE), you may need to apply the following tweak before typing make. Edit the configure script and add $OPTSTAR (possibly /opt/star) to it as follow.

JD_Search_Dirs="$JD_Search_Dirs \
                $includedir,$libdir \
                /opt/star/include,/opt/star/lib \
                /usr/local/include,/usr/local/lib \
                /usr/include,/usr/lib \
                /usr/include/slang,/usr/lib \
                /usr/include/slang,/usr/lib/slang" 

32 / 64 bit issue?

The problem desribed below DOES NOT exist if you use 32 bits kernel OS and is specific to 64 bits kernel with 32 bits support.

The variables described here will make configure pick up the right comiler and compiler options. On our initial system, the 32 bits compilation under the 64 bits kernel Makefile tried to do something along the line of -L/usr/X11R6/lib64 -lX11 but did not find X11 libs (since the path is not adequate). To correct for this problem, edit src/Makefile and replace XLIBDIR = -L/usr/lib64 by XLIBDIR = -L/usr/lib . You MUST have the 32 bits compatibility libraries installed on your 64 bits kernel for this to work.

AIX

I had to make some hack on AIX (well, who wants to run on AIX in the first place right ?? but since AIX do not have any emacs etc ... jed is great) as follow

• make a copy of unistd.h and comment the sleep() prototype
• modify file.c to include the local version (replace <> by "")
• modify main.c to include sys/io.h (and not io.h) and comment out direct.h

Voila (works like a charm, don't ask).

 
emacs

Version 24.3

In the below options, I recommend the with-x-toolkit=motif as the default GTK will lead to many warnings depending on the user's X11 server version and supported features. Motif may create an "old look and feel" but will work. However, you may have a local fix for GTK (by installing all required modules and dependencies) and not need to go to the Motif UI.

% ./configure --with-x-toolkit=motif --prefix=$OPTSTAR

For the 32 bits version supporting 64/32 bits, use the below

% ./configure --with-crt-dir=/usr/lib --with-x-toolkit=motif --prefix=$OPTSTAR CFLAGS="-m32 -fPIC" CXXFLAGS="-m32 -fPIC" LDFLAGS="-m32"

Then the usual 'make' and 'make install'.

Below are old instructions you should ignore

- emacs
  Was repacked with leim package (instead of keeping both separatly)
  in addition of having a patch in src/s/sol2.h for solaris as follow
 #define HAVE_VFORK 1
 #endif
 
+/* Newer versions of Solaris have bcopy etc. as functions, with
+   prototypes in strings.h.  They lose if the defines from usg5-4.h
+   are visible, which happens when X headers are included.  */
+#ifdef HAVE_BCOPY
+#undef bcopy
+#undef bzero
+#undef bcmp
+#ifndef NOT_C_CODE
+#include <strings.h>
+#endif
+#endif
+

  Nothing to do differently here. This is just a note to keep track
  of changes found from community mailing lists.

  % ./configure --prefix=$OPTSTAR --without-gcc
  

- Xemacs (Solaris)
  % ./configure --without-gcc --prefix=$OPTSTAR
  Other solution, forcing Xpm 
  % ./configure --without-gcc --prefix=$OPTSTAR --with-xpm --site-prefixes=$OPTSTAR

  Possible code problem :
  /* #include <X11/xpm.h> */
  #include <xpm.h> 

- gcc-2.95 On Solaris was used as a base compiler
  % ./configure --prefix=$OPTSTAR
  % make bootstrap

  o Additional gcc on Linux
  Had to do it in multiple passes (you do not need to do the first pass
  elsewhere ; this is just because we started without a valid node).

  A gcc version < 2.95.2 had to be used. I used a 6.1 node to assemble
  it and install in a specific AFS tree (cross version)
  % cd /opt/star && ln -s /afs/rhic/i386_linux24/opt/star/alt .
  Move to the gcc source directory
  % ./configure --prefix=/opt/star/alt
  % make bootstrap
  % make install
  install may fail in AFS land. Edit gcc/Makefile and remove "p" option
  to the tar options TAROUTOPTS .
  
  For it work under 7.2, go on a 7.2 node and
  % cp /opt/star/alt/include/g++-3/streambuf.h /opt/star/alt/include/g++-3/streambuf.h-init
  % cp -f /usr/include/g++-3/streambuf.h /opt/star/alt/include/g++-3/streambuf.h
  ... don't ask ...


  o On Solaris, no problems
  % ./configure --prefix=/opt/star/alt
  etc ...

- Compress-Zlib-1.12 --> zlib-1.1.4
  If installed in $OPTSTAR,
  % setenv ZLIB_LIB $OPTSTAR/lib
  % setenv ZLIB_INCLUDE $OPTSTAR/include
  

- findutil
  Needed a patch in lib/fnmatch.h for True64
  as follow :
  + addition of defined(__GNUC__) on line 49
  + do a chmod +rw lib/fnmatch.h  first

#if !defined (_POSIX_C_SOURCE) || _POSIX_C_SOURCE < 2 || defined (_GNU_SOURCE) || defined(__GNUC__)






* CLHEP1.8                      *** Experimental only ***
 printVersion.cc needs a correction #include <string> to <string.h>
 for True64 which is a bit strict in terms of compilation.

 On Solaris, 2 caveats
  o gcc was used (claim that CC is used but do not have the include)
  o install failed running a "kdir" command instead of mkdir so do a
    % make install MKDIR='mkdir -p'

 Using icc was not tried and this package when then removed.



- mysqlcc
 ./configure --prefix=$OPTSTAR --with-mysql-include=$OPTSTAR/include/mysql --with-mysql-lib=$OPTSTAR/lib/mysql
 The excutable do not install itself so, one needs to
 % cp -f  mysqlcc $OPTSTAR/bin/

libtool

First, please note that the package distributed for STAR contains a patch for support of the 32 / 64 bits environment. If you intend to download from the original site, please apply the patch below as indicated. If you do not use our distributed package and attempt to assemble a 32 bits library under a 64 bits kernel, we found cases where the default libtool will fail.

Why the replacement of libtool? Sometimes, "a" version of libtool is added along software packages indicated in this help. However, those do not consider the 32 bits / 64 bits mix and often, their use lead to the wrong linkage (typical problem is that a 32 bits executable or shared library is linked against the 64 bits stdc++ versions, creating a clash).

This problem does not existswhen you assemble a 64 bits code under a 64 bits kernel or assemble a 32 bits codes under a 32 bits kernel.

In all cases, to compile and assemble, use a command line like the below:

% ./configure --prefix=$XOPTSTAR CFLAGS="-m32 -fPIC" CXXFLAGS="-m32 -fPIC" \
FFLAGS="-m32 -fPIC" FCFLAGS="-m32 -fPIC" LDFLAGS="-m32"                        # 32 bits version
% ./configure --prefix=$XOPTSTAR CFLAGS="-m64 -fPIC" CXXFLAGS="-m64 -fPIC" \
FFLAGS="-m64 -fPIC" FCFLAGS="-m64 -fPIC" LDFLAGS="-m64"                        # 64 bits version
% make
% make install

Patches

libtool 2.4

The file ./libltdl/config/ltmain.sh needs the following patch

< 
< 	# JL patch 2010 -->
< 	if [ -z "$m32test" ]; then
< 	    #echo "Defining m32test"
< 	    m32test=$($ECHO "${LTCFLAGS}" | $GREP m32)
<    fi	
< 	if [ "$m32test" != "" ] ; then
< 	  dependency_libs=`$ECHO " $dependency_libs" | $SED 's% \([^ $]*\).ltframework% -framework \1%g' | $SED 's|lib64|lib|g'`
< 	else
< 	  dependency_libs=`$ECHO " $dependency_libs" | $SED 's% \([^ $]*\).ltframework% -framework \1%g'`
< 	fi
< 	# <-- end JL patch
< 
---
> 	dependency_libs=`$ECHO " $dependency_libs" | $SED 's% \([^ $]*\).ltframework% -framework \1%g'`

gdb (patch)

in gdb/linux-nat.c

         /*
   fprintf_filtered (gdb_stdout,
                     "Detaching after fork from child process %d.\n",
                     child_pid);
       */

and go (no, I will not explain).

astyle

Version 2.03

% cd astyle_2.03/src
% make -f ../build/gcc/Makefile CXX="g++ -m32 -fPIC"   # for the 64 bits version, use the same command
                                                       # with   CXX="g++ -m64 -fPIC" 
% cp bin/astyle $XOPTSTAR/bin/
% test -d $XOPTSTAR/share/doc/astyle || mkdir -p $XOPTSTAR/share/doc/astyle
% cp ../doc/*.* $XOPTSTAR/share/doc/astyle

The target

% make -f ../build/gcc/Makefile clean

also works fine and is needed between versions.


Version 1.23

Directory structure changes but easier to make the package so use instead

% cd astyle_1.23/src/
% make -f ../buildgcc/Makefile  CXX="$CXX $CFLAGS"
% cp ../bin/astyle $OPTSTAR/bin/
% cd .. 

Note that the compressed command above assumes you have define dthe envrionment variables as described in this section. Between OS (32 / 64 bits) you may need to % rm -f obj/* as the make system will not reocgnize the change between kernels (you alternately may make -f ../buildgcc/Makefile clean but a rm will be faster :-) ).

Documentation

A crummy man page was added (will make it better later if really important). It was generted as follow and provided for convenience in the packages for STAR (do not overwrite because I will not tell you what to do to make the file a good pod):

% cd doc/
% lynx -dump astyle.html >astyle.pod 

[... some massage beyond the scope of this help - use what I provided ...]

% pod2man astyle.pod >astyle.man 
% cp astyle.man $OPTSTAR/man/man1/astyle.1 

Versions < 1.23

Find where the code really unpacks. There are no configure for this package.

% cd astyle_1.15.3 ! or 
% cd astyle/src 
% make 
% cp astyle $OPTSTAR/bin/

Version 1.15.3

The package comes as a zip archive. Be aware that unpacking extracts files in the current directory. So, the package was remade for convenience. Although written in C++, this executable will perform as expected under icc environment. On SL4 and for versions, gcc 3.4.3, add -fpermissive to the Makefile CPPFLAGS.

valgrind

MUST be installed using $XOPTSTAR because there is an explicit reference to the install path. Copying to a local /opt/star would therefore not work. For icc, use the regular command as this is a self-contained program without C++ crap and can be copied from gcc/icc directory. The command is

% ./configure --prefix=$XOPTSTAR  

Note: valgrind version >= 3.4 may ignore additional compiler options (but will respect the CC and CXX variables) as it will assemble both 32 bits and 64 bits version on a dual architecture platform. You could force a 32 build only by adding the command line options --enable-only32bit.

Caveats for earlier revisions below:

Version 2.2

A few hacks were made on the package, a go-and-learn experience as problems appeared
 

coregrind/vg_include.h
123c123
< #define VG_N_RWLOCKS 5000
---
> #define VG_N_RWLOCKS 500

coregrind/vg_libpthread.vs
195a196
> __pthread_clock_gettime; __pthread_clock_settime;

to solve problems encountered with large programs and pthread.

APR

The problem desribed below DOES NOT exist if you use 32 bits kernel OS and is specific to 64 bits kernel with 32 bits support.

For a 32 bits compilation under a 64 bits kernel, please use % cp -f $OPTSTAR/bin/libtool .  after the ./configure and before the make (see this section for an explaination of why) for both the apr and expat packages.

apr is an (almost) straight forward installation:                

% ./configure --prefix=$OPTSTAR

apr-util needs to have one more argument i.e. 

% ./configure --prefix=$OPTSTAR --with-apr=$OPTSTAR

The configure script will respect the environment variables described in this section and, provided you have defined them properly for the intended target (32 or 64 bits executable), the resulting Mkaefile will be properly generated without further modifications needed.

Note however that the package distributed in STAR has one hack to the fconfigure script a follows (apply if you download from anoher source than STAR's distributed packages):

% diff configure.orig configure
4255c4255,4256
<     CFLAGS="-g -O2"
---
>     # CFLAGS="-g -O2"
>     CFLAGS="-g -m32"
4261c4262,4263
<     CFLAGS="-O2"
---
>     # CFLAGS="-O2"
>     CFLAGS="-m32"

This will allow another way to assemble the package (without having to define the env variables) but you will need to substitute -m32 by -m64 as appropriate.

expat package is similar

% ./configure --prefix=$OPTSTAR

log4cxx

log4cxx 0.10.x

This distribution is part of the Apache project and requires APR library (see above).

The package was taken nearly as-is apart from the following patches:

• inputstreamreader.cpp, socketoutputstream.cpp, console.cxx    - requires adding #include <string.h> on top as memmove, mencpy are no longer implicit but declared in string.h

After installing APR and using the patches as indicated, use

% ./configure --prefix=$XOPTSTAR --with-apr=$XOPTSTAR CFLAGS="-m32 -fPIC" CXXFLAGS="-fno-inline -g -m32" LDFLAGS="-m32"
or
% ./configure --prefix=$XOPTSTAR --with-apr=$XOPTSTAR CFLAGS="-m64 -fPIC" CXXFLAGS="-fno-inline -g -m64" LDFLAGS="-m64"

% cp -f $OPTSTAR/bin/libtool . 
% make
% make install

Please do NOT forget to use % cp -f $OPTSTAR/bin/libtool .  after the ./configure and before the make (see this section for an explaination of why). This assummes you installed libtool as instructed.

Finally, there is one patch needed if you download the package from other sources than where STAR provides the packages. The patch relates to a problem with atomic operations handling.

Index: src/main/cpp/objectimpl.cpp
===================================================================
--- src/main/cpp/objectimpl.cpp    (revision 654826)
+++ src/main/cpp/objectimpl.cpp    (working copy)
@@ -36,12 +36,12 @@

  void ObjectImpl::addRef() const
  {
-  apr_atomic_inc32( & ref );
+  ref++;
  }

  void ObjectImpl::releaseRef() const
  {
-  if ( apr_atomic_dec32( & ref ) == 0 )
+  if ( --ref == 0 )
    {
      delete this;
    }

log4cxx 0.9.5

There is a bug on Linux so, start with commenting all lines related to HAVE_LINUX_ATOMIC_OPERATIONS in configure.in before the below. Finally, two code had to be patched are now repacked

• filewatchdog.cpp line 21: comment #ifdef WIN32 and the associated #endif
• cocketimpl.cpp line 41: add #include <errno.h>

For ODBC support, one needs

% setenv CPPFLAGS "-I$XOPTSTAR/include"
% setenv LDFLAGS  "-L$XOPTSTAR/lib"

log4cxx 0.9.7

Also need to do the below or it will not even find the libs at configure.

% setenv CPPFLAGS "-I$XOPTSTAR/include"
% setenv LDFLAGS  "-L$XOPTSTAR/lib"

On Scientific Linux 4.4 aka Linux 2.6 replace as follows

#AC_CHECK_FUNCS(gettimeofday ftime setenv)
AC_CHECK_FUNCS(setenv)

Linux 7.3 distributions note

On Version 7.3 of Linux, this is hard to install. You will need to upgrade m4, autoconf to at the least the versions specified for "other platforms".  It won't compile easily with gcc 2.96 though. But it can using

% ./configure --prefix=$OPTSTAR CC=/usr/bin/gcc3 CXX=/usr/bin/g++3

if you have all gcc 3 ready.

Finally, if you install log4cxx from a new Linux version (especially one having a different version of autoconf tools), you better start from a fresh directory and not attempt to use the 'clean' target (it will fail).

Summay then:

  Linux gcc  (general instructions, all log4cxx) 

% ./autogen.sh 
% ./configure --prefix=$OPTSTAR [--with-ODBC=unixODBC]

  Linux icc

% setup icc 
% ./configure --prefix=$XOPTSTAR CC=icc CXX=icc [--with-ODBC=unixODBC]

   If icc is the second target, you should use 'make clean' before the configure.

  Solaris
 
   Does not configure (need the autoconf tools)

  True64
   Not tried yet

libxml2

Platform: so far needed to update RH 8.0 only, add to propagate to other platform in 2006 due to a component dependence issue.

% ./configure --without-python --prefix=$XOPTSTAR

pine

Re-added at BNL since SL4.4 because it was removed from the base installation, this may not be needed for your site (install from RPM, it exists).

Scientific Linux (don't get fooled by the targets)

% ./build lrh 
% cp bin/pine bin/pico $OPTSTAR/bin/

On a mixed of 32 / 64 bits architecture and/or with alternate  gcc versions, the command examples below could be used:

% ./build lrh CC=`which gcc` SSLDIR=/usr/include/openssl/ EXTRALDFLAGS="-m32" EXTRACFLAGS="-m32 -fPIC"
[or]
% ./build lrh CC=`which gcc` SSLDIR=/usr/include/openssl/ EXTRALDFLAGS="-m64" EXTRACFLAGS="-m64 -fPIC"

GSL - GNU Scientific Library

The install is straight forward with the usual configure but on 64 bit machine, you will need to add the CCFLAGS and LDFLAGS as showed below

% ./configure --prefix=$OPTSTAR                                    ! default bits 
% ./configure --prefix=$OPTSTAR CFLAGS="-m32 -fPIC" LDFLAGS="-m32" ! 32 bits
% ./configure --prefix=$OPTSTAR CFLAGS="-m64 -fPIC" LDFLAGS="-m64" ! 64 bits
% make
% make install

Python

The below were fine with verison 2.7.1

% setenv BASECFLAGS "-m32 -fPIC"
% setenv CXXFLAGS "-m32 -fPIC"
% setenv LDFLAGS "-m32"
%  ./configure --prefix=$XOPTSTAR

On a mixed architecture, I had to modify the generated pyconfig.h as the use of VA_LIST_IS_ARRAY would get pythong to crash.

1069c1069
< //#define VA_LIST_IS_ARRAY 1
---
> #define VA_LIST_IS_ARRAY 1

For the 64 bits version, please substitute -m32 with -m64  as follows

% setenv BASECFLAGS "-m64 -fPIC"
% setenv CXXFLAGS "-m64 -fPIC"
% setenv LDFLAGS "-m64"
%  ./configure --prefix=$XOPTSTAR

Note: The default compilation (without using the environment variable setting) may succeed but binding with ROOT and other package will fail and require -fPIC and additionally, it is best to have in all configurations -m32/-m64 specified explicitely.

MySQL-python

Build is straight forward in principle i.e.

% python setup.py build 
% python setup.py install 

but

• on a mixed 32/64 bits platform, the 32 bits will need to be persuaded by
  • editing site.cfg and setting mysql_config to the 32 bit version (likely /usr/lib/mysql/mysql_config ) - you can be sure of this as the return value for --cflags would have -m32
  • Add the following code in setup_posix.py
    

    65,68d64
    <     for i in range(len(extra_compile_args)):
    <         if extra_compile_args[i] == '-m32':
    <             extra_link_args += ['-m32']
    <
    

    I am sure there are other more ellegant ways but this works fine.
     
• If your default MySQL distribution resides in $OPTSTAR, you will also need to edit the site.cfg script before running the setup script and modify the mysql_config variable accordingly. This will prevent running into symnbol loading conflicts later (usually happening because a third party product would, in this case, use one version of MySQL and STAR code another). 

fastjet

The basic compilation requires

./configure --prefix=$OPTSTAR CXXFLAGS="-m32 -fPIC -fno-inline" CFLAGS="-m32 -fPIC -fno-inline" LDFLAGS="-m32"

For the 64 bits, replace -m32 by -m64. -fno-inline is needed still to circuvnet a gcc bug with inlining.

boost

Certainly, the most helpful reference was this boost reference. But those are not immediate instructions. Here is what you will need to do:

% ./bootstrap.sh --prefix=$XOPTSTAR

In any cases, this will build a few 64 bits executables on a 32/64 bits machine but don't panic yet ... To build, use one of the below (as appropriate):

% ./bjam cflags="-m64 -fPIC" cxxflags="-m64 -fPIC" linkflags="-m64 -fPIC" address-model=64 threading=multi architecture=x86 stage
or
% ./bjam cflags="-m32 -fPIC" cxxflags="-m32 -fPIC" linkflags="-m32 -fPIC" address-model=32 threading=multi architecture=x86 stage

I am sure you already see the problem  - on AMD processors, you may have a different "arhitecture" so we cannot give you the exact instruction to use here. Possible architectures are x86, x86_amd64, x86_ia64, amd64 or ia64.

When you are done wth compiling, execute nearly the same command but instead of "stage" use

% ... install --prefix=$XOPTSTAR

and you will be hopefully done.

unuran

This package follows a typical install i.e.

% ./configure --prefix=$XOPTSTAR CXXFLAGS="-m32 -fPIC" CFLAGS="-m32 -fPIC" LDFLAGS="-m32"
or
% ./configure --prefix=$XOPTSTAR CXXFLAGS="-m64 -fPIC" CFLAGS="-m64 -fPIC" LDFLAGS="-m64"

% make
% make install

This will allow ROOT to build the TUnuran classes.

LHAPDF-6.1.6

This package is not straight forward to install. Use the usual initial setup i.e.
 

./configure --prefix=$XOPTSTAR CXXFLAGS="-m32 -fPIC" CFLAGS="-m32 -fPIC" LDFLAGS="-m32"

or

./configure --prefix=$XOPTSTAR CXXFLAGS="-m64 -fPIC" CFLAGS="-m64 -fPIC" LDFLAGS="-m64"

then the usual

% make

However, before make install, modify  lhapdf-config and add -m32 -fPIC (-m64 -fPIC for 64 bits platform) to cflags and -m32 (-m64) to the ldflags i.e.
 

40c40
< test -n "$tmp" && OUT="$OUT -m32 -fPIC -I${prefix}/include "
---
> test -n "$tmp" && OUT="$OUT -I${prefix}/include "
46c46
< test -n "$tmp" && OUT="$OUT -m32 -L${exec_prefix}/lib -lLHAPDF"
---
> test -n "$tmp" && OUT="$OUT -L${exec_prefix}/lib -lLHAPDF"

as the configure will not do that and hence, not generate a config script suitable for a a mix 32/64 bits.

 
vim

Configure is standard, use a minimal option set and features=big as below

% ./configure --enable-pythoninterp=yes --enable-perlinterp=yes --enable-cscope --with-features=big \
--prefix=$XOPTSTAR CFLAGS="-m64 -fPIC" CXXFLAGS="-m64 -fPIC" LDFLAGS="-m64"

or

./configure --enable-pythoninterp=yes --enable-perlinterp=yes --enable-cscope --with-features=big \--prefix=$XOPTSTAR CFLAGS="-m32 -fPIC" CXXFLAGS="-m32 -fPIC" LDFLAGS="-m32"

then the usual make and make install.





 

CERN libraries

The STAR simulation framework will require the CERN libraries to be installed. This will likely be the most problematic portion of the STAR software installation as there is little support for the CERNLib nowadays (so, you must rely on existing supported versions).

• Information about the CERN libraries can be found here with the appropriate download area.
  If you do not find a version of the CERNlib for your platform, well ... you are out of luck.
  If you have the same OS as BNL and do not want to spend to much time, copy from BNL and move on.
   
• If you experience problems with the 64 bit versions and this was not copied from BNL, consider one of the following:
  • 64bit CERN libs showed to work at first were taken from DESY colleagues from this link.
  • CERNLib are also available in rpm format for Fedora EL4/EL5 (SL4/SL5). See this link for more information.

Note however that (for example) a generic Linux distribution or an older Linux version based distribution may work for respectively a different flavor of Linux or a more recent of Linux.

Building ROOT in STAR

Building ROOT in STAR

How to build ROOT at BNL and other sites (support documentation)

Some of the above help is similar than what you will find in the PDSF page mentioned above.
Older help version could be found from the revision tab.

ATTENTION

• BEWARE that the target for 64 bit machine is specific to your platform. We use linuxx8664gcc at BNL.
   
• Several patches are available for this revision (see at the bottom as usual). All sources were repackages in Additional software components as root_v5.34.30_2-star.source.tar.gz  (this revision was last made on 2014/06/25). Taking the repacked tar ball is the preffered and recomended approach. Alternatively, you may download the root code from root.cern.ch (the source tar balls are available on their site) but extra steps/actions noted in orange below are needed.
   
• This revision supports Qt4 in its latest incarnation i.e. qt-everywhere-opensource-src-4.8.7 (see Additional software components for more information) - qt 4.4 was left for use for SL5.3
   
• We substantially dropped several OS support since the previous release and this still applies to 5.34.09 i.e.:
  • cygwin not tested at all
  • Mac OSX not tested
  • Solaris and True64 are no longer available to us as build platform and hence, we cannot conclude
  • Linux revisions prior to 5.3 (Boron) are no longer supported.
     
• Only Pythia6 is supported in this version (and is default)

The build

The build is in several steps ; we will assume for this example that we are building root version 5.22.00 ; the % sign is used for the Unix prompt.

1. Go to the $ROOT directory, create a directory named after the version i.e. 5.22.00,  and unpack the tar ball using
   
   % cd $ROOT
   % mkdir 5.34.30_2
   % cd 5.34.30_2
   % setenv ROOTB `pwd`
   
   One note : the ROOTB environment variable will be used throughout this help but is not used by the STAR environment. It only serves the purpose of finding back easily the base tree for the ROOT version you are trying to install.
   
   If you are a STAR user, you can also find a copy of the package in AFS at the location
   /afs/rhic/star/common/General/Sources/  (again, this is the reocmmended approach to avoid extra complications).
   % tar -xzf root_v5.34.30_2-star.source.tar.gz
   
   Remember to remove the archive tar.gz file after unpacking. It is not helpful to keep it around.
   ALL required files (including the STAR specific configurations) are part of our repackaged sources. Starting from ROOT 5.34, no other means are recommended (you may still download the original codes from the ROOT Web site but the STAR specific configs and patches will not be present).
    
2. The structure is not ready ... yet ... You still have to create the OS dependent tree structure we use to allow concurrent platform support in addition of the two layers (with debugging, without debugging information). This is done in a one-does-it-all script.
   
   % $STAR/mgr/MakeRootDir.pl
   
   This command needs to be executed ONCE on EACH supported platform.
   IMPORTANT NOTE: $STAR environment variable needs to be defined here but it case it is not, the latest version available in /afs/rhic.bnl.gov/star/packages/dev/mgr is the safest to use (because it is the latest available). You need to access this script from the AFS path (or make a local copy) if you start building a site from scratch as well (ROOT needs to be installed before $STAR).
    
3. The next step will give an example on how to build it under the linux platform. The above assumes that root is installed in AFS (to make it generic) and that .@sys allows to separate the OS/compiler flavors (which is the case, but not valid for off-site NFS resident tree structure where you will have to specifically use .$STAR_HOST_SYS instead of .@sys ).
   
   % cd $ROOTB/.$STAR_HOST_SYS/rootdeb
   % setenv ROOTSYS `pwd`
   % $STAR/mgr/fixrootmk -unlink
   % ./configure linux --build=debug ... remaining options ... +
   
   % cd $ROOTB/.$STAR_HOST_SYS/root
   % setenv ROOTSYS `pwd`
   % $STAR/mgr/fixrootmk -unlink
   % ./configure linux  ... remaining options ... +
   
   The above examples are both building the standard root distribution and option. This IS NOT what we are using in STAR and the above table shows the default options currently in use. You MUST of course add the --build=debug flag where appropriate as shown above while using that table (i.e. the main point of the two commands as shown is that the rootdeb tree is build with the configure command --build=debug while the optimized version is not).
   
   Notes:
   1. You MUST issue a similar command on ALL platform you are supporting before proceeding to the next step.
   2. You need to run ./configure ONCE only. In case of any further updates, you should NOT repeat this action.
      
    
4. Execute the following script before compiling
   % $STAR/mgr/fixrootmk
   This will check the configuration file for special tags (in the old versions, it would check and fix CERNLIB for example).
   Note that the same command prior with -unlink moved the system.rootrc aside before the configure and make local copies of some of the files which should not be shared between multiple-platforms. At the end of compilation, we will ask you to execute the script again to re-install the system.rootrc and similar files (i.e. make use of the global one for all platforms).
   
   
5. Now we are ready to compile. To do this, simply use the below sequence where you have configured the package.
   % gmake
   
   Notes:
   1. the gmake step may display messages like No such file or directory .
      Please, ignore since this is normal and will trigger the proper directory tree creation.
   2. All is compiled AND INSTALLED. There is NO NEED to use make install.
   3. For those compiling on AFS, if gmake (or make) keeps calling the reconfigure script over and over again, you have two solutions. Either 'sleep 1 && touch Makefile'  and type make again or use 'make --assume-new=Makefile'. The former will reset the Makefile timestamp to a second later of the last command executed and the later tell make to consider the timestamp of the Makefile file as if it has just been modified. You may also try a 'make clean' and try again.
    
6. As noted in a previous step, at the end, finalize the build by doing
   % $STAR/mgr/fixrootmk -fix
   
   and at least once
   
   % cd $ROOT
   % test -e 5.34.30 && rm -f 5.34.09 && ln -s 5.34.09_1 ./5.34.09
   
   This last command link your current build tree to the official 5.34.09. If you had a previous patch level version, it will be replaced at that stage and the new build will become active. There should be NO need to rebuild root4star for incremental patch levels.
   You are now done.

Table of options

Notes:

1. You should use $OPTSTAR at remote sites having a local /opt/star and $XOPTSTAR if you provide support over AFS.
2. If you build Python support, please refer to You do not have access to view this node for quick tests to see if it works
3. To enable the build with QT4, you MUST define QTROOT to point to the QT4 directory prior to executing the configure script (the build will oherwise be silent on missing it)
4. For SL5.3, this version of ROOT was built with the additional option --enable-builtin-pcre
5. If you intend to build ROOT with another compiler revision, ad the options --with-cc=`which gcc` --with-cxx=`which g++`.
6. --disable-xrootd should be used wherever you do not have Xrootd support (or do not need it)
7. If you change the Python verison on your system, note that the PyROOT binding will need to be remade. However, this will not be as easy as typing make as specific version-keyed includes will be needed. You may however try something like
   % modify python2.4 python2.7 ./config/Makefile.config
   in the $ROOTSYS directory and type make afterward (this will work). Ultimately, you may also re-build in your private area and
   % cp -fp lib/libPyROOT.so $ROOTSYS/lib
   % cp -fp lib/ROOT.py* $ROOTSYS/lib
   i.e. no need to compile in-place.

List of updated files 

The below list is provided for convenience but you should send a note if you note ANY differences from this list and what was packaged for use by remote sites. In the below,  A=added, P=patched, U=updated:

     
P    root/cint/cint/inc/G__ci.h
P    root/math/vc/Module.mk
P    root/bindings/pyroot/Module.mk

Typical patched codes The following codes are tweaked

STAR codes

Basic

The STAR code and libraries follows a structure and policy described in Library release structure and policy. Changes in each version is described in Library release history.

Installing the core STAR software is (should be) as simple as getting a full set of code for a given library, unpacking it into $STAR_PATH (default is $STAR_ROOT/packages as described in Setting up your computing environment) and issuing the following commands (in our example, we use STAR_LEVEL=SL09b with revision 1 from that library).

% cd $STAR_PATH
% mkdir SL11d && cd SL11d
% cvs co -rSL11d asps mgr QtRoot StarVMC StRoot kumacs pams StarDb StDb OnlTools
% starver SL11d 
% cd $STAR
% cons

And wait ... until all is done. This will actually build the non-optimized version of our libraries.

Modifiers

To build the optimized version, use

% setenv NODEBUG yes

before you execute the starver command. If you need both, you will hence have to build twice per libraries.

To build using alternate compilers, you will need to run the setup command before running cons. For example, for the icc compiler you will need an appropriate version of $OPTSTAR and

% setup icc

and for an alternate version of gcc (and pending the fact you have the specific version installed), you will need to use something similar to

% setup gcc 4.5.1

Note that those syntax assumes specific path for gcc (installed in either /opt/gcc/$version or $OPTSTAR/alt/) while icc is expected to have an setup program located in $GROUP_DIR (as intelcc.csh) defining the paths.

Finally, on kernel supporting it, you can also switch to an alternate bits environment like this

% setup 64bits

and get the compilation proceed with the 64 bits support.

Tips

Excluding problematic directories

Sometimes, our libraries get packed with the "Pool" (user space) libraries and their support may vary. To be on the safe side, exclude several of them from compilation by setting the environment variable SKIP_DIRS before executing cons.

% setenv SKIP_DIRS "StEbyePool StHighptPool StAngleCorrMaker StSpinMaker StEbyeScaTagsMaker 
StEbye2ptMaker StDaqClfMaker StFtpcV0Maker StStrangePool GeoTestMaker"

Special levels

The levels pro, new and dev are special levels as described in Library release structure and policy. pro is especially relevant as if no level is specified, the STAR login will revert to whatever pro is set to be. You may then do something like the below (again, our example assumes the default library is SL07b - please adjust accordingly).

% cd $STAR_PATH
% test -e pro && rm -f pro
% ln -s SL11d ./pro

Your default STAR library is then set for your site.

Post installation

Soft-links for compatibility

In /opt/star (or equivalent), $STAR and $ROOT/$ROOT_LEVEL, run the script $STAR/mgr/CreateLinks. This will create a few compatibility links to support additional (tested and proven to work) OS / sysname version.

Other

The list below is not exhaustive. Note that most does not need to be done and we separate the action items in two categories

To be checked or modify

• Consult the script site_pre_setup.csh and site_post_setup.csh . ANY site specific envrionment variables (such as STAR_ROOT, OPTSTAR and so on) can be set there. See the Setting up your computing environment used in STAR for more information on defaults (you should have read it before reaching this section though ;-)  ).
   
• Please, modify your local template login files ${GROUP_DIR}/templates/ files and make sure the GROUP_DIR directory is properly reflected. Note that ANY other variables set for your site should be set in site_pre_setup.csh
   

Optional - for large site and Tier centers wanting network independence.

• Database connection setup - a local database setup could be used and is expected to be located in ${STAR_PATH}/conf/dbLoadBalancerLocalConfig_${SITE}.xml . Modify this file (based on existing template) to point to the local database service.
  To setup a local database service, consult You do not have access to view this node for getting a local database service going.
  Note: Attention, if you run on the Cloud, those instructions may be useful (a service can be created per VM)
   
• Setting up SUMS - only for site intending to submit jobs locally
  SUMS will need a local batch system and a local config (ask the scheduler Hypernews).
  We support LSF, PBS, Condor and SGE.
    
• FileCatalog - not needed unless you are a Tier1 center, holder of files in our distributed data management system.
  A copy of the FileCatalog can be set to the local site and/or a multi-master setup can be done.
  To first order, your local FileCatalog replica would minimally allow querrying the FC without the load we see at BNL (but is not that useful). A multi-master would allow full implementation of our data-management system and scheme including file transfer between sites and multi-site transfers.
  But for the API to know about the local setup, you need to modify the unique local configuration file $STAR_PATH/conf/Catalog.xml and add your configuration in. The key is SITE name="XXXX" (add a new block for your site and please, pass on the information back to the Catalog maintainer). Make sute this value matches the value of $SITE set in group_env.csh .
   
• [*** data management / multi-site transfer instructions needed ***]

LDAP scalability

The scripts below "dump" the entry list to a local file so that network lookups are un-necessary while LDAP is still used to centralized the account information. They were written and deployed at PDSF and passed as-is to the RCF team for consideration.

create_files in /etc/cron.hourly

Formatting scripts

groups.pl

users.pl

rterm

For users

Basic help follows:

Syntax is

  % rterm [Options] [NodeSpec] [UserName]

Currently implemented options are :

  -i         interactive mode i.e. do not open an xterm but use slogin
             directly to connect.
  -p port    use port number 'port' to connect
  -x node    Exclude 'node' from possible node to connect to. May be
             a comma separated list of nodes.
  -funky     pick a color randomly
  -bg color  pick specific color

The 'NodeSpec' argument may be a node name (specific login to a given node)  or a partial node name followed by the '+' sign (wildcard). For example,

  % rterm rcas6+

will open a connection on the least loaded node amongst all available rcas6* nodes. By default, this command will determine the appropriate wildcarded node specification for your GroupID. However, if this help is displayed when the command '% rterm' is used, contact the RCF support  team (your group ID is probably not supported by this script).

The 'UserName' argument is also optional. If unspecified, it will revert to the current user ID.

Finally, you may modify the xterminal layout by using the following environment variables

  TERM_BKG_COLOR     sets the xterm background color
  TERM_OPTIONS       sets any other xterm options

For administrators

• If you are using LSF: add a static resource inter to the nodes which will be accessible interactively. Of course, we assume that access is restricted/allowed to nodes on your cluster via the usual mechanism (allow/deny files)
  Condor: have interactive_node matchmaking keyword in place
• be sure to have the lsload (LSF) or condor_status (Condor) programs installed and configured on your gatekeeper.
• perl is used for the -funky option (random color) but is not necessary.

Autobuild & Code Sanity

This page will contain results from code-sanity tools i.e. tools checking for code syntax, memory leak, profiling etc ... and tips on how to correct those problems.
Most, if not all, pages here are generated automatically. If you encountered an error in the formatting, please send me a note ...

• AutoBuild compilation builds
• cppcheck and coverity dashboards (coverity not completed yet)
• Valgrind tool
• How to run: cppcheck, coverity

AutoBuild

• Official OS version (SL7.3, optimized and non optimized, 32 bits) Linux Last AutoBuild compilation status).
• Official OS version (SL7.3 non optimized only for 64bits)  Linux 64 bits AutoBuild status

AutoBuild is a tool which automatically builds/compiles our libraries based on what has been committed during the day. From time to time, please consult those pages and remember the following guideline for code submission into the repository :

• Code submitted to cvs must compile and run under Linux platform.
• Compilation should not give any serious warnings.

AutoBuild is self documented, just type AutoBuild.pl -h for all command line options.

AutoBuild behavior may be altered using directives. Directives are tags (key=value pairs) to be placed in a configuration file. The configuration file name is .ABrc_$^O where $^O is the string returned by the perl command % perl -e 'print $^O'. The following directives are allowed. If you have multiple values for one directive, each should be on a separate line.

• SKIP_DIRS=XXX
  Excludes XXX from compilation, this directive will be passed to cons (the build system in STAR). This works only for directories.
  Example:

  SKIP_DIRS=StEbyePool

   
• CO_DIRS=XXX   
  This directive forces the full tree to be CVS check out. Normally, AutoBuild only updates trees (which means that new sub-directories would not appear by defaults). This directive only works on directories and should not be used extensively as it is resource intensive (re-check out the entire directory).
  Example:

  CO_DIRS=StRoot/StEEmcPool

   
• EX_DIRS=XXX     - This directives indicates that the item XXX (already present and deployed) should be ignored from CVS UPDATE. Practically, AutoBuild will rename the item XXX to a temporary name and restore. This directive works on files or directories.
  Example:

  EX_DIRS=StChain

   
• #
  Not a directive per-se, this is indicate the start of a comment (the rest of the line is ignored)
  Examples: 

  SKIP_DIRS=StJetmaker  # Disabled 2008/07/23
  # the following lines are for blabla
  

   

Valgrind

Valgrind is a flexible tool for debugging and profiling Linux-x86 executable. The tool consists of a core, which provides a synthetic x86 CPU in software, and a series of "skins", each of which is a debugging or profiling tool.For more information valgrind, check its documentation page and the quick How to use valgrind.

 

Quick guide to run cppcheck

% cppcheck --force --enable=information StRoot/ 

In his mode, you will see all errors as it finds it.

 

Quickly running coverity

Nothing quick about it here as you will need to compile your code and generate the report ... But here is what you will need to do

• Add the coverity path to your PATH 

  % set COVPATH=/afs/rhic.bnl.gov/x8664_sl6/app/coverity-7.6.0
  % set path=($COVPATH/bin $path)
  % rehash

• Run a build using cov-build command. In the below command, the coverity output and results go into to a temporary directory named $MYDIR/covtmp

  % cov-build --dir $MYDIR/covtmp cons -k

• "Analyze" the results (i.e. it will relate errors with one another)

  % cov-analyze --dir $MYDIR/covtmp --all -j auto

• Finally, generate the HTML report

  % cov-format-errors --dir $MYDIR/covtmp  --filesort -x  --title "Bla" --html-output $MYDIR/html

STACK_USE

Coverity uses an arbitrary limit for the checking of a single variable size allocation. This default is 10000 bytes and far too small. In C/C++ only the total number of bytes for the whole stack is relevant and this is provided by the value of stacksize (provided by the shell limit command). Our dashboard will analyze the report using a max_total_use_bytes equal to the return value stacksize from the limit command and 1/10th of that value for a single variable allocation. Any excess will be reported as a defect but may not be.

Practically, add similar options to cov-analyze command:

• --checker-option STACK_USE:max_total_use_bytes:10240000  --checker-option STACK_USE:max_single_base_use_bytes:1024000
• ...

Was also available ...

Profiling, Valgrind Results

Profiling using gprof

Library release history

Documents and information for current STAR Software releases

 ---------------------  
       SL02e (SL02e)    ROOT_LEVEL 3.02.07 pp and AuAu 200GeV real data production  
       SL03f (SL03f)    ROOT_LEVEL 3.05.04 ppMinBias 2001/2002 rerun, pp200 Pythia  
       SL03h (SL03h)    ROOT_LEVEL 3.10.01 dAu and pp data reproduction  
       SL04d (SL04d)    ROOT_LEVEL 3.10.01 62 GeV production (continue with SL04e) 
       SL04e (SL04e)    ROOT_LEVEL 3.10.01 AuAu 200 & 62 GeV Hijing production  
       SL04f (SL04f_a)  ROOT_LEVEL 3.10.01 dAu 200GeV reproduction  
       SL05a (SL05a)    ROOT_LEVEL 4.00.04 AuAu200, productionMinBias  
       SL05c (SL05c)    ROOT_LEVEL 4.00.04 AuAu200 production  
       SL05d (SL05d_1)  ROOT_LEVEL 4.00.04 CuCu 200&62 Gev production  
       SL05e (SL05e)    ROOT_LEVEL 4.00.04 pp200 MC production  
       SL05f (SL05f_3)  ROOT_LEVEL 4.04.02 pp run 2005 production  
       SL05h (SL05h)    ROOT_LEVEL 4.04.02 SL 3.0.5  
       SL06b (SL06b_1)  ROOT_LEVEL 4.04.02 cucu 200GeV production  
       SL06d (SL06d_2)  ROOT_LEVEL 4.04.02 MC prod for SVT&SSD review 
       SL06e (SL06e)    ROOT_LEVEL 4.04.02 pp 2006 production  
       SL06f (SL06f_2)  ROOT_LEVEL 4.04.02 MC production for TUP  
       SL06g (SL06g_2)  ROOT_LEVEL 5.12.00 MC production for TUP, SL4.4  
       SL07a (SL07a_3)  ROOT_LEVEL 5.12.00 MC production, SL4.4  
 old-> SL07b (SL07b_2)  ROOT_LEVEL 5.12.00 CuCu reproduction, SL4.4  
 pro-> SL07c (SL07c_3)  ROOT_LEVEL 5.12.00 CuCu reproduction, pp200 pythia,SL4.4  
       SL07d (SL07d_2)  ROOT_LEVEL 5.12.00 auau 200GeV, run 2007,SL4.4  
 new-> SL07e (SL07e)    ROOT_LEVEL 5.12.00 auau 200GeV, run 2007,SL4.4  
 dev-> DEV              ROOT_LEVEL 5.12.00 SL4.4  
.dev-> .DEV             ROOT_LEVEL 5.12.00 
 -------------------------------------------------