HLT Linux Cluster User's Guide


HLT Linux Cluster User's Guide

- Howe to get an account on HLT Farm

  1. Get an account to the STAR Protected Network. See instructions at https://drupal.star.bnl.gov/STAR/comp/onl/accessing-star-protected-network
  2. Once you can access STAR online gate way stargw.starp.bnl.gov, send an email to starsupport@bnl.gov (cc to me: kehw@bnl.gov) to request your access to xeon-phi-dev.starp.bnl.gov, which is the entry point to the HLT cluster.

- How to login HLT cluster
The STAR HLT cluster sites in a local network, which has an entry point to the STAR online network.
To reach the cluster, one need to jump a few time via ssh.
1. (if you are outside of BNL) Connect to the RCF gateway rssh.rhic.bnl.gov or cssh.rhic.bnl.gov.
The RCF NX service is recommended for better graphics performance.

2. From the RCF gateway or the NX node, connect to the STAR gateway: stargw.starp.bnl.gov
You must have a STAR online enclave account to be able to do this.

3. From the STAR gateway, connect to the HLT cluster entry point: xeon-phi-dev.starp.bnl.gov (aka L409)
Now you are on STAR HLT cluster.

The ssh ProxyJump or ProxyCommand is recommended to jump through multiple hosts

- Computing Environment

   * SL 7.3
   * No STAR environment pre-configured.
   * If you need the STAR environment, add the following lines in you ~/.cshrc

source /net/l402/data/fisyak/STAR/packages/.DEV2/setupDEV2.csh
setup 64b
starver TFG18m

- NFS mounted home dir: /star/u/[user name]
Size: 1.3TB, RAID 5, no quota yet, but please use your home dir wisely

 - NFS mounted Storage
* 10+TB each, available on all HLT nodes.

- Ceph storate
On all nodes

 - Condor Queue: Intel Xeon E5-2670 @ 2.50GHz
condor submitter: xeon-phi-dev.starp.bnl.gov (aka l409), l404

 - Intel Xeon Phi coprocessor 7120P
l405-l408, l410-l417, two cards per node
l409(xeon-phi-dev) one card
In order to use the coprocessor, Intel C/C++ compiler is needed, which can be found at
Documentation for Intel compiler and libraries can be found at
More information about Xeon Phi KNC (1st generation, x100 family) can be found at

- PGI Compiler
The PGI compiler community edition has been installed in

One need the following settings to use the compilers

% setenv PGI /software/pgi
% set path=(/software/pgi/linux86-64/18.10/bin $path)
% setenv MANPATH "$MANPATH":/software/pgi/linux86-64/18.10/man
% setenv LM_LICENSE_FILE /software/pgi/license.dat:"$LM_LICENSE_FILE" 

Contact Person:
Hongwei Ke