HLT Linux Cluster User's Guide

- Howe to get an account on HLT Farm

1. Get an account to the STAR Protected Network. See instructions at https://drupal.star.bnl.gov/STAR/comp/onl/accessing-star-protected-network
2. Once you can access STAR online gate way stargw.starp.bnl.gov, send an email to starsupport@bnl.gov (cc to me: kehw@bnl.gov) to request your access to xeon-phi-dev.starp.bnl.gov, which is the entry point to the HLT cluster.

- How to login HLT cluster
The STAR HLT cluster sites in a local network, which has an entry point to the STAR online network.
To reach the cluster, one need to jump a few time via ssh.
1. (if you are outside of BNL) Connect to the RCF gateway rssh.rhic.bnl.gov or cssh.rhic.bnl.gov.
The RCF NX service is recommended for better graphics performance.
https://www.racf.bnl.gov/docs/services/nx

2. From the RCF gateway or the NX node, connect to the STAR gateway: stargw.starp.bnl.gov
You must have a STAR online enclave account to be able to do this.
https://drupal.star.bnl.gov/STAR/comp/onl/accessing-star-protected-network

3. From the STAR gateway, connect to the HLT cluster entry point: xeon-phi-dev.starp.bnl.gov (aka L409)
Now you are on STAR HLT cluster.

The ssh ProxyJump or ProxyCommand is recommended to jump through multiple hosts
https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#Passing_Through_One_or_More_Gateways_Using_ProxyJump

- Computing Environment

   * SL 7.3
   * No STAR environment pre-configured.
   * If you need the STAR environment, add the following lines in you ~/.cshrc
--------------------------------------------------------------------------

- NFS mounted home dir: /star/u/[user name]
Size: 1.3TB, RAID 5, no quota yet, but please use your home dir wisely

 - NFS mounted Storage
/net/l401/data/scratch[1,2]
/net/l402/data
/net/l403/data
/net/l404/data
* 10+TB each, available on all HLT nodes.

- Ceph storate
On all nodes
/hlt/cephfs

 - Condor Queue: Intel Xeon E5-2670 @ 2.50GHz
condor submitter: xeon-phi-dev.starp.bnl.gov (aka l409), l404

 - Intel Xeon Phi coprocessor 7120P
l405-l408, l410-l417, two cards per node
l409(xeon-phi-dev) one card
In order to use the coprocessor, Intel C/C++ compiler is needed, which can be found at
/opt/intel_17
Documentation for Intel compiler and libraries can be found at
https://software.intel.com/en-us/intel-parallel-studio-xe-support/documentation
More information about Xeon Phi KNC (1st generation, x100 family) can be found at
https://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-codename-knights-corner

- PGI Compiler
The PGI compiler community edition has been installed in
/software/pgi

One need the following settings to use the compilers

% setenv PGI /software/pgi
% set path=(/software/pgi/linux86-64/18.10/bin $path)
% setenv MANPATH "$MANPATH":/software/pgi/linux86-64/18.10/man
% setenv LM_LICENSE_FILE /software/pgi/license.dat:"$LM_LICENSE_FILE" 

---------------------------------------------------------------------------------
Contact Person:
Hongwei Ke
kehw@bnl.gov
631-344-4575