General

The Online Compute Cluster (ONLXX) is composed of 30 nodes, each containing 6 - 2TB drives. We dedicated 4 drives per node to the distributed storage system Ceph. The storage system uses data replication (3 copies) and offers a total of 80TB. Ceph offers a POSIX interface that can be mounted similiar to an NFS mount. If you want to use you the CephFS and mount it to your system at the STAR experiment follow the steps below.

Before you begin

You must be root in order to complete the full suite of tests.

Mounting CephFS requires kernel version 2.6.34 or later. At this point Scientific Linux 6.6 only supports/releases kernel version 2.6.32. Scientific Linux 7 is shipping with kernel version 3.10.0 at this time.
If upgrading your OS to Scientific Linux 7 is out of the question, these intructions provide steps to install a later version kernel and if needed, a pointer to instructions for reconfiguring OpenAFS as the kernel modules will need to be reloaded.

1. Get the ceph.repo repository file

# /usr/bin/wget -N -P /etc/yum.repos.d/ http://www.star.bnl.gov/~mpoat/ceph/ceph.repo

2. Install the Ceph Packages

# yum install ceph

3. Get the CephFS Client Key

# /usr/bin/wget -N -P /etc/ceph/ http://www.star.bnl.gov/~mpoat/ceph/client.cephfs

4.  Get the elrepo rpm

# wget http://www.star.bnl.gov/~mpoat/ceph/elrepo-release-6-6.el6.elrepo.noarch.rpm
# yum install elrepo-release-6-6.el6.elrepo.noarch.rpm

Ensure the /etc/yum.repos.d/elrepo.repo file has the [elrepo-kernel] portion set to enabled=1

5.  Install the new kernel-lt and kernel-lt-devel

# yum install kernel-lt kernel-lt-devel

6. Edit grub

Open up /boot/grub/grub.conf with your favorite text editor. With my experience, you will more than likely see the first kernel in the to be the newly installed 3.10.xx kernel. However, the setting for default may be set to 1 when it needs to be set to 0. This number indicates which kernel will be booted from top to bottom starting at 0. In most cases you will need to set default=0.

7. Reboot the machine

Ensure the new kernel has loaded

# uname -r

NOTE: If your node requires OpenAFS installed. Please refer to https://drupal.star.bnl.gov/STAR/blog/jeromel/afs-not-starting-module-fail-load

8. Ensure Ceph kernel modules are loaded

# cat /proc/modules | grep ceph

You should get output that looks something like...

libceph 148556 1 ceph, Live 0xffffffffa072c000
libcrc32c 12644 2 libceph,xfs, Live 0xffffffffa061c000


If you get no return then you do not have the Ceph modules loaded. Ensure the correct kernel is installed.

9. Mount the CephFS

Create a directory of choice, in this case created /ceph

# mkdir /ceph

You mount the cephfs using this command

# mount -t ceph cephmon01.starp.bnl.gov:6789:/ /ceph -o name=cephfs,secretfile=/etc/ceph/client.cephfs

Or edit /etc/fstab with your favorite editor and at the end of the file create a new line like below

 
cephmon01.starp.bnl.gov:6789:/ /ceph ceph name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime 0 2

Then Enter:

# mount -a

We no longer recommend mounting Ceph as permanent mount. Due to issues with multiple clients mounting simultaneously and metadata not syncing it is best to use AutoFS. See below for steps
 

Edit the file /etc/auto.misc and at the line below to the end of the file

ceph       -fstype=ceph,name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime     cephmon01.starp.bnl.gov:6789:/

Now edit the file /etc/auto.master and add the line below to the end of the file

/mnt    /etc/auto.misc --timeout 60

Restart the autofs service

# service autofs restart

10. Check to see if Ceph will mount

# ls /mnt/ceph

Check

# mount

Should see..

130.199.60.224:6789:/ on /ceph type ceph (name=cephfs,key=client.cephfs)

If you would like to have Ceph mounted on /ceph create a softlink using

# ln -s /mnt/ceph /ceph

References

http://ceph.com/docs/master/cephfs/kernel/