Home » Detector Sub-systems » FGT » FGT. . . . . . . . . . I N S T R U C T I O N S

HPSS I/O

Updated on Wed, 2009-06-03 14:10. Originally created by balewski on 2008-03-18 07:38. Under:

 

 

HTAR :  save/restore  private directory (large set of small files) in HPSS

For saving all files in directory 'aaa' on disk to your HPSS target bbb/ccc.tar and keep log file in ~/0x/ddd.log1 (needed if you want to kill/quit the window) do: 
 
cd aaa
du -hs .
htar -cf bbb/ccc.tar file1 file2 fileX* ... >& ~/0x/ddd.log1 &
(wait until job is done, for 50GB make take severa hours)
hsi (goes directly to HPSS, be careful!)
?ls -l  bbb/
?quit (exit hsi)
 
You should see 2 files : ccc.tar & ccc.tar.idx, the size of file ccc.tar should be close to the size of your oryginal directory.
The .tar file shows up immediately but sinking is NOT finished until you see  .idx file.
The log file should contain the string: HTAR: HTAR SUCCESSFUL , no warnings.
However the most reliable way to verify the storage did not failed is to:
 
retrieving files from HPSS  to disk directory eee do:
 
mkdir eee
cd eee
htar -xf bbb/ccc.tar  >& ~/0x/ddd.log2 &
 
Other options of HTAR command are described here: 
       http://drupal.star.bnl.gov/STAR/comp/sofi/hpss/htar

Known problems:

WARNING: htar_PreallocateSpace: HPSS OUT-OF-SPACE error preallocating
**** bytes for file=[/home/salur/myfile.tar]
This is a directory with 350G. Do I have to divide my files into  smaller.
 there is a 60GB max file size limit [when creating tarballs],   Another
limit to be aware of, however, is an 8GB limit on member files
 

 

 Mass restoring of ~100GB files from HPSS using Data Carusel

Prepare input file xxx.hpss with list of source +destination paths for all files  you need, one line per file

Execute the command (at any location at rcas6nnn)

 hpss_user.pl -f xxx.hpss

Wait 10-1000 minutes fro the files to show up at your destination (if all paths are correct)

 

 

 

»