Using your Grid CERT to sign or encrypt Emails

• first export your certificate from your Web browser. This is explain on the how to Export your key pair for use by Globus grid-proxy-init page. In summary:

• Find in your browser certificate management interface an 'export' or 'backup' option. THis interface varies from browser to browser and from Email client to Email client. We have checked only in Thudenrbird as an Email client and inventoried possible location for browser-based tools.
  
  • Internet Explorer: "Tools -> Internet Options -> Content"
  • Netscape Communicator as a "Security" vutton on the top bar menu
  • Mozilla: "Edit -> Prefercences -> Privacy and Security -> Certificates"
    
  Thudenrbird: "Tools -> Options -> Privacy -> Securiry -> View Certificate"
• The file usually end-up withe xtension .p12 or .pfx.
  ATTENTION: Although the backup/export process will ask you for a "backup password" (and further encrypt your CERT), please guard this file carefully. Store it OFF your computer or remove the file once you are done with this process.
• After exporting your certificate from your Web browser, you will need to re-import it into your Mail client. Let's assume it is Thuderbird for simplicity.
• FIRST:
  Verify you have the DOEGrids Certificate Authority already imported in your Mail client and/or upload them.
  Note that the DOEGrid Certificate Authority is a subordinate CA of the ESnet CA ; therefore the ESnet CA root certificate should also be present. To check this
  
• Go to "Tools -> Options -> Privacy -> Security -> View Cretificate"
• Click on the "Authorities" tab
• You should see both "DOEGrids CA 1" and "ESnet Root CA 1" under an "Esnet" tree as illustrated in this first picture
  
  
  
  
• Be certain the "DOEGrids CA 1" is allowed to allow mail users. To do this, select the cert, click Edit. A window as illustrated in the next picture should appear. Both This certificate can indentify Web sites and This certificate can identify mail users should be checked.
  
  
• If you DO NOT SEE those certificate authorities, you will need to import them.
  
• Do so by downloading the doe-certs.zip attached at the bottom of this document, unzip . Two files should be there
  
• Load them using the "Tools -> Options -> Privacy -> Security -> View Cretificate -> Aurthorities -> Import" button.
  
• A similar window as displayed above will appear and you will need to check box at least This certificate can identify mail users.
• Now, import your certificate.
  
• Use the "Tools -> Options -> Privacy -> Security -> View Cretificate -> Your Certificate" menu and click "Import"
• A file browser will appear, select the file you have exported from your browser. It will ask you for a password. You will need to use the smae password you used during the export phase from your Web Browser.
  
• Click OK
• Your are set to go ...
  

• If you want a remote partner to send you encrypted messages, you MUST send first a digitally signed Email so your certificate public part could be imported into his/her Email client Certificate Manager under "Other People's". When done for the first time, THuderbird will ask you to set a certificate as default certificate ; the interface and selection is straight forwardso we will not detail the steps ...
• If you want to send an encrypted message to a remote partner, you MUST have his public part imported into your Email client and then select the "Encrypt This Message" option in the Security drop down menu of Thunderbird.
  
• Whenever a certificate expires, DO NOT remove from you Certificate Manager. If so, you will no longer be able to read / decrypt old encrypted Emails.

SGE Job Manager patch

• Missing environment variables definition
  
  • In the BEGIN section check if $SGE_ROOT, $SGE_CELL and the commands ($qsub, $qstat, etc) are defined properly
  • in the SUBMIT, POOL and CLEAR sections, locate the line

    $ENV{"SGE_ROOT"} = $SGE_ROOT;
    

    and add the line

    $ENV{"SGE_CELL"} = $SGE_CELL;
    

• Bug finding the correct job id when clearing jobs
  • in the CLEAR section, locate the line

    system("$qdel $job_id > /dev/null 2 > /dev/null");

    and replace for the following block

    $ENV{"SGE_ROOT"} = $SGE_ROOT;
    $ENV{"SGE_CELL"} = $SGE_CELL;
    $job_id =~ /(.*)\|(.*)\|(.*)/;
    $job_id = $1;
    system("$qdel $job_id > /dev/null 2 > /dev/null");

• SGE Job Manager modifies definitions of both the standard output and standard error file names by appending .real. This procedure fails when a user specifies /dev/null for either of those files. The problem happens twice - once starting at line 318

    #####
    # Where to write output and error?
    #
    if(($description->jobtype() eq "single") && ($description->count() > 1))
    {
      #####
      # It's a single job and we use job arrays
      #
      $sge_job_script->print("#\$ -o "
                             . $description->stdout() . ".\$TASK_ID\n");
      $sge_job_script->print("#\$ -e "
                             . $description->stderr() . ".\$TASK_ID\n");
    }
    else
    {
        # [dwm] Don't use real output paths; copy the output there later.
        #       Globus doesn't seem to handle streaming of the output
        #       properly and can result in the output being lost.
        # FIXME: We would prefer continuous streaming.  Try to determine
        #       precisely what's failing so that we can fix the problem.
        #       See Globus bug #1288.
      $sge_job_script->print("#\$ -o " . $description->stdout() . ".real\n");
      $sge_job_script->print("#\$ -e " . $description->stderr() . ".real\n");
    }
 

and then again at line 659:

      if(($description->jobtype() eq "single") && ($description->count() > 1))
      #####
      # Jobtype is single and count>1. Therefore, we used job arrays. We
      # need to merge individual output/error files into one.
      #
      {
        # [dwm] Use append, not overwrite to work around file streaming issues.
        system ("$cat $job_out.* >> $job_out");
        system ("$cat $job_err.* >> $job_err");
      }
      else
      {
        # [dwm] We still need to append the job output to the GASS cache file.
        #       We can't let SGE do this directly because it appears to
        #       *overwrite* the file, not append to it -- which the Globus
        #       file streaming components don't seem to handle properly.
        #       So append the output manually now.
        system("$cat $job_out.real >> $job_out");
      }

• The snipped of code above is also missing a statement for the standard error. At the end instead of:

        #       So append the output manually now.
        system("$cat $job_out.real >> $job_out");
      }

it should read:

        #       So append the output manually now.
        system("$cat $job_out.real >> $job_out");
        system("$cat $job_err.real >> $job_err");
      }

• Additionally, if deployed in a CHOS environment, the job manager should be modified with the following additions at line 567:

    $ENV{"SGE_ROOT"} = $SGE_ROOT;
    if ( -r "$ENV{HOME}/.chos" ){
      $chos=`cat $ENV{HOME}/.chos`;
      $chos=~s/\n.*//;
      $ENV{CHOS}=$chos;
    }

gridftp update for VDT 1.3.9 or VDT 1.3.10

for VDT 1.3.9 (which is what I got with OSG 0.4.0) in the OSG/VDT directory, do:

pacman -get http://vdt.cs.wisc.edu/vdt_139_cache:Globus-Updates

This nominally makes your VDT installation 1.3.9c, though it didn't update my vdt-version.info file accordingly -- it still says 1.3.9b

for VDT 1.3.10, similar installation should work:

pacman -get http://vdt.cs.wisc.edu/vdt_1310_cache:Globus-Updates

BNL

LBL

MIT

USP

Installation

Installing the batch system (SGE)

Instructions to install SGE

1. Download from the SGE web site
2. gunzip and untar the file
3. cd to the directory

1. In the batch system server (in our case, STAR1)

1. Create the SGE_ROOT directory. In our case, mkdir /home/sge-root. This directory HAS to be available in all the exec nodes
   
2. copy the entire content of the installation directory to the SGE_ROOT directory
3. add the lines bellow to your /etc/services file
   sge_execd        19001/udp
   sge_qmaster     19000/tcp
   sge_qmaster     19000/udp
   sge_execd        19001/tcp
4. cd to the SGE_ROOT directory
5. Type ./install_qmaster
6. follow the instructions in the screen. In our case, the answers to the questions were:
1. Do you want to install Grid Engine under an user id other than >root< (y/n) >> n
2. $SGE_ROOT = /home/sge-root
   
3. Enter cell name >> star
4. Do you want to select another qmaster spool directory (y/n) [n] >> n
5. verify and set the file permissions of your distribution (y/n) [y] >> y
6. Are all hosts of your cluster in a single DNS domain (y/n) [y] >> y
7. Please choose a spooling method (berkeleydb|classic) [berkeleydb] >> classic
8. You can change at any time the group id range in your cluster configuration. Please enter a range >> 20000-21000
9. The pathname of the spool directory of the execution hosts. Default: [/home/sge-root/star/spool] >> [ENTER]
10. Please enter an email address in the form >user@foo.com<. Default: [none] >> [PUT YOUR EMAIL]
11. Do you want to change the configuration parameters (y/n) [n] >> n
12. We can install the startup script that will start qmaster/scheduler at machine boot (y/n) [y] >> y
    
13. Adding Grid Engine hosts. Do you want to use a file which contains the list of hosts (y/n) [n] >> n
14. Host(s): star1 star2 star3 star4 ...... (ADD ALL HOSTS THAT WILL BE CONTROLED BY THE BATCH SYSTEM)
15. Do you want to add your shadow host(s) now? (y/n) [y] >> n
16. Scheduler Tuning. Default configuration is [1] >> 1
17. Proceed with the default answers until the end of the script
7. You have installed the master system. To make sure the system will start at boot time type
   ln -s /etc/init.d/sgemaster /etc/rc3.d/S95sgemaster
   ln -s /etc/init.d/sgemaster /etc/rc5.d/S95sgemaster

2. Install the execution nodes (including the server, if it will be a exec node). This needs to be done in ALL exec nodes

1. add the lines bellow to your /etc/services file
   sge_execd        19001/udp
   sge_qmaster     19000/tcp
   sge_qmaster     19000/udp
   sge_execd        19001/tcp
2. cd to your SGE_ROOT directory
3. type ./install_execd
1. Answer the question about the SGE_ROOT directory location
2. Please enter cell name which you used for the qmaster. >> star
3. Do you want to configure a local spool directory for this host (y/n) [n] >> n
4. We can install the startup script that will start execd at machine boot (y/n) [y] >> y
5. Do you want to add a default queue instance for this host (y/n) [y] >> n (WE WILL CREATE A QUEUE LATER)
6. follow the default instructions until the end
4. You have now installed the master system. To start the system at boot time. type
   ln -s /etc/init.d/sgeexecd /etc/rc3.d/S96sgeexecd
   ln -s /etc/init.d/sgeexecd /etc/rc5.d/S96sgeexecd

3. Install a default queue to your batch system

1. type qmon
   It opens a GUI window where you can configure all the batch system.
2. Click in the buttom QUEUE CONTROL
3. It opens another screen with the queues you have in your system
4. Click on ADD
5. Fill the instructions. See the file sge-admin.pdf for instructions. It is very simple.
   

Installing GANGLIA

• gmond - the monitoring system. Should be installed in ALL machines in the cluster
• gmetad - the gathering information system. Should be installed in the machine that will collect the data (in our case, STAR1)
• the web front end. This is nice to have but not essential. It creates a web page, like this one, with all the information in your cluster. You should have a web server running in the collector machine (STAR1) for this to work
• rrdtool - this is a package that creates the plots in the web page. Necessary only if you have the web frontend.

1. In each machine in the cluster

1. Install the gmond package (change the name to match the version you are installing)
   rpm -ivh ganglia-gmond-3.0.1-1.i386.rpm
2. edit the /etc/gmond.conf file. The only change I made in this file was
   cluster {
     name = "STAR"
   }
3. Type
   ln -s /etc/init.d/gmond /etc/rc5.d/S97gmond
   ln -s /etc/init.d/gmond /etc/rc3.d/S97gmond
   /etc/init.d/gmond stop
   /etc/init.d/gmond start

2. In the collector machine (STAR1)

1. Install the gmetad, web and rrdtool packages (change the name to match the version you are installing)
   rpm -ivh ganglia-gmetad-3.0.1-1.i386.rpm
   rpm -ivh ganglia-web-3.0.1-1.noarch.rpm
   rpm -ivh rrdtool-1.0.28-1.i386.rpm
2. edit the /etc/gmetad.conf file. The only change I made in this file was
   data_source "STAR" 10 star1:8649 star2:8649 star3:8649 star4:8649 star5:8649
3. Type
   ln -s /etc/init.d/gmetad /etc/rc5.d/S98gmetad
   ln -s /etc/init.d/gmetad /etc/rc3.d/S98gmetad
   /etc/init.d/gmetad stop
   /etc/init.d/gmetad start

MonaLISA

1. Create a monalisa user in your master computer and its home directory
2. cd to the monalisa installation dir
3. type ./install.sh
4. Answer the following questions:
1. Please specify an account for the MonALISA service [monalisa]: [ENTER]
1. Where do you want MonaLisa installed ? [/home/monalisa/MonaLisa] : [ENTER]
2. Path to the java home []: [enter the path name for your java distribution]
3. Please specify the farm name [star1]: [star]
4. Answer the next questions as you wish
5. Make sure that Monalisa will run after reboot by typing:
   ln -s /etc/init.d/MLD /etc/rc5.d/S80MLD
   ln -s /etc/init.d/MLD /etc/rc3.d/S80MLD
6. You need to edit the following files in the directory /home/monalisa/MonaLisa/Services
1. ml.properties
   MonaLisa.ContactName=your name
   MonaLisa.ContactEmail=xxx@yyyy.yyy
   MonaLisa.LAT=-23.25
   MonaLisa.LONG=-47.19
   lia.Monitor.group=OSG, star (Note that we are being part of both OSG and STAR groups)
   lia.Monitor.useIPaddress=xxx.xxx.xxx.xxx (your IP)
   lia.Monitor.MIN_BIND_PORT=9000
   lia.Monitor.MAX_BIND_PORT=9010
7. Need to tell MonaLisa that I am using SGE as a batch system. For this, edit the Service/CMD/site_env file and add
   

   SGE_LOCATION=/home/sge-root
   export SGE_LOCATION
   SGE_ROOT=/home/sge-root
   export SGE_ROOT

Requesting a GRID certificate

Installing OSG

1. Make sure pacman is installed. For this I had to update python to a version above 2.3. Pacman is a package management system. It can be downloaded from here
2. create a directory at /home/grid. This is where I installed the grid stuff. Thid directory needs to be visible in all the cluster machines
3. I typed

   export VDT_LOCATION=/home/grid
   cd $VDT_LOCATION
   pacman -get OSG:ce
   

   I  just followed the log and answered the questions.
   

pacman -get http://www.cs.wisc.edu/vdt/vdt_136_cache:Globus-SGE-Setup

$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2119 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2119 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2135 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2135 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2136 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2136 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2811 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2811 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2812 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2812 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 2912 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 2912 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 7512 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 7512 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 8443 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 8443 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 19000 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 19000 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 19001 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 19001 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p udp -d $GLOBALIP --dport 20000:65000 -j DNAT --to $STAR1
$filter -t nat -A PREROUTING -p tcp -d $GLOBALIP --dport 20000:65000 -j DNAT --to $STAR1

setenv GLOBUS_TCP_PORT_RANGE "60000 65000"
setenv GLOBUS_HOSTNAME="stars.if.usp.br"

export GLOBUS_TCP_PORT_RANGE="60000 65000"
export GLOBUS_HOSTNAME="stars.if.usp.br"

GridCat and making things to work...

• Authentication test
• Hello world test
• Batch submition (depends on your batch system)
• submit a job
• query the status of the job
• canceling the job
• file transfer (gridFtp)

How to turn authentication and hello world to green?

"/DC=org/DC=doegrids/OU=People/CN=Leigh Grundhoefer (GridCat) 693100" XXXX
"/DC=org/DC=doegrids/OU=People/CN=Bockjoo Kim 740786" XXXX

How to turn the batch system test green

• mis-ci-functions
• This file, located at $VDT_LOCATION/MIS-CI/etc/misci/ is responsible for checking your system basically every 10 minutes and extract information about your cluster. It uses the batch system to grab information. Of course, it does not work with SGE. Replace the file with the version 0.2.7, located here. Please check if your version is newer than this one before replacing...
• sge.pm
• This file is located at $VDT_LOCATION/globus/lib/perl/Globus/GRAM/JobManager/
• Please check the following
• In the BEGIN section
• if $SGE_ROOT, $SGE_CELL and the commands ($qsub, $qstat, etc) are defined properly
• In the submit section
• Locate the line
  
• $ENV{"SGE_ROOT"} = $SGE_ROOT;
• add the line
• $ENV{"SGE_CELL"} = $SGE_CELL;
• The same in the pool section
• In the clear section
• locate the line  system("$qdel $job_id >/dev/null 2>/dev/null");
• replace for the following
•     $ENV{"SGE_ROOT"} = $SGE_ROOT;
      $ENV{"SGE_CELL"} = $SGE_CELL;
      $job_id =~ /(.*)\|(.*)\|(.*)/;
      $job_id = $1;
      system("$qdel $job_id");

Making the gridFTP to work

  globus-gatekeeper       2119/tcp        # Added by the VDT
  gsiftp                  2811/tcp        # Added by the VDT
  gsiftp2                 2812/tcp        # Added by the VDT
  gsiftp                  2811/udp        # Added by the VDT
  gsiftp2                 2812/udp        # Added by the VDT

globus-url-copy -dbg file:///star/u/suaide/gram_job_mgr_13594.log gsiftp://stars.if.usp.br/home/star/c

• replace file /etc/xinetd.d/gsiftp for this one

  service gsiftp
  {
       socket_type = stream
       protocol = tcp
       wait = no
       user = root
       instances = UNLIMITED
       cps = 400 10
       server = /auto/home/grid/vdt/sbin/vdt-run-gsiftp2.sh
       disable = no
  }

•  restarted xinetd
• modified the file /hom/grid/globus/etc/gridftp.conf to

  # Configuration for file the new (3.9.5) GridFTP Server
  inetd 1
  log_level ERROR,WARN,INFO,ALL
  log_single /auto/home/grid/globus/var/log/gridftp.log
  hostname "XXX.XXX.XXX.XXX"

•  XXX.XXX.XXX.XXX is the IP of the gateway for the outside world

WSU