Getting Started on Kure

Table of Contents

Introduction

System Information

Getting an Account

Logging In

Directory Spaces

Mass Storage

Development and Application environment

Applications available

Software Development Tools

Compiling

Submitting Jobs

Transferring Files

Additional Help

Introduction

  • The Kure cluster is a Linux-based computing system available to researchers across the campus. With more than 1840 computing cores across 230 blade servers and a large scratch disk space, it provides an environment that can accommodate many types of computational problems. The blades are interconnected with a high speed Infiniband network, making this especially appropriate for large parallel jobs. 48 or more GB of memory on each host also provides room for large memory jobs. (Note: “”Kure” is named after the island on the North Carolina coast and is pronounced like Madam Curie’s last name.)

System Information

  • Login node : kure.unc.edu 8 cores @ 2.80 GHz Intel EM64T with 2x4M L2 cache (Model X5560), 48 GB memory
  • Compute nodes: 122 blade servers, each with 8-cores 2.80 GHz Intel processors, 2x4M L2 cache (Model X5560), and 48 GB memory for a total of 976 processing cores, two similar 8 core blades with 96 GB memory, and three more blades with 192 GB memory and 24 total cores.
  • Operating System : RHEL 5.6 (Tikanga)
  • Shared Filesystem : 40 TB IBM GPFS, 85 TB NetApp NFS, 42 TB “/netscr” storage
  • Interconnect: Infiniband 4x QDR
  • Resource management is handled by LSF, through which all jobs are submitted for processing

Getting an Account

You can visit the Onyen Services page, then click on the Subscribe to Services button and select Kure Cluster.

If you attempt to subscribe for a Kure account through Onyen Services you may see the following message

  You are ineligible for the following services for the reasons cited:
   KillDevil Cluster: missing '(LIVE|EXCH)' prerequsite.
   Kure Cluster: missing '(LIVE|EXCH)' prerequsite.

In this case, you should send an email to research@unc.edu requesting an account on Kure. In the email, please do indicate that you were not able to subscribe for a Kure account through Onyen Services and include the following information in your request:

  • Onyen
  • Your “@email.unc.edu” email address
  • Full name
  • Campus address
  • Campus phone number (if any) and number where you can be reached while running jobs
  • Department you are affiliated with (the one relevant to the work you will do on Kure)
  • Faculty sponsor’s (PI) name (and onyen if known) if you are not a faculty member
  • A description of the work you expect to do on Kure

You will receive an email notification once your account has been created.

Note that the phone number you provide needs to be one where we can reliably and routinely reach you, and the email address onyen@email.unc.edu must work. If you are running jobs on the cluster, you must check your UNC email of record (Onyen@email.unc.edu) periodically while your jobs are running. If your job creates problems for other users, we will attempt to contact you via phone and email. Depending on the situation, if we are not able to reach you immediately, it is likely that we will kill your jobs. Remember that these are shared resources, and we do our best to preserve a working environment for everyone. If your jobs repeatedly affect others and we are unable to reach you, your account on the cluster may be suspended until you contact us to reach resolution. The best way to ensure ongoing accurate contact information is to keep your email and phone number up-to-date in the UNC directory http://dir.unc.edu.

Logging In

Once you have an account on Kure, use ssh to connect to kure.unc.edu and login with your Onyen:

  ssh kure.unc.edu

In order to connect to Kure from an off-campus location, a connection to the campus network through a VPN client is required. Your very first login will begin normally with your onyen and password. Then your session will run ssh-keygen for you. Accept the defaults by pressing at each prompt. No password is necessary for this key generation. If this environment is not established correctly, jobs will fail with “permission denied” messages. (See the KillDevil FAQ for details.)

Even though the Kure cluster has many compute nodes, you never actually login to any of them. Instead, you login as above to the cluster. A successful login takes you to “login node” resources that have been set aside for user access. On the login node, you will edit and compile your code, and then you will use the LSF job scheduler to submit your code to the compute nodes for processing. Interactive use on the login node must be restricted to compiling and debugging. Other processes running on the login node are subject to immediate termination by the system administrators.

Directory Spaces

NAS home space

Your home directory will be in /nas02/home/o/n/(the “o/n/” are the first two letters of your onyen, of course.). Your home directory has a 10 GB soft limit and a 15 GB hard limit.

Work/Scratch Space

  • Netscratch scratch space:

For Netscratch scratch space, /nestcr, job performance is best in instances where users have many small data files less than 64K in size. Netscratch scratch space is an NFS-mounted file system and thus shared by all Kure users as well as the users of other Research Computing systems.

The following apply to both scratch spaces:

  • Scratch space uses standard UNIX permissions to control access to files and directories. By default other users in your group (graduate students, faculty, employees) have read access to your Netscratch directory. You can easily remove this read permission with the “chmod” command.
  • A policy has been established for cleaning out files. Scratch file deletion will be enforced with files older than 21 days being removed. Any file not used or modified in the last 21 days will be deleted.
  • Scratch space is a shared, temporary work space. Please note that scratch space is not backed up and is, therefore, not intended for permanent data storage. See the “Mass Storage” section below about how to store permanent data.
  • Note it is a violation of research computing policy to use artificial means, such as the “touch” command, to maintain unused files in the scratch directory beyond their natural lifetime. Violators will be warned and repeat violators are subject to loss of privileges and access. This is a shared resource, please be courteous to other users.

What follows are suggested “best practices” to keep in mind when using scratch space on the Kure cluster:

  • Try to avoid using “ls –l” and use “ls” with no options instead.
  • Never have a large number of files (>1000) in a single directory.
  • Avoid submitting jobs in a way that will access the same file(s) at the same point(s) in time.
  • Limit the number of processes performing parallel I/O work, SAS work, or other highly intensive I/O jobs.

Mass Storage

The Mass Storage system (also known as StorNext or /ms) is intended for archiving files and storing very large files. Files located in mass storage are not accessible to jobs running in LSF. Mass storage is not to be used as a work directory or as a backup location for local disk drives, operating systems, or software. In general, files that change often or directories with more than a thousand files in them will cause performance problems and consume tape resources. The PC backup software provided by UNC might be an alternative solution rather than having to copy your PC files to mass storage.

Mass Storage is similar to an ordinary disk file system in that it keeps an inode (for recording data location, etc.) and data blocks for each file. Files can be moved in and out of mass storage by using simple UNIX commands such as “cp” and “mv” or by using sftp/scp. As the Mass Storage system is optimized for archiving data, your programs should not directly read or write from the Mass Storage system. Instead copy your data from “~/ms” to scratch space such as “/netscr/”.

If you are routinely storing large numbers of small files (more than several hundred files at a time) in Mass Storage, you should “tar” or “zip” those smaller files into one tarball or zip file outside of mass storage and then move that tarball or zip file to mass storage. You are not required to compress the tarball or zip file since the mass storage tape drive hardware will compress your data. Reducing the number of individual small files will help the overall performance of the StorNext Mass Storage system. See the more detailed list of things to avoid.

To access Mass Storage from Kure, type:

  cd ~/ms

Any files in the scratch space that you wish to save, can be moved to the mass storage preferably in tar or zip format.

IMPORTANT NOTE:

If you are currently doing any large moves or copies of data (to or from mass storage) we hope you will use the following LSF command:

  bsub -q ms cp /netscr/onyen/text.txt /ms/home/o/n/onyen

This bsub command, issued with the “-R ms” parameter, will submit your copy or move job to a host with very good connectivity to the mass storage system. We expect these hosts to handle multiple data moves well, therefore removing this burden from the login node.

Development and Application environment

The environment on Kure is presented as modules. The basic module commands are

  module [ add | avail | help | list | load | initadd | unload | initrm | show ]

When you first log in you should run

  module list 

And the response should be

  1) null 

To add a module for this session only, use “module add [application]” where “[application]” is the name given on the output of the “module avail” command.

To add a module for every time you login, use “module initadd [application]”. This does not change your current session, only later logins.

Please refer to the Help document on modules for further information.

Applications available

Applications used by many groups across campus have been compiled and made available on Kure. To see the full list of applications currently available run

  module avail

To see the matrix of applications and clusters visit the Application Matrix.

Software Development Tools

  • Intel Compiler Suite
    • v. 11 for Fortran77, Fortran90, C and C++, Math Kernel Library
  • Portland Group (PGI) Compiler Suite
    • v.10.3 for Fortran77, Fortran90, C and C++
  • GNU (GCC) Compiler Suite
    • V.4.1.2 for Fortran77, Fortran90, C and C++
  • MPI for parallel programming via OFED v.1.34.1
    • MVAPICH
    • OPENMPI
  • Totalview Debugger
  • Job scheduler
    • LSF v. 7 update 6

Note that one and only one of these MPI environments can be loaded at a given time. If you load the mvapich or openmpi environment for a compiler, that will also load its standard environment. For example, you don’t do “module load pgi” and then “module load mvapich_pgi”. The latter command loads all the PGI compiler commands including the mpi ones. If you do not want the MPI compiler commands you use just the “module load pgi”.

Compiling

The available MPI compiler commands are:

  mpicc
  mpif77
  mpif90
  mpiCC
  mpicxx

Once you have the default compile module added to your environment with the command

  module initadd mvapich_[intel|pgi|gcc] (specify which compiler to use)

or

  module initadd openmpi_[intel|pgi|gcc)

then both your compiles and your job submissions will have available all the appropriate environment including man pages, paths, libraries, include files and any required environment variables.

Submitting Jobs

Once you have decided what software you need to use, added those packages to your environment using modules or you have successfully compiled your serial or parallel code, you can then submit your jobs to run on Kure. We use LSF (Load Sharing Facility) software to schedule and manage jobs that are submitted to run on Kure.

To submit a job to run, you will need to use the LSF bsub command as shown below. LSF submits jobs to particular job queues you specify. So in your “bsub” command, you will need to specify the queue in which the job is to run by using the “-q” bsub option. If you don’t provide a queue explicitly, the job will be given to the “week” queue by default. If you do not add a “-n” to your bsub statement, it is assumed the job should run on 1 CPU. There is a default limit of a total of 64 CPU/User across the cluster.

A short description of the queues available to users in the Kure cluster can be found below. You can also use the “bqueues” command to list the properties of a specific queue. For example, you could type "bqueues -l debug" (that’s a lower case “-l” for “long listing”) to find out more about the debug queue. Additional queues may be added as need dictates. All queues share a common fairshare allocation policy that governs which PENDing job will be dispatched next based on the recent runtime history of each user or group with jobs PENDing to run. The following queues are available on Kure:

Queue name Job Duration CPU Range/Job Default # CPUs Total # CPUs/User across all jobs in queue
Int 2 hr. 1-2 1 2
Debug 2 hr. 1-32 1 32
week (default queue) 7 days 1-32 1 32
Patrons Unlimited unlimited 1 unlimited
p96 unlimited unlimited 1 unlimited
Idle unlimited unlimited 1 unlimited
Staff n/a n/a n/a n/a
ms 7 days 1 1 12

A list of resources defined for a given node can be seen in the last column of output of the following command:

  lshosts | more

Submitting Batch Jobs:

The general form of syntax for submitting a serial batch job is:

  bsub [- bsub options] executable [- executable options]

To learn more about bsub and bsub options:

  man bsub

Submitting Interactive Jobs

Submitting jobs to the interactive queue requires a specific bsub “-q” option (int) and the bsub option “-Ip” as in the “bsub” command shown below:

  bsub -q int -Ip my_interactive_job

For example, the command below will give you an interactive bash shell session:

  bsub –q int -Ip /bin/bash
Note. Jobs submitted to the interactive queue “int” will not run on Kure login nodes, but run on a compute node instead.

Submitting Parallel Jobs:

The general form of syntax for submitting a parallel batch job, in this example “mycode,” is:

  # be sure you have the appropriate module added for use at login
module initls
# response: one of the MPI suites such as mvapich_intel, openmpi_intel, etc.
  bsub -n "< number CPUs >" -o out.%J -e err.%J \
   mpirun ./mycode 

Another way to submit a parallel job is to create a file (for example, call it mycode.bsub) with the following lines in it

  #### run_mycode ####
  #BSUB -n "< number CPUs >"
  #BSUB -e err.%J 
  #BSUB -o out.%J
  mpirun ./mycode
  ##### end of run_mycode ####

and then do the following command at the Kure prompt:

  bsub < mycode.bsub

For more basic LSF commands refer to the Help document on LSF (Load Sharing Facility).

Monitoring and Controlling Jobs:

You can check the status of your submitted LSF jobs with the command “bjobs”. The output of that command will include a Job ID, the status of your job (typically “PEND” or “RUN”), the queue to which you submitted the job, the job name, and other information. Additional details can be obtained with:

  bjobs -l [JobID]

To display the stdout and stderr of an unfinished job, use the “bpeek” command:

  bpeek [JobID]

If you need to kill/end a running job, use the “bkill” command:

  bkill [JobID]

Where JobID is the LSF job ID displayed with the “bjobs” command.

Finally, if you don’t provide an output file to LSF (“-o filename” in the bsub command) LSF will send email to your email address when the job finishes, whether it completes successfully or not (unless you are running in the interactive queue of course). For any jobs that produce large amounts of output you should use the “-o filename” bsub option.

Note
Jobs running outside the LSF queues will be killed. The logon privileges of users who repeatedly run jobs outside of the LSF queues will be suspended.

Transferring Files

It is likely that you will need to transfer files between your campus computer systems that have AFS and Kure. You will need to use the “sftp” or “scp” command to move your files since Kure does not have any /afs/isis file systems available. The command “sftp” works similarly to the popular “ftp” command but is more secure. From your host UNIX/Linux or Mac computer terminal window, type:

  sftp @kure.unc.edu

Enter your Onyen password and you will be presented with the sftp prompt. Use the “put” and “get” commands to transfer files, as you would do with standard ftp.

To use the “scp” command follow this example to get a file, named “temp.txt”, from Kure and store it in your local computer’s “/tmp” directory with the same file name. You will be prompted for your password.

  scp @kure.unc.edu:/netscr//temp.txt /tmp/

You can also copy a whole directory. The following command will recursively copy the whole directory “/tmp/temp_dir/” from your local computer to Kure, and place it in the “/netscr//” directory with the name “temp_dir”:

  scp -r /tmp/temp_dir/ @kure.unc.edu:/netscr//

IMPORTANT NOTE:

If you are currently doing any large moves or copies of data (to or from mass storage) we hope you will use the following LSF command:

  bsub -R ms cp /netscr//text.txt /ms/home/o/n//

This bsub command, issued with the “-R ms” parameter, will submit your copy or move job to a host with very good connectivity to the mass storage system. We expect these hosts to handle multiple data moves well, therefore removing this burden from the login node.

Additional Help

Be sure to check the Research Computing home page for information about other resources available to you.

We encourage you to attend a training session on Getting Started on Kure and other related topics. Please refer to the Research Computing Training site for further information.

If you have any questions, please feel free to call 962-HELP or submit an Online Web Ticket.