Getting Started on Cedar/Cypress

Introduction

This document describes how to use the Research Computing cluster called Cedar/Cypress. The intended use of this cluster is for UNC-CH affiliated researchers to do research-related computing.

System information

  • Login node: cedar.isis.unc.edu, 8 CPUs, 1500 MHz Itanium2 Rev.1 Processor, 8 GB memory;
  • Computing Node: cypress.isis.unc.edu, 128 CPUs, 1600 MHz Itanium2 Rev.1 Processor, 512 GB memory;
  • Resource management is handled by LSF (Load Sharing Facility), through which all computational jobs are submitted for processing.

Logging in

To obtain and/or manage your account on our servers, please visit the Onyen Services page. Once you have an account on the cluster, you can login using ssh to connect to Cedar, the login node of the SGI Altix cluster: 

ssh cedar.isis.unc.edu

Telnet access is not allowed. Upon a successful login your home directory will be:

/nas/uncch/home/o/n/onyen

For example, the home directory for the onyen mason is: 

/nas/uncch/home/m/a/mason

You will note that this is not your usual home directory. There is no access to “/afs” space from Cedar/Cypress. Therefore this home directory is independent of your “/afs” home directory. To have access to the same aliases and other features of your usual login to other network systems you will need to copy your profile or other configuration files to your Cedar/Cypress home directory.

Your Cedar/Cypress home directory is accessible from other computing systems that mount “/nas”. This home directory is primarily intended for your environment configuration information and should not be used to store data or other large amounts of files.

Scratch/work space

Cedar/Cypress has access to several scratch spaces. When you login to Cedar you will have available the following scratch file systems:

/scr_local 400 GB directly mounted on Cedar, not available on Cypress
/netscr 4.5 TB NFS mounted, shared with Cypress and other clusters
/scr 4.5 TB NFS mounted on Cedar from Cypress.  Fast IO for Cypress jobs
              This is default work space for Gaussian /scr/APPS_SCRDIR
/scr1 2.1 TB NFS mounted on Cedar from Cypress.  Fast IO for Cypress jobs
/bigscratch 6.5 TB NFS mounted on Cedar from Cypress.  Just more scratch space.

Scratch space is not backed up and is, therefore, not intended for permanent data storage.

The “/scr_local” space is intended only for interactive jobs which may need some local work space.

“/netscr” space is not directly mounted on Cypress so it will be significantly slower to access than the three scratch spaces “/scr”, “/scr1”, and “/bigscratch” for batch jobs running on Cypress.

mkdir /netscr/mason

In the above example, the mason directory was created for the person whose Onyen is “mason.” You and your jobs can access “/netscr” from both Cedar and Cypress and from the Emerald cluster. Since this storage is shared with many other users, please remove any files that are not associated with currently running jobs. If this space fills completely, all jobs that attempt to write there will hang or die. To avoid that, the administrators run a program that detects when scratch usage is at a high level. When that threshold is reached the oldest files not associated with running jobs are deleted until a lower threshold level is reached. This process is automatic and deleted files cannot be retrieved. This scratch space is also not backed up.

The “/scr” space should be your most commonly used scratch space for jobs running on Cypress. It is fast, has lots of room, and you can check on progress from Cedar for its files. It is the default location for Gaussian work files (“/scr/APPS_SCRDIR”). Nothing in “/scr” is backed up.

“/scr1” is intended mostly as a test file system for jobs which may write far more than expected.

While you are still debugging your code or when you have a run which may produce huge amounts of data and you don’t want to risk filling shared scratch space, have your job write to “/scr1”. If it fills it will only kill your test job or other test jobs. Then when you are confident that the job will not write far more than you expected, change its output to “/scr” or “/bigscratch”. Nothing in “/scr1” is backed up.

“/bigscratch” is larger than “/scr” and is intended for the really big output files. Most users should not need to employ “/bigscratch” at all. We don’t even think about backing up “/bigscratch”.

Mass storage

The ITS mass storage system (also known as SAM-FS or “/ms”) is intended to be used for archiving files and storing very large files, files that are too large to fit in your AFS quota. SAM-FS is similar to an ordinary disk file system in that it keeps an inode (for recording data location, etc.) and data blocks for each file. For the user of mass storage, this file system appears to be a subdirectory of the user's AFS home directory. Files can be moved in and out of mass storage by using simple Unix commands such as “cp” and “mv” or by using sftp/scp. As the Mass Storage system is optimized for archiving data, your programs should not directly read from or write to the Mass Storage system.

To access mass storage from Cedar, type:

cd ~/ms

Any files to be saved from your finished jobs in the scratch space, “/netscr/[onyen]”, “/scr”, or “/bigscratch” should be moved to mass storage or other permanent storage.

For more information about Mass Storage, click this link

Software

It is important to notice that ALL APPLICATIONS WERE BUILT LOCALLY on the Cedar/Cypress cluster because they have NO AFS access. This means that the “ipm” command has no effect on your work on Cedar/Cypress. All available software (including compilers) built for the cluster has been put in the directory “/opt”. To view, type: 

ls /opt

They include Intel compilers, Gaussian, Amber, GROMACS, NWChem, NAMD, PMEMD, etc.

For more information about these packages, click this link

Compiling serial code

There are two C/C++ and FORTRAN 77/90/95 compilers available on the SGI Altix LINUX64 cluster. One set is from GNU and the other is from Intel. The GNU compilers (gcc, g77, c++/g++) are installed at “/usr/bin”, whereas the Intel compilers ( ifc/ifort, icc) are at “/opt/intel/cc/10.1.008/bin”. We highly recommend that you use the Intel compilers unless you have good reasons for doing otherwise.

For example, to compile with the Intel compiler one serial FORTRAN 77 code, in the file “source.f”, type: 

ifort -O -o source.x  source.f

The executable generated is placed in “source.x”.

For more information about issues related to compilers, click this link.

Compiling MPI and OpenMP codes

OpenMP, MPI and hybrid (OpenMP/MPI) codes in the FORTRAN 77/90/95 and C/C++ programming languages can be run in the shared-memory environment of the SGI Altix cluster. These capabilities are available with the Intel compilers.

For example, to compile an OpenMP FORTRAN code in, “source_openmp.f”, using the Intel compiler, type: 

ifort -O -openmp -o source_openmp.x source_openmp.f

The executable, “source_openmp.x”, is thereby generated.

To compile an MPI Fortran code with the Intel compiler, type:

ifort -O -o source_mpi.x source_mpi.f -lmpi

To compile an MPI C code with the Intel compiler, type:

icc -O -o source_mpi.x source_mpi.c -lmpi

To compile an MPI C++ code with the Intel compiler, type:

icc -O -o source_mpi.x source_mpi.C -lmpi -lmpi++

The above MPI examples generate the executable “source_mpi.x”.

For compiling C code with the Intel Compilers, it may be necessary to change some of the default library settings.

  • First, the following default library paths should be set as:

1. "bin_path" : [ "/opt/intel/cc/10.1.008/bin/","/usr/bin","/bin" ]

2. "library_path" : [ "/opt/intel/cc/10.1.008/lib","/usr/lib" ]

3. "include_path" : [ "/opt/intel/cc/10.1.008/include" ]

  • Next, the linker library path should be set appropriately:

1. For bash shell users:

export LD_LIBRARY_PATH=/opt/intel/cc/10.1.008/lib:/usr/lib:${LD_LIBRARY_PATH}

2. For tcsh/csh shell users:

setenv  LD_LIBRARY_PATH /opt/intel/cc/10.1.008/lib:/usr/lib:${LD_LIBRARY_PATH}

For more information about issues related to compilers, click this link.

Intel Math Kernel Libraries

Intel Math Kernel Libraries are available at Cedar/Cypress, whose current version is 10.1.01.

The libraries have been installed at “/opt/intel/mkl/10.1.010/lib/64”

For instance, to compile a FORTRAN code with LAPACK libraries, “code.f”, the correct syntax is as follows: 

ifc -O -o code.exe code.f -L/opt/intel/mkl/10.1.010/lib/64/ -lmkl -lmkl_lapack \
        -L/opt/intel/fc/10.1.008/lib  -lguide

For more information about its capability, usage, manuals, visit its homepage.

Submitting serial/parallel jobs

We use LSF software to schedule and manage jobs that are submitted to run on Cedar/Cypress. Cedar has 8 CPUs and Cypress has 128. To LSF each CPU is roughly equivalent to what LSF calls a job slot. A job slot is the basic unit of processor allocation in LSF. A serial job uses one job slot; a parallel job requesting N processors should use N job slots. Each user may have up to 36 job slots in use at any one time. If you are already using 36 job slots and you submit another job to run, that job will be pending until job slots are freed as your running jobs finish. Similarly, if all 128 job slots on Cypress are in use when you submit a job, even if you are not using any job slots yourself, your job will be pending.

The Cedar/Cypress cluster has a different queuing system from other clusters. Type:

bqueues

to show all available queues in this cluster. Three queues, now, week, month, are most commonly used. For more information about a particular queue and the job slot limit per user type:

bqueues -l queue_name

where “queue_name” is the name of the queue.

To submit a serial job, type:

bsub -q week -m cypress ./source.x

To submit an MPI job with 4 CPUs on the command line type:

bsub -q week -n 4 -m cypress mpirun -np 4 ./source_mpi.x
  • For csh/tcsh users: 
setenv OMP_NUM_THREADS 4
bsub -q week -n 4 -m cypress ./source_openmp.x
  • For bash/ksh users: 
export OMP_NUM_THREADS=4
bsub -q week -n 4 -m cypress ./source_openmp.x

Transferring files

It is likely that you will need to transfer files between your campus computer systems and Cedar/Cypress. You will need to use the “sftp” or “scp” command to move your files. The command “sftp” works similarly to the popular “ftp” command. From your host Unix computer, type: 

sftp yourOnyen@cedar.isis.unc.edu

Enter your Onyen password and you will be presented with the “sftp” prompt. Use the “put” and “get” commands to transfer files, as you would do with standard “ftp”.

To use the “scp” command, follow this example to get a file, named “temp.txt”, from Cedar/Cypress and store it in your local “/tmp” directory with the same file name. You will be prompted for your password. Type all (including the ending period) on one command line even if it wraps: 

scp yourOnyen@cedar.isis.unc.edu:/netscr/yourOnyen/temp.txt /tmp/.

You can also copy a whole directory at once. The following command will copy the whole directory, “temp.dir” from your local computer in “/tmp” to Cedar/Cypress, and place it in the “/netscr/yourOnyen” directory with the name “temp1.txt”. Type all on one line: 

scp -r /tmp/temp.dir cedar.isis.unc.edu:/netscr/yourOnyen/temp1.txt

Research Computing home page


Top
University of North Carolina - Chapel Hill