Table of Contents
System Information
-
Login node : topsail.unc.edu 8 CPUs @ 2.3 GHz Intel EM64T with 2x4M L2 cache (Model E5345/Clovertown), 12 GB memory
-
Compute nodes : 4,160 CPUs @ 2.3 GHz Intel EM64T with 2x4M L2 cache (Model E5345/Clovertown), 12 GB memory
-
Shared Disk : (/ifs1) 39 TB [ http://www.ibrix.com/ ] IBRIX Parallel File System
-
Interconnect: Infiniband 4x SDR
-
Resource management is handled by [ http://help.unc.edu/?id= ] [ http://www.platform.com/services/support/docs/index_62.html ] LSF v.6.2, through which all computational jobs are submitted for processing
Getting an Account and Allocation
Eighty percent of the cycles on this cluster will be allocated using an
[ http://www.renci.org/unc/computing/resources/allocations.php ]
allocation process requiring submission of a proposal detailing research to be done, computational requirements, number of CPU hours needed, and including demonstration of the scalability of code to a large number of processors. At most times, up to 20% of the system's cycles may be available for other large parallel work on a first-come, first-served basis. You may request an account on Topsail by sending an email to
<research@unc.edu>. Please include your
[ http://onyen.unc.edu ]
Onyen, full name, campus address, campus phone number, department, faculty sponsor's name if you are not a faculty member, and a description of the work you plan to do. You must also provide your email.unc.edu or first_last@unc.edu address; that is the address that will be used for communication with you. Topsail accounts are not available to collaborators from other institutions; you must be a UNC-Chapel Hill faculty member, staff member or a graduate student with a faculty sponsor in order to be considered for an account. Note that commercial applications software is not installed on the cluster.
More about [ http://www.renci.org/unc/computing/resources/allocations.php ] Topsail Allocations.
Logging In
-
Use ssh to connect to topsail.unc.edu and login with your ONYEN.
-
On the first login, ssh-keygen will run. Accept the defaults. If this environment is not established correctly, jobs will fail with [ http://help.unc.edu/?id=6220 ] "permission denied" messages.
-
Your home directory will be /ifs1/home/ONYEN. Any user home directory in excess of 4GB will be excluded from backups. Home is not intended to hold more than basic code and environment files. All data and output files should be directed to /ifs1/scr. Create a directory there for your work files. Note that /ifs1/scr will never be backed up and files not associated with a currently running process will be deleted after 14 days. Users have a hard quota of 500GB for data stored in /ifs1/scr.
-
Interactive use on the login node must be restricted to compiling and debugging. Other processes running on the login node will be terminated.
Software Development Tools
-
Intel Compiler Suite v.9.1 for Fortran77, Fortran90, C and C++, and v.9.0 Math Kernel Library
Compiling
The development suite is toggled using the module command. To see current environment:
module list
module show hpc/mvapich-intel
To change to gcc:
module unload hpc/mvapich-intel
module load hpc/mvapich-gcc
In either environment the available mpi compiler commands are:
mpicc
mpif77
mpif90
mpiCC
mpicxx
Note
mpiCC and mpicxx rely on mpi2c++ which is currently only available under gcc.
Submitting Parallel Jobs
bsub -n "< number CPUs >" -o out.%J -e err.%J -a mvapich mpirun ./mycode
or
#### run_hpl
#BSUB -n "< number CPUs >"
#BSUB -e err.%J
#BSUB -o out.%J
#BSUB -a mvapich
mpirun ./mycode
########
bsub < run_hpl
For more basic LSF commands refer to the brief Fundamental LSF Commands.
Queue Structure
-
A description of the queues available to users in the Topsail cluster can be found below. The list is in order of priority. You can also use the bqueues command to list the properties of a specific queue. For example, you could type "bqueues -l debug" to find out more about the debug queue. Additional queues may be added as need dictates.
-
All queues share a common fairshare allocation policy that governs what job will be dispatched next based on the recent runtime history of each user or group with jobs queued to run.
-
Time limits are measured by wallclock time.
-
All queues have exclusive enabled. A user can indicate with the bsub option -x that their job should be sent to hosts that have no other jobs running, and that no further job can be dispatched to those hosts until the job completes.
Table 1. Public Access Queues
Queue Name Min/Max CPU Limit per Job Time Limit Comments debug 1 to 4 2 hours 128cpu 32 to 128 2 days Default queue 32cpu 4 to 32 2 days
Table 2. Public Access Queues
Queue Name Min/Max CPU Limit per Job Time Limit Comments 512cpu 256 to 512 2 days Job must be shown to scale reserve_rack Research project specific chunk Research project specific staff Research project specific
For further information on job management, please see [ http://help.unc.edu/?id=4484 ] LSF (Load Sharing Facility)
Additional Help
Transferring Files
It is likely that you will need to transfer files between your computer and [ http://help.unc.edu/?id=4196 ] Research Computing systems like Topsail. You will need to use the sftp "secure file transfer protocol" or scp "secure copy" commands to transfer your files. The command sftp works similarly to the unsecure ftp command. From your UNIX/Linux computer, type:
sftp yourOnyen@topsail.unc.edu
Enter your
[ http://help.unc.edu/?id=1959 ]
Onyen password and you will be presented with the
sftp
prompt. Use the
put
and
get
sftp
commands to transfer files. To use the
scp
command, follow this example to get a file named
temp.txt from your scratch directory on Topsail and store it in your local
/tmp directory with the same file name. You will be prompted for your password. Type all of the following at a prompt even if it wraps:
scp yourOnyen@topsail.unc.edu:/ifs1/scr/yourOnyen/temp.txt /tmp/
You can also copy a whole directory at once. The following command will copy the whole directory
temp.dir from your local computer in
/tmp to Topsail, and place it in the
/scr/yourOnyen directory with the name
temp1.txt. Type all on one line:
scp -r /tmp/temp.dir topsail.unc.edu:/scr/yourOnyen/temp1.txt
Note
A Windows PC with the command line utilities from SecureCRT installed has the command vcp available which works in the same way as scp . Use Windows style paths for anything that refers to files or directories on your local machine when using vcp .
Mass Storage
The Mass storage system (also known as SAM-FS or
/ms) is intended for archiving files and storing very large files that cannot be recreated through programming steps. It is not intended to be used as a backup location for disk drives, operating systems, software or scratch files that can be recreated by re-running jobs. In addition, files that are changed often or directories with many files in them will cause performance problems and consume too many storage resources. We monitor the use of mass storage and will inform you if you are using it inappropriately or if you need to purchase tapes to accommodate the volume of data that you need to store.
For further information on using Mass Storage, please see [ http://help.unc.edu/?id=6291 ] Mass Storage
Scratch/Work Space
Approximately 39 terabytes of scratch space are available for jobs running on Topsail. Mounted as
/ifs1/scratch, this space is not backed up. It is therefore NOT intended to be permanent file storage. Files older than 14 days that are not associated with currently running jobs will be deleted.
Known Issues
Resolved
Known Limitations
-
Profiling-enabled libraries are not yet available.
[ http://its.unc.edu/research-computing.html ] Research Computing home page
Copyright 2002-2007 The University of North Carolina at Chapel Hill.
This
work is licensed under a
Creative Commons Attribution 3.0 United States License.
