Getting Started on Emerald - Short Course
Table of Contents
Research Computing Server Emerald
Logging Into Emerald and Using Scratch Space
Setting Up and Submitting Jobs on Emerald
Introduction
- Purpose of Research Computing Group at UNC
UNC Research Computing aims to provide a world-class computing infrastructure as well as other technology tools and capabilities to support the research needs of UNC researchers. Our goal is to provide a state-of-the-art environment that will support the highest level of multidisciplinary research and help UNC Chapel Hill become the premier research university in the U.S. The Research Computing group currently offers a number of computing services for researchers at UNC-Chapel Hill: 128-processor SGI Altix ( Cedar/Cypress), a 4160-processor Dell Linux Cluster ( Topsail) and a 352-processor Beowulf Linux cluster ( Emerald) with access to 4 16-processor IBM P575 machines and a 32-processor IBM P690 machine.
Research Computing Server Emerald
- What is Emerald?
Emerald is a 352-processor Beowulf Linux Cluster that is intended for use by a wide variety of researchers at UNC-Chapel Hill. Emerald has access to a variety of statistical software packages (such as SAS and Stata), mathematical software packages (such as Matlab and Mathematica) and scientific software packages (such as Gaussian, Amber and GROMACS). Emerald has been designed to be appropriate for a first-time user of Research Computing servers as well as have the necessary computing power to be attractive for larger computational jobs of more experienced users. - What Happened to Happy/Yatta and Statapps?
In previous years, Research Computing at UNC-Chapel Hill maintained the computing servers Statapps and Happy/Yatta. The Statapps server was developed specifically to run statistical software packages such as SAS or Stata. However, as Emerald has been adapted as the appropriate server for a variety of these applications, the Statapps server was no longer necessary with SAS, Stata and other similar software being offered on Emerald. The Yatta server, a 32-processor IBM P690 machine, was developed for running larger computations requiring significant memory resources. It was decided that the best use of Yatta was to no longer allow direct login to the Happy login server and to wrap the Yatta machine into the Emerald cluster. Thus, it is possible to submit jobs to Yatta after logging into one of the Emerald login nodes. More details about Yatta and other higher memory nodes available on Emerald can be found in the Higher Memory/AIX Resources section of the Getting Started on Emerald help documentation. - Signing Up for an Account on Emerald
All researchers affiliated with UNC who have a valid Onyen and password can have access to the Emerald cluster. Most individuals can sign up for an account on Emerald using the Subscribe to Services button found on the UNC Onyen webpage. After clicking on the Subscribe to Services button and entering an Onyen and password, Emerald should be one of the options that can be subscribed to. After confirming the choice of Emerald, you will be emailed a confirmation within 24 hours that you have been granted access to the Emerald cluster and can login and begin using the services.
If you do not see the Emerald option on the Subscribe to Services page (this does occur for postdoctoral students and undergraduate students sometimes), you will need to use a different method for gaining access to Emerald. You will need to fill out and have a departmental sponsor sign the Affiliate Request form and fax this form to the PID office at (919) 962-8217. Once the PID office has the form, the PID office will appropriately enter you into the system so that you will have the option to subscribe to Emerald using the method described above at the Subscribe to Services webpage. - Signing up for Mass Storage Usage
The procedure for signing up for a Mass Storage account is exactly the same as for signing up for an account on Emerald. When you reach the Subscribe to Services webpage, Mass Storage will be one of the options to subscribe to. Also, for those who do not have the options for Emerald or Mass Storage on the Subscribe to Services webpage, once the Affiliate Request form is filled out and sent to the PID office, the Mass Storage option will also be enabled on the Subscribe to Services webpage. Similar to the subscribing to Emerald procedure, a confirming email will be sent within 24 hours of subscribing detailing that you have been granted access to Mass Storage space. More details about Mass Storage space can be found on the Mass Storage help page.
Logging Into Emerald and Using Scratch Space
- Logging into Emerald
Emerald can be accessed both on UNIX/Linux-based machines and on Windows machines using the SSH protocol. However, the methods for accessing Emerald on both sessions is different.
- Linux, OSX, UNIX terminals
Open a terminal window and, if your username on your local computer is the same as your Onyen, enter the following command to login:
ssh emerald.unc.edu
You will be prompted for your Onyen password upon signing in. This procedure will open a command prompt in your AFS home directory. However, if your local computer username is different from your Onyen, use the following command to login to Emerald:
ssh my_onyen@emerald.unc.edu
where my_onyen is your personal Onyen. This will again bring you to your personal AFS home directory.
- Windows - SSH Secure Shell
From a Windows computer, the easiest method to connect to Emerald is through use of the SSH Secure Shell software. This software is offered as a free download from UNC and can be found at the UNC Software Acquisition Page. Once this program is installed on a personal computer, it can be used to access Emerald through the steps as detailed on the Research Computing SSH Secure Shell documentation under the "Invoking the Software" heading. To connect specifically to Emerald using the steps outline in the documentation, you will want to enter “emerald.unc.edu” as the "Host Name". Further, make sure to follow the instructions for how to enable Port Tunneling on this same page if you plan on using any of the software on Emerald that uses a GUI interface, such as interactive SAS, Stata or Matlab. - File storage space and AFS home directory
AFS home directories have a very limited storage space. An AFS home directory space is limited to 100MB for undergraduates and 250MB for faculty, staff and graduate students. Therefore, the AFS home directory is not an ideal space for backing up large amounts of data or results from calculations performed on Emerald. We recommend that all important data be backed up either to a personal computer or to the Mass Storage space, which has a 100GB storage limit. The amount of free space remaining in an AFS home directory can be listed by using the following command when in your personal AFS home directory:
fs listquota
- Working directory - /netscr space
On Emerald, it is not possible to run programs or calculations on the compute nodes from the AFS home directory space. The compute nodes on Emerald do not have access to the AFS home directory space and will not be able to access files stored there. However, Emerald has a work space, known as Netscratch space, where all calculations and job submissions can be performed. This directory is named “/netscr” and all users on Emerald have a directory created named “/netscr/my_onyen” where all computational work can be submitted and performed on the compute nodes. The “/netscr” space does not have a storage limit and can therefore handle a large number of output files and data.
- Working directory - GPFS (General Parallel File System) scratch space
On Emerald, a new scratch space is currently being implemented and users are recommended to try this space for future work on Emerald. Similar to “/netscr” space, the GPFS scratch space is a shared temporary work directory for current job processes. The GPFS scratch space is broken into two sets: “/smallfs” and “/largefs”. The “/smallfs” file space has 15TB of scratch space and is intended for research data files smaller than 1MB. The “/largefs” file space has 18TB of scratch space and is intended for research data files larger then 1MB. All users on Emerald have directories on GPFS space at:
/smallfs/my_onyen
and:
/largefs/my_onyen
More information regarding usage of the GPFS scratch space and any updates regarding usage of this space can be found on the Getting Started on Emerald help document.
- File transfer from home computer to Emerald
- Linux, OSX, UNIX terminals
In order to transfer files from a local computer to the Emerald server, the “scp” (secure copy) command can be used. For this command, open a terminal window and change directory to where the file you want to transfer is located. Then, use the following command to copy the file (which will be called “testfile.txt” for our purposes):
scp testfile.txt my_onyen@emerald.unc.edu:/largefs/my_onyen/
This will copy the file into your personal “/largefs” directory. If you want to copy the file into your AFS home directory, replace the “/largefs/my_onyen” with “/afs/isis/home/m/y/my_onyen” where “m” and “y” are the first two letters of your Onyen (i.e. for the onyen davis, the path would be “/afs/isis/home/d/a/davis”). In order to copy a directory to your “/largefs” space, the following command can be used:
scp -r TestDirectory/ my_onyen@emerald.unc.edu:/largefs/my_onyen/
- Windows - SSH Secure File Transfer
Similar to connecting to Emerald using SSH Secure Shell on a Windows computer, SSH Secure File Transfer is an easy method for transferring files to Emerald without having to use commands on a UNIX Terminal. To use SSH Secure File Transfer, start up the SSH Secure File Transfer program that is also installed when SSH Secure Shell is installed on the computer. Once the program is started, click on the Quick Connect button and enter “emerald.unc.edu” as the Host Name and your Onyen as the User Name and click Connect. Once you are connected, the screen on the left will list the Desktop of the local computer and the screen on the right will list the content of your AFS home directory. To transfer files or directories, right-click on the file or directory and click on either Upload or Download to move the file. To change directories to the “/largefs” directory, click in the drop-down menu that lists the AFS home directory path and type in “/largefs/my_onyen” and transfer files to and from your “/largefs” directory.
- Transferring files between your AFS home directory and your “/largefs” space
To transfer files between directories on Emerald or between your AFS home directory and your scratch space, the “cp” (copy) command can be used at a terminal prompt. Once you are logged into Emerald, a file can be transferred from your AFS home directory to your scratch space using the following command:
cp testfile.txt /largefs/my_onyen/
This example assumes you are in your AFS home directory where the file “testfile.txt” resides. The “cp” (copy) command can be used to copy files between any directories on Emerald as long as the full path is listed. To determine the full path of a given directory, the “pwd” command can be used. Also, if a full directory needs to be copied between the AFS home directory and the scratch space, the following command can be used:
cp -r TestDirectory/ /largefs/my_onyen/
- Transferring files to Mass Storage
- Preparing files for transfer to Mass Storage
As mentioned previously, it is important to backup all important data from your scratch space. Since your space on the AFS home directory is very limited, Mass Storage is an ideal space to backup data. However, it is also important to make sure all data stored on Mass Storage is efficiently packaged. Therefore, we request that all users archive their data before moving it to Mass Storage. In order to package the data, use the “tar” command. The “tar” command will take all files from a given directory and package them into a single “*.tar” file. This command can be used as follows:
tar -cvf TestDirectory.tar TestDirectory/
Once the tar file is created, all contents of the tar file can be listed with the “t” option as follows:
tar -tvf TestDirectory.tar
If you need to untar the directory in order to change a file within the tar file, you will need to untar the directory using the following command:
tar -xvf TestDirectory.tar
- Transferring files to Mass Storage
Once the directory containing all data has been archived, the “tar” file can be transferred to Mass Storage space using the “mv” (move) command:
mv TestDirectory.tar ~/ms/
where "~/" is UNIX shorthand for "home directory" and "ms/" is a subdirectory off your AFS home directory that links to your Mass Storage directory (“/ms/home/m/y/my_onyen/”).
Setting Up and Submitting Jobs on Emerald
All calculations and programs that are to be run on Emerald are handled using the Load Sharing Facility (LSF) for job management. This software helps to ensure that compute nodes are efficiently distributed to all users. Further, Emerald uses the Isis package manager (ipm) to distribute popular software on Emerald without requiring the user to compile code on their own.
- Packages and “ipm” commands
Popular software, such as SAS, Stata, Matlab and Gaussian are offered to the Research Computing community through the above mentioned Isis package manager (ipm). This manager uses AFS as a means of sharing an executable amongst all users with access to Emerald. This provides all users the ability to access popular software without the hassle of installing or compiling the source code on their own. Also, this allows novice users of Emerald and UNIX to use software without the technical expertise necessary to compile software on their own.
Many software packages are available through the Isis package manager. A list of popular software packages which are offered through the package manager can be found at the Research Computing: Applications page. In order to add a package, the “ipm” command is used as follows:
ipm add package_name
For example, if you want to use the default version of Matlab, you would type in:
ipm add matlab
Listings of common ipm commands for popular Science and Visualization applications can be found on each of the respective application's help documentation while ipm commands for popular Statistical and Mathematical applications can be found on the Mathematical and Statistical Software Application Notes website. A listing of all applications and versions available through the package manager, listed by their package name, can be found on the Inventory of AFS Packages webpage.
- LSF
As mentioned before, LSF handles all job submissions on Emerald. The purpose of LSF is to prevent any one user from taking up all compute nodes at a given time by distributing these nodes to all users based on their past history of submission. Thus, a user who infrequently submits jobs to Emerald will be given a higher priority to these resources than frequent users. Further, LSF provides options for the user to follow the progress of a job once it has been submitted using a series of commands. Finally, LSF provides a cap to the total number of processors a given user can utilize at a given time. For Emerald, only 60 job slots (processors) can be used by one user at a given time. If jobs are submitted after the 60 processor limit is reached, the jobs will be placed in PENDING status and will be run once other jobs are completed. This system prevents any one user from overwhelming the system and dominating the compute nodes on Emerald. - LSF Queue System
On top of efficiently distributing jobs amongst users and setting an upper limit on the number of processors available to users, LSF also limits how long a given job can run. This is done using the queue system. When you submit your job on Emerald, a queue is either designated in the submission command or is chosen by default. Some queues have a time limit that the job is allowed to run for. More information about queues in Emerald and the limits for each queue can be found on the Getting Started on Emerald page under the "Submitting Jobs" section. Finally, there is a queue on Emerald, called the “idle” queue, which has no limit for the run time or the number of processors that can be used (except the 60 processor/user limit still exists). However, the “idle” queue can be preempted by the “patrons” queue. The “patrons” queue is composed of groups that have paid for the right to unlimited usage of a subsection of compute nodes on Emerald and thus have preference over other users for jobs on these nodes. Thus, if a job is running in the “idle” queue and a “patrons” queue job requests nodes that are currently being used by that job in the “idle” queue, the job in the “idle” queue will be paused / suspended until the “patrons” queue job completes. Thus, use of the “idle” queue is unrestricted but runs the risk of suspension by jobs run by the patrons groups. - LSF Commands - Job Submission
The following commands are integral to using LSF for job submission on Emerald. More detailed descriptions of LSF commands used in job submission and tracking can be found on the Getting Started on Emerald page and also the LSF - Submitting Jobs help document. - “bsub”
The command used to submit jobs to Emerald through LSF is the “bsub” command. The “bsub” command is usually entered as follows for a single processor job:
bsub -q queue_name my_executable executable_options
for example, if you wanted to submit a Gaussian calculation to Emerald using the input file “test.com” on the “idle” queue, the “bsub” command would be:
bsub -q idle g03 test.com
If no queue is given, the job will be submitted to the “week” queue by default. Many of the submission commands for Statistical and Mathematical programs differ from this basic format and example submissions can be found on the Mathematical and Statistical Software Application Notes help document. Also, example “bsub” commands for much of the software offered through the Isis package manager are given in the application's help documentation found on the Research Computing: Applications help page. Further, if your submission will require multiple processors, the “bsub” command will need the “-n” option:
bsub -q idle -n 4 my_executable executable_options
if 4 processors are requested. Also, the “bsub” command can be used to specify a resource (“-R”) on Emerald. There are various compute nodes available on Emerald, such as 2.4GHz Xeon (xeon24), 2.8GHz Xeon (xeon28) and 3.2GHz Xeon (xeon32) processors. All of these processors are IBM Blade (blade) processors. Thus, to specify one of these resources, you would use the command:
bsub -q idle -R xeon32 my_executable executable_options
if you wanted to specify these blade processors in general, you would use the command:
bsub -q idle -R blade my_executable executable_options
If a resource you specify is not available or is currently full, the job will be placed in PENDING until a processor on the resource is open. Therefore, it is usually better to specify "blade" as a resource instead of one of the specific resources as there are many more available nodes until the "blade" designation then under a specific processor resources.
As mentioned previously, all necessary files must be in a scratch directory in order for the compute node to be able to access them. Further, the “bsub” command should be submitted in the same directory as the executable or the full path of the executable should be given in the bsub command.
- bqueues
In the “bsub” command, the queue is specified using the "-q" option as shown above. In order to see a real time list of all queues on Emerald and the current status of those queues, the following command can be used:
bqueues
This command will give how many job slots are currently in use on each plus the slot limits for the queue and per user. This information can be used to give insight into which queue is appropriate for job submission.
- Specialty Submission Scripts
Certain common programs, such as SAS and Matlab, have scripts which include the “bsub” commands and options that are needed. A listing of some of these scripts can be found at the Mathematical and Statistical Software Application Notes help document - Example Submission - SAS
For SAS, the normal “bsub” command might be:
bsub -q queue -R [resource] sas program.sas
where "resource" represents the resource that is chosen for the job, such as "xeon32" or "blade". The script way to submit a SAS job is given by:
bsas program.sas
In the above example by running the "bsas" script with a program file, a job will be submited on the default resource "blade" and on the default queue "week".
- Example Submission - Matlab
Matlab has an extensive “bsub” command necessary to prevent the software from launching in a graphical-display mode so that a matlab “*.m” file can be submitted in batch mode. Usually, a Matlab job submission run from the command line (known as running in "batch mode") would require something like the following bsub command:
bsub -q queue -R [resource] matlab -nodisplay -nojvm -nosplash \
program.m -logfile program.log
However, this command can be run using the following command:
bmatlab program.m
Using this command, “program.m” will be submitted to Matlab in the “week” queue using the default resource "blade". These specialty submission scripts greatly simplify use of the bsub command for many Mathematical and Statistical software packages available for use on Emerald.
- Compiling code on Emerald
There are four families of compilers available on Emerald. These include the GNU set of compilers and those from third party vendors such as Absoft, Intel and the Portland Group. Performance benchmarks by the Research Computing group at UNC suggest that the Intel and Portland group compilers generally tend to produce faster code. As described above, each compiler group must be added using the “ipm” command. - The Portland Group compiler can be added using the following command:
ipm add pgi
- A Fortran program can be compiled with the Portland Group compiler using the following command:
pgf77 program.f -o program
- The Intel compiler(s) can be added using the following command:
ipm add intel_fortan
ipm add intel_CC
- A Fortran program can be compiled with the Intel compiler using the following command:
ifort program.f -o program
- The Absoft compiler(s) can be added using the following command:
ipm add profotran
- A Fortran program can be compiled with the Intel compiler using the following command:
f77 program.f -o program
- After the program has been compiled, it can be run on an Emerald compute node using the “bsub” command:
bsub -q [queue] -R [resource] program
where "queue" should be replaced by the appropriate queue depending on the runtime of the program and "resource" is one of the computing resources on Emerald, such as "blade"
- More details about different compilers on Emerald and compiling parallel programs on Emerald can be found on the Getting Started on Emerald help document.
- LSF Commands - Following Jobs After Submission
Once a program or job has been submitted to an Emerald compute node using the “bsub” command, there are various LSF commands that can be used to follow the process of the jobs as they run on a compute node. - bjobs
This command displays the current status of one or more jobs. If used without any options, it displays all of your pending, running or suspended jobs.
emerald$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
765339 nazar RUN week bc01-n14 p575-n00 *itled2.do Mar 27 16:42
765338 nazar RUN week bc01-n14 p575-n01 *itled1.do Mar 27 16:42
765340 nazar PEND week bc01-n14 *wage01.do Mar 27 16:42
If you want to check the status of a specific job, use “bjobs JobID” :
emerald$ bjobs 125532
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
125532 jobuser DONE week bc01-n13 bc03-n01 jobname Mar 28 13:37
Using “bjobs” with “-l” option displays detailed information for a particular job in a multiline format. For instance, this option displays project name, job command, current working directory on the submission host, pending and suspending reasons, job status, resource usage, resource usage limits information, runtime resource usage information on the execution hosts.
- bhist
This command displays historical information about jobs. For instance, it shows the amount of time a job was pending, running and suspended. Using it with the “-l” option provides historical information about a particular job in the long format.
emerald$ bhist -l 765189
Job <765189>, User <nazarov>, Project <noproj>, Command <factor>
Thu Mar 27 15:22:45: Submitted from host <bc01-n14>, to Queue <week>
, CWD </netscr/nazarov/MPI>, Requested Resources <blade>;
Thu Mar 27 15:22:48: Dispatched to <bc01-n08>;
Thu Mar 27 15:22:48: Starting (Pid 3431);
Thu Mar 27 15:22:48: Running with execution home </afs/isis.unc.edu
/home/n/a/nazarov>, Execution CWD </netscr/nazarov/MPI>,
Execution Pid <3431>;
Thu Mar 27 15:22:48: Done successfully. The CPU time used
is 0.1 seconds;
Thu Mar 27 15:22:48: Post job process failed;
Summary of time in seconds spent in various states by
Thu Mar 27 15:22:48
PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL
3 0 0 0 0 0 3
- bkill
The “bkill” command sends signals to kill, suspend, or resume unfinished jobs. The “bkill” command is useful if the job appears to be stuck or if there is a problem with the code or input that is discovered after the job is submitted to LSF. The “bkill” command is submitted in the form of “bkill JobID”:
emerald$ bkill 765333
Job <765333> is being terminated
- bswitch
The “bswitch” command is used to move a submitted job into another queue. For example, if a job was submitted to the “week” queue but it appears that the job will require longer to finish, the job can be moved into the “month” or “idle” queues using the command:
bswitch month JOBID
or
bswitch idle JOBID
- bpeek
The “bpeek” command is used to follow the output of a job as it is running. The “bpeek” command will display the contents of the “STDOUT” file that is created while the job is running. This command is a helpful way of tracking the job as it is running and determining if there are any errors occurring. The format for using the “bpeek” command is:
bpeek JOBID
- bfree
The “bfree” command provides a clear description of how many job slots are currently open and available to a user at a given time. To show this output in table format, use the command as follows:
emerald% bfree -table -q all
This is a table of the number of job slots that are available for
the combinations of resources and queues you specified
that you are able to use.
'X' denotes a combination of resource and queue that is not possible.
Row and column totals are not relevant since one resource can be available
for many queues and hosts in a resource can also be included in more than
one resource.
Resources int now week month idle
--------- --- --- ---- ----- ----
select[model==any] 2 2 24 4 60
As shown, the output of “bfree” provides the number of job slots available for the default resource for each queue available to the user. If the user is a member of a patrons group, the job slots limit within the patrons queue will also be given. To specify a given resource, bfree can be used as follows:
emerald% bfree -table -q all -R blade
This is a table of the number of job slots that are available for
the combinations of resources and queues you specified
that you are able to use.
'X' denotes a combination of resource and queue that is not possible.
Row and column totals are not relevant since one resource can be available
for many queues and hosts in a resource can also be included in more than
one resource.
Resources int now week month idle
--------- --- --- ---- ----- ----
blade 2 2 24 4 60
Finally, to see how many job slots are available to all users as well as how many job slots are available to a given user, the “-nolimits” tag can be used:
emerald% bfree -table -q all -nolimits
This is a table of the number of job slots that are available for
the combinations of resources and queues you specified
that you are able to use.
The first row per resource is the total number of job slots available.
'X' denotes a combination of resource and queue that is not possible.
Row and column totals are not relevant since one resource can be available
for many queues and hosts in a resource can also be included in more than
one resource.
Resources int now week month idle
--------- --- --- ---- ----- ----
select[model==any] 115 113 114 114 154
select[model==any] 2 2 24 4 60
--------- --- --- ---- ----- ----
The first listing in the above output gives the available job slots to all users while the second listing gives the available job slots to a given user. The lower number of available job slots for the individual user is due to the limit placed on each queue. The 60 processor limit in the idle queue represents the 60 processor limit for each user for all total jobs on Emerald. Finally, to see more options available with the “bfree” command, help documentation can be found using the command:
bfree -h
- Output After Job is Submitted to LSF
Once the job is finished, an email is sent to the user's default email address that details the results of the calculations. This email contains the results from the “stdout” file and also includes any LSF error messages that may occur while the job is running. However, if it is preferred that the output is saved to a file instead of sent by email, the “–o” option can be used:
bsub -q queue_name -o output_file my_executable executable_options
Further, the following “bsub” command can also be used to specifically label the output file with the JobID:
bsub -q queue_name -o out.%J my_executable executable_options
This will create an output file, named “out.JobID” ("%J" is translated into the JobID by LSF), that contains all the relevant results and details of the calculation. Further, a standard error output file can be created using the “-e” option:
bsub -q queue_name -o output_file -e error_file my_executable executable_options
bsub -q queue_name -o out.%J -e err.%J my_executable executable_options
The second bsub command will create an error file, named “err.JobID”, that will contain all the necessary details of any errors that occur during the calculations. Both the output file and the error file will be created in the directory where the “bsub” command was submitted.
Interactive Jobs on Emerald
In order to run interactive jobs that have a graphical user interface (GUI) on the Research Computing servers, an X-Windows server should be running on the local computer. X-Windows, also known as X11, is a system implementing the X display protocol which provides an application window on the local computer. The main purpose of using the X11 system is to allow a program to display a graphical user interface while running from a remote UNIX/Linux server. This interface allows a user to directly interact with a program through the use of a mouse or keyboard and also by drawing and moving windows on the screen. Due to the fact that an X11 system is sometimes not an integral part of an operating system, a user must sometimes install an X11 application on his or her computer.
- X-Windows for Computers running Linux or OSX (Mac)
X-Windows clients for Linux or OSX are commonly included with the installation media for the operating system. For example, recent releases of the OSX operating system on Mac include the X11 application. This application is not installed by default when OSX is installed or upgraded, but can be installed from the installation media. The X11 application in OSX is similar to a terminal but allows for display of a graphical interface when necessary. X11 applications for Linux desktop computers are likely already installed.
In order to allow X-Windows applications to function correctly when connecting to Emerald, the “ssh” login command needs to be modified. If interactive jobs will be used, the following command should be used to login:
ssh -X my_onyen@emerald.unc.edu
If an error is displayed when trying to login with the “-X” option or, when trying to run an interactive job, an error message is displayed that is similar to "X connection to bc01-n14.isis.unc.edu:11.0 broken", then try to login using:
ssh -Y my_onyen@emerald.unc.edu
The “-X” and “-Y” options in the ssh command will tunnel your X11 connection through your ssh connection. This will set your DISPLAY variable on Emerald to be the same as the DISPLAY variable on your local computer. Therefore, X11 will be able to display X-Windows applications when requested.
- X-Windows for Computers running Windows
There are a variety of X11 applications that exist for Windows computers. However, for security reasons, UNC is only supporting the use of StarNet's X-Win32 application for use on UNC computers. X-Win32 can be downloaded for free from the ITS Software Acquisition homepage and extensive instructions for setting up X-Win32 for use with Emerald can be found in the Research Computing X-Win32 help document. Once invoked, X-Win32 will provide a terminal window with a connection to Emerald that can display X-Windows applications. X-Win32 can also be used in conjunction with SSH Secure Shell. In such a situation, X-Win32 runs in the background while all login details are handled on SSH Secure Shell. Once logged into Emerald, if an application is started that requires X-Windows, the application interface will appear in an X-Win32 window. Make sure that Port Tunneling is allowed in SSH Secure Shell, as described in the SSH Secure Shell help document, if SSH Secure Shell will be used with X-Windows applications. - Example Submission for Interactive Jobs
Submitting jobs to the interactive queue requires one additional parameter in the bsub command, “-Ip”. Further, all interactive jobs should be submitted to the “int” queue. For instance, the job submission command to invoke Stata interactively is:
bsub -q int -Ip -R blade xstata
Many applications can be invoked also interactively by using shorter, specialty submission scripts, such as:
xstata
xsas
xmatlab
Other mathematical and statistical applications, which can be invoked interctively on Emerald, can be found in Mathematical and Statistical Software Application Notes. Further, many of the scientific applications listed on the Research Computing Visualization Software Applications help document require similar “bsub” commands to the example listed above.
Summary
The Research Computing server Emerald is a powerful resource for performing a variety of scientific, statistical and mathematical tasks. While the extensive directions given in this and other documentation may seem cumbersome, the use of the Emerald server and the nuances of the LSF system will become much easier to understand with practice. Further, both Research Computing and ITS have many help documents covering a wide variety of issues, such as Learning UNIX, Application Specific Notes and detailed LSF documentation, that can be accessed through the ITS Help & Support Site. More information about the Emerald server can also be found on the Research Computing Getting Started on Emerald help document. Finally, any specific questions about using Emerald can be send to the Research Computing help desk through email to research@unc.edu or by submitting an Online Web Ticket.
More Information
If you have any questions, please feel free to call 962-HELP or submit an Online Web Ticket.
Document Overview
Tags: research computing , scientific computing , statistical computingSubjects: Advanced Computing
Audience: Everyone
This work is licensed under a Creative Commons Attribution 3.0 United States License.
