Jump to: Overview

LSF - Submitting Jobs

Introduction

This document covers how to submit jobs to Research Computing servers using the LSF command “bsub”.

Note: Jobs running outside the LSF queues will be killed. The logon privileges of users who repeatedly run jobs outside of the LSF queues will be suspended.

You can read a more detailed explanation of the “bsub” command by reading the man page when logged onto one of our servers:

man  bsub

Submitting Jobs

You use the “bsub” command to submit jobs via LSF.

Here is a very simple “bsub” command submission of the “uname” command which prints the system information for the host it is run on:

bsub uname -a

  Job  <5871>  is  submitted  to  the  default  queue  <week>.

Note: “bsub” options come after the word "bsub" and options to the command being submitted by “bsub” come after the command. In this example “-a” is an option of the “uname” command.

Each LSF job runs in a queue. If you do not give LSF a queue name, your job will go to the default queue which is the “week” queue, which is intended for batch jobs that will not run longer than a week. If you want your job to run in a different queue, use the “-q” option. For example:

bsub -q month

After you submit your job, LSF will send it to a server for execution. This is called dispatching the job. If you don't specify a server, LSF will choose one for you. LSF will choose the host with the lightest load given the resource(s) you have requested.

If you want your job to run on a specific resource, such as the IBM BladeCenter nodes on Emerald, use the “-R” option:

bsub  -R  blade  ...

After your job has finished, LSF will send you an e-mail containing the standard output (stdout) and standard error (stderr) from your job, as well as some basic information about the execution of the job. If your program produces additional output files, they are separate and are not included in this e-mail. LSF will use the address Onyen@email.unc.edu to send you e-mail. If you do not read e-mail at that address, you can set up a forwarding address at the Onyen services page. You can also specify a different e-mail address on the “bsub” command using the “-u” option described below.

Jobs submitted to queues such as “week” or “month” run in the background, but sometimes you need to run a job interactively (in the foreground). To do this you need to request the “int” queue explicitly and also use the “-Ip” (interactive pseudo terminal) option. Here is a simple example:

bsub -R blade -q int -Ip uname -a

      Job  <855678>  is  submitted  to  queue  <int>.

      <<Waiting  for  dispatch  ...>>

      <<Starting  on  bc03-n02>>

      Your  home  directory  for  this  job  is  /largefs/my_onyen

      Linux  bc03-n02.isis.unc.edu  2.6.9-42.ELsmp  #1  SMP  Wed  Jul  12  23:27:17

      EDT  2006  i686  i686  i386  GNU/Linux

Note: When you run an interactive job, LSF does not send you e-mail.

Examples of programs that are often run in the “int” queue include SAS, Stata, Matlab, and Mathematica. Members of a patrons group can submit interactive jobs using the “patrons” queue:

More bsub options

Table 1. More Useful options

Option	Details
-b	The “begin” option forces your job to wait until a specific time before being considered for dispatching. For example: bsub -b 11:00 tells LSF to start your job no earlier than 11:00 am. If the current time is after 11:00 am, the job will be held until the next day. bsub -b 2:15:20:15 tells LSF to start the job after 8:15 pm on February 15.
-n	Specifies the number of CPUs or slots that your job needs. This option is required for parallel jobs. Without the number option, LSF will treat your job as if it needed only one job slot. If your parallel job has only one slot it will run its multiple threads on one host. System administrators are likely to kill your job for overloading that host. LSF will believe your number option request for job slots since LSF has no idea how many CPUs your job will actually use. If you ask for too many slots, the system administrator may kill your job so that other jobs can use the resources that your job is wasting.
-m	Specifies the machine on which the job will run. For example, to submit a job on Cypress when logged into Cedar, the -m option can be used (though all batch jobs from Cedar will run on Cypress anyway): bsub -m cypress ...
-o, -e	Saves your job's output in a file instead of sending it to you in e-mail. For example: bsub -o my_output ... You can put stdout and stderr in different files if you wish: bsub -o my_out -e my_err ... To make it easier to keep track of the output from multiple runs of the same program, you can use the special %J variable in your file names. LSF will substitute the LSF JobID number for the %J variable: bsub -o out.%J
-R	Specifies a list of resources required by the job. Examples of resources include software packages, operating systems, type of processors, memory and hard disk space. A list of resources defined for a given node can be seen in the last column (everything within the parentheses) of output of the following command: lshosts \| more To see an explanation of a resource use the “lsinfo” command: lsinfo xeon32
-u	Specifies the e-mail address where you wish to receive your job output. For example: bsub -u mymail@unc.edu If you don't use this option, LSF sends e-mail to the address onyen@email.unc.edu, where “onyen” is your Onyen .

Submitting jobs with bsas and other scripts

Some programs are used so frequently that we have created special wrapper scripts. These scripts construct the bsub command and submit the job for you. More information about these wrapper scripts can be found at the Research Computing - Mathematical and Statistical Software Application Notes page.

Submitting parallel jobs

Parallel jobs execute on more than one CPU at a time so that they finish more quickly than they would if they ran on a single CPU. These jobs run programs that have been designed to run in parallel.

More information about compiling and running parallel applications can be found here.

Additional help

About help.unc.edu

LSF - Submitting Jobs

Table 1. More Useful options

Training