Hoffman2:Submitting Jobs

Back to all things Hoffman2

Job Submission Templates

Please use these job submission template scripts to simplify job submission.

job.q

Once you've identified or written a script you'd like to run, SSH into Hoffman2 and enter job.q. Then it is just a matter of following its step-by-step instructions.

From the tool's main menu, you can type Info to read up about how to use it and we highly encourage you to do so.

But we know patience is a virtue that most of us aren't blessed with. So we'll walk you through submitting a basic job so you can hit the ground running.

Example

1. Once on Hoffman2, you'll need to edit one file so pull out your favorite text editor and edit the file

   ~/.queuerc

2. Add the line

   set qqodir = ~/job-output

3. You've just set the default directory where your job command files will be created. Save the configuration file and close your text editor.
4. Make that directory using the command

   $ mkdir ~/job-output

5. Now execute

   $ job.q

6. Press enter to acknowledge the message about some files that get created (READ IT FIRST THOUGH).
7. Type Build <ENTER> to begin creating an SGE command file.
8. The program now asks you which script you'd like to run, enter the following text to use our example script

   /u/project/CCN/apps/examples/qsub/gather.sh

9. The program now asks how much memory the job will need (in Megabytes). This script is really simple, so let's go with the minimum and enter 64.
10. The program now asks how long will the job take (in hours). Go with the minimum 1 hour; it will complete in much less than this.
11. The program now asks if your job should be limited to only your resource group's cores. Answer n because you do not need to be limiting yourself here and the job is not going to be running for more than 24 hours.
12. Soon, the program will tell you that gather.sh.cmd has been built and saved.
13. When it asks you if you would like to submit your job, say no. Then type Quit <ENTER> to leave the program.
14. Now you should be able to run

    ls ~/job-output

    and see gather.sh.cmd. This file will stay there until you delete it and can be run over and over again. Making a command file like this is especially useful if there is a task you'll be running repeatedly on Hoffman2. But if this is something you only need to run once, you should delete the file so you don't needlessly approach your quota.

15. The time has come to actually run the program (thought we'd never get to that, didn't you?). Type

    qsub job-output/gather.sh.cmd

    and after hitting enter, a message similar to this will pop up:

    Your job 1882940 ("gather.sh.cmd") has been submitted

    where the number is your JobID, a unique numerical identifier for the computer job you have submitted to the queue.

16. Now you can check if the job has finished running by doing

    ls ~/job-output

17. When two files named gather.sh.output.[JOBID] and gather.sh.joblog.[JOBID] (where JOBID is your job's unique identifier) appear, your job has run.

    gather.sh.output.[JOBID]

    This file has all the standard output generated by your script. In this case it will just have the line

    Standard output would appear here.

    gather.sh.joblog.[JOBID]

    This file has all the details about when, where, and how your job was processed. Useful information if you are going to be running this job over and over and need to fine tune the resources it uses.

18. Better ways of checking on your job can be found here.
19. The script you ran is an aggregator. It looks in a list of directories, each assumed to contain a specifically named file, and gathers the contents of each of those files into one central file in your home directory. This file is named gather-[TIMESTAMP].txt where TIMESTAMP is when the script was run and follows ISO 8601 style encoding. You are encouraged to type

    /u/project/CCN/apps/examples/qsub/gather.sh -h

    or

    /u/project/CCN/apps/examples/qsub/gather.sh --help

    to see how this script works.

20. Finally, go check the inbox of the email you used to sign up for your Hoffman2 account. There will be two emails from "root@mail.hoffman2.idre.ucla.edu" that indicate when the job was started and when the job was completed. This is one of the neat features of the queue so that you can be alerted about the progress of your job without having to stay logged into Hoffman2 and checking on it constantly.

qsub

Everything that job.q did can be done on the command line. And it can be done better.

Example

Run the command:

$ qsub -cwd -V -N J1 -l h_data=64M,h_rt=00:05:00 -M $USER -m bea /u/project/CCN/apps/examples/qsub/gather.sh

And something like the following will be printed out:

Your job 1875395 ("J1") has been submitted

Where the number is your JOBID, a unique numerical identifier for your job.

Let's break down the arguments in that command.

-cwd

Change working directory

When your script runs, change the working directory to where you currently are in the filesystem.

e.g. If you were in the director /u/home/mscohen/data/ when you ran the command, the queue will change directories to that location and then execute the script you gave it. This means output and error directories will be placed here for that job.

-V

Export environment variables

Exports all the environment variables to the context of the job. Useful if you have extra environment variables that are needed in your script.

e.g. If you had defined the variable SUBJECT_ID in your session on Hoffman2 (export SUBJECT_ID=42) before submitting a job and that variable was called on by your script, then you would need to use this flag. Tools like FreeSurfer look for certain environment variables to be set.

-N J1

Name my job

Names your job "J1." When you look at the queue, this will be the text that shows up in the "name" column. This will also be the beginning of the output (J1.o[JOBID]) and error (J1.e[JOBID]) files for your job.

-l h_data=64M,h_rt=00:05:00

Resource allocation (that's a lower case "elle")

This is the resources flag meaning that the text immediately after it will ask for things like:

• certain amount of memory, in Megabytes, or Gigabytes
  • h_data=64M (64 MB RAM) or h_data=1G (1 GB RAM)

If your job uses more RAM than it requested, your job WILL be killed in order to avoid it hurting other jobs running on the same node. It is imperative that you set this RAM request properly.

• certain length of computing time, in the form HH:MM:SS
  • h_rt=00:05:00

In this case the script will complete its task rapidly, hence we are only asking for 5 minutes of computing time.

• highp

  • Job length maximum of 14 days but can only be run on nodes belonging to your resource group (type mygroup to see what type of resources you have available). If you are in the ccn group on Hoffman2, you have access to some of these highp nodes.

-M $USER

Define mailing list

This defines the user that will be mailed if email updates are requested. The default address is that of the job-owner.

e.g. In this case, the email will be sent to the address on file for the user.

-m bea

Define mailing rules

This defines when Hoffman2 should email you about your job. There are five options here

• b - when the job begins
• e - when the job ends
• a - when the job is aborted (ends in an error state)

Job Arrays

Example

Sumitting a job array in one qsub command:

qsub -cwd -V -N PJ -l h_data=1024M,h_rt=01:00:00 -M $HOME -m bea -t 1-100:1 myFuncWrapper.sh