Running codes with Condor

1. Useful links
2. Condor in a nutshell
3. Best practices for codes
4. Preparing your job for Condor
5. Submitting your job
6. Checking the queue
7. Checking status of the nodes
8. Using Condor DAGman and DAGs on the Nemo cluster
9. FAQ
10. Future directions

Useful links

• Condor web pages
• Condor manual

Top

Condor in a nutshell

Condor is a sophisticated batch and queuing system running on the Nemo cluster. Using Condor, LSC scientists can manage and monitor the submission and execution of many, many independent jobs on the cluster.

Condor jobs on the cluster are intended to be "stand-alone" executables, though it is possible to manage some MPI jobs using Condor (send mail to the Nemo administrators for more details).

We have chosen Condor as a batch and queing system on the cluster because

• Condor plays nicely and will "vacate" a node if it is being used for other purpose, such as running LDAS jobs.
• Condor can manage tens of thousands of jobs.
• Condor will not "lose" a job. If a node or even the whole cluster goes down Condor will restart your job (from a recent checkpoint if available).
• Condor checkpoints your jobs now and then so that if a node must be vacated or for some reason the node goes down your work is not lost.
• Condor has a simple, command-line user interface.

In addition it is strongly preferred that codes running on the Nemo cluster run in the Condor "standard" universe (see the Condor manual for definition and details) so that checkpointing is available and the cluster resources are used as efficiently as possible. To run in the standard universe codes must meet the following restrictions:

1. Multi-process jobs are not allowed. This includes system calls such as fork(), exec(), and system().
2. Interprocess communication is not allowed. This includes pipes, semaphores, and shared memory.
3. Network communication must be brief. A job may make network connections using system calls such as socket(), but a network connection left open for long periods will delay checkpointing and migration.
4. Sending or receiving the SIGUSR2 or SIGTSTP signals is not allowed. Condor reserves these signals for its own use. Sending or receiving all other signals is allowed.
5. Alarms, timers, and sleeping are not allowed. This includes system calls such as alarm(), getitimer(), and sleep().
6. Multiple kernel-level threads are not allowed. However, multiple user-level threads are allowed.
7. Memory mapped files are not allowed. This includes system calls such as mmap() and munmap().
8. File locks are allowed, but not retained between checkpoints.
9. All files must be opened read-only or write-only. A file opened for both reading and writing will cause trouble if a job must be rolled back to an old checkpoint image. For compatibility reasons, a file opened for both reading and writing will result in a warning but not an error.
10. Your executable must be statically linked.

Submitting your job

You submit jobs to Condor using a Condor submit script. The submit script contains a list of Condor commands and assigned values of the form

The most often used commands are executable, universe, input, output, error, log, and queue. For a complete list of commands see the manual page for condor_submit.

A submit script for the hello.c job outlined above might look like this:

[user@hydra]$ cat hello.sub
universe   = standard
executable = hello
input      = in.$(Process)
output     = out.$(Process)
error      = err.$(Process)
log        = log.$(Process)

queue 100

When you submit this file to Condor it will queue up 100 instances of your executable hello.

Note the use of the $(Process) macro. Condor will substitute "0" for the first job it queues up, "1" for the second job, and so on. Thus the file in.0 will be used as the stdin for the first job, out.0 as stdout for the first job, and err.0 for the first job.

To submit your jobs to Condor using your prepared submit file use the condor_submit command:

[user@hydra temp]$ condor_submit hello.sub
Submitting job(s)............................................................
........................................
100 job(s) submitted to cluster 1.

Checking the queue

You can check the Condor queue to see the status of your jobs using the condor_q command:

[user@hydra]$ condor_q     


-- Submitter: hydra.phys.uwm.edu : <129.89.201.232:35435> : hydra.phys.uwm.edu
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
  16.19  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.20  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.21  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.22  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.24  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.27  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.30  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.35  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.37  someuser        1/21 16:53   0+00:00:01 R  0   2.7  hello             
  16.40  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.45  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.48  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.55  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.60  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.64  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.70  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.71  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.72  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.73  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.75  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.79  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.80  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.91  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.93  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.95  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.96  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.97  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.98  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             
  16.99  someuser        1/21 16:53   0+00:00:00 R  0   2.7  hello             

29 jobs; 0 idle, 29 running, 0 held

Checking status of the nodes

You can check the status of the cluster nodes using the condor_nemo or condor_status command:

[user@hydra]$ condor_nemo 

Name                    State   Activity        LoadAv

hydra.phys.uwm.e        Unclaimed       Idle            0.020000
nemo-slave0001.        Unclaimed       Idle            0.050000
nemo-slave0002.        Unclaimed       Idle            0.180000
nemo-slave0003.        Unclaimed       Idle            0.040000
nemo-slave0004.        Owner           Idle            0.530000
nemo-slave0005.        Unclaimed       Idle            0.000000
nemo-slave0006.        Unclaimed       Idle            0.000000

...

nemo-slave0291.        Unclaimed       Idle            0.000000
nemo-slave0292.        Unclaimed       Idle            0.000000
nemo-slave0293.        Unclaimed       Idle            0.000000
nemo-slave0294.        Unclaimed       Idle            0.000000
nemo-slave0295.        Unclaimed       Idle            0.000000
nemo-slave0296.        Unclaimed       Idle            0.000000


                     Machines Owner Claimed Unclaimed Matched Preempting

         INTEL/LINUX      297    13       0       284       0          0

               Total      297    13       0       284       0          0

A node in the "Unclaimed" state is ready to run Condor jobs. A node in the "Owner" state is currently being used by jobs outside of Condor\'s control (possibly LDAS) and will not accept Condor jobs at this time. A node in the "Claimed" state has been matched with a Condor job.

Using Condor DAGman and DAGs on the Nemo cluster

From section 2.11 of the Condor Manual:

A directed acyclic graph (DAG) can be used to represent a set of programs where the input, output, or execution of one or more programs is dependent on one or more other programs.

In other words, DAGs can be used to represent an analysis pipeline, and Condor DAGman can be used to ensure that the jobs making up the DAG are executed with the correct dependencies. Please see the link above for more details.

Condor users running DAGman jobs on Nemo should use a directory under /people on the login node hydra.phys.uwm.edu for the log files that DAGman monitors as part of the DAG. This is necessary since file locking via Linux NSF is not robust. The /people filesystem is local to the login node.

Please send email to nemo-admin@gravity.phys.uwm.edu if you need a directory in /people created for you.

FAQ

1. Condor is sending me a lot of mail. How can I stop that?

   Use the notification command in your Condor submit description file. See the documentation for the condor_submit command.

2. My jobs are in the queue but they will not start. Why?

   There are any number of reasons why jobs that are queued may not start:

   1. There may not be any resources (ie. nodes) currently available. Try running condor_q -analyze for details about why your jobs are not running.
   2. Your jobs may have unusual or impossible requirments. Condor uses the concepts of Matchmaking and ClassAds to match jobs with requirements to resources that can meet said requirements. Try running condor_q -long jobid for one of your jobs to see the full set of requirments being advertised for your jobs, and look for anything that you might not have intended.
3. How do I pass unique command-line arguments to each job?

   Use the arguments command in your Condor submit description file. See the documentation for the condor_submit command.

4. My program needs certain environment variables to be set. How do I get Condor to do that?

   Use the environment command in your Condor submit description file. See the documentation for the condor_submit command.

5. Do I really need a seperate input file for each instance of my code?

   No. If each instance of your program reads the same input from stdin then you can simply set

   input = myInput
   

   in your submit file to have each program read from the file myInput.

Future directions

The Condor system includes the notion of jobs "flocking" from one Condor pool (such as Nemo) to another pool (perhaps at another LIGO Tier II site). This enables one to exploit as many resources (CPUs) as are available with no extra effort. At this time, however, flocking can only happen between pools (clusters) which do not live behind a firewall since the "submit machine" (the machine from which jobs are submitted) must be able to communicate directly with the machine on which a job runs.

The UWM-LSC group is helping test a development version of Condor which will enable flocking to proceed even if one or both of the machines involved in a Condor matchmaking are behind a firewall. When in place this version of Condor will allow jobs to flock from Nemo to other pools automatically, increasing job throughput.

In addition, this new version of Condor will allow LSC scientists to submit Condor jobs to the UWM-LSC Nemo cluster directly from their workstations using the Condor-G client. Hence LSC scientists will be able to develop codes and run them in production all from their desktop workstations.