Deb's files

The initial set of input files comprised:

Some success

Alt Text

Other files


Building the Compiled Executable: MATLAB 2010a v713

You can fire up MATLAB without the desktop as all we will be doing is typing deploytool, however you do need the JVM here.

Alt Text

Alt Text

The default project, if you are starting a new one, is a Windows Standalone Application, probably not what you want for a batch-processing, non-interactive environment.

Alt Text

so be sure to select Console Application before you start adding files into the project, as there does not seem to be a way to alter the Target afterwards.

Alt Text

On opening the project, you are presented with a tabbed dialog that allows you to specify the fles that will form the project

You can add files into three seperate catergories

1 Main M file

2 Other M files

3 Exteraneous files, eg data that you may wish to package up

Alt Text

Alt Text

Be aware that whilst the packaging of all the files together here may be useful, when you come to run the programs within a grid environment, you may want to have access to the individual files, so there may be no need to package data files just to create the executable.

Note that the execuatble Unititled3.exe takes the name of the project. Changing this can be achieved from within the Settings dialog.

Alt Text

after which

Alt Text

Once you have added the files you want to see in the package, you'll have something like this:

Alt Text

Alt Text

whilst within the Settings dialogs, the following customisations are available

Alt Text

Alt Text

Alt Text

Disabling the JVM, which your console application is unlikely to need, is, reportedly, a "good thing"

Alt Text

Build Log

  <mkdir dir="C:\Documents and Settings\kevin\DebTest\Untitled1\distrib" />
  <mkdir dir="C:\Documents and Settings\kevin\DebTest\Untitled1\src" />
mcc -o Untitled1 -W WinMain:Untitled1 -T link:exe -d 'C:\Documents and Settings\kevin\DebTest\Untitled1\src' -w enable:specified_file_mismatch -w enable:repeated_file -w enable:switch_ignored -w enable:missing_lib_sentinel -w enable:demo_license -v 'C:\Documents and Settings\kevin\DebTest\runwaimarino_multi.m' -a 'C:\Documents and Settings\kevin\DebTest\klinggupta.m' -a 'C:\Documents and Settings\kevin\DebTest\nashsutcliffe.m' -a 'C:\Documents and Settings\kevin\DebTest\performance_multi.m' -a 'C:\Documents and Settings\kevin\DebTest\soilmodel.m' 
Compiler version: 4.13 (R2010a) 
Processing C:\Program Files\MATLAB\R2010a\toolbox\nnet\mcc.enc
Processing C:\Program Files\MATLAB\R2010a\toolbox\stats\mcc.enc
Processing include files...
2 item(s) added.
Processing directories installed with MCR...
The file C:\Documents and Settings\kevin\DebTest\Untitled1\src\mccExcludedFiles.log contains a list of functions excluded from the CTF archive.
2 item(s) added.
Generating MATLAB path for the compiled application...
Created 52 path items.
Begin validation of MEX files: Thu Aug 12 15:21:53 2010 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\DNToMatrix.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\MatrixToDN.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\computeDN.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\convertToDouble.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\matrixBitXor.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolIndexed.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolNet.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolPoint.mexw32'. 
No conflicting M-file found. 
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolSequence.mexw32'. 
No conflicting M-file found. 
End validation of MEX files: Thu Aug 12 15:21:53 2010 
Parsing file "C:\Documents and Settings\kevin\DebTest\runwaimarino_multi.m"
	(Referenced from: "Compiler Command Line"). 
Parsing file "C:\Program Files\MATLAB\R2010a\toolbox\compiler\deploy\deployprint.m"
	(Referenced from: "Compiler Command Line"). 
Parsing file "C:\Program Files\MATLAB\R2010a\toolbox\compiler\deploy\printdlg.m"
	(Referenced from: "Compiler Command Line"). 
Deleting 9 temporary MEX authorization files. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_1880_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_29945_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_32656_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_16162_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_18273_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_15372_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_27761_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_10969_10260.auth'. 
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_8073_10260.auth'. 
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_main.c". 
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_delay_load.c". 
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\readme.txt". 
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_mcc_component_data.c". 
Executing command: ""C:\Program Files\MATLAB\R2010a\bin\mbuild"  -O -v -output "Untitled1" -I"C:\Documents and Settings\kevin\DebTest\Untitled1\src" "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_main.c" "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_delay_load.c" "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_mcc_component_data.c" -link exe  -env MCR_DELAYLOAD=/delayload:mclmcrrt713.dll -env MCR_DELAYLIB=delayimp.lib -outdir "C:\Documents and Settings\kevin\DebTest\Untitled1\src"" 
 This is mbuild Copyright 1984-2006 The MathWorks, Inc.
-> Default options filename found in C:\Documents and Settings\kevin\Application Data\MathWorks\MATLAB\R2010a
->    Options file           = C:\Documents and Settings\kevin\Application Data\MathWorks\MATLAB\R2010a\compopts.bat
->    COMPILER               = cl
->    Compiler flags:
         COMPFLAGS           = -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE
         OPTIMFLAGS          = -O2 -DNDEBUG
         DEBUGFLAGS          = -Z7
         arguments           =  -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC
         Name switch         = /Fo
->    Pre-linking commands   = 
->    LINKER                 = link
->    Link directives:
         LINKFLAGS           = kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib oleaut32.lib ole32.lib uuid.lib odbc32.lib odbccp32.lib /LIBPATH:"C:\PROGRA~1\MATLAB\R2010a\extern\lib\win32\microsoft" /nologo /manifest /manifestfile:"C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest" mclmcrrt.lib delayimp.lib /delayload:mclmcrrt713.dll
         LINKFLAGSPOST       = 
         Name directive      = "/out:C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.exe"
         File link directive = 
         Lib. link directive = 
         Rsp file indicator  = @
->    Resource Compiler      = rc /fo "C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\.res"
->    Resource Linker        = 
--> cl  -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE /FoC:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_main.obj -IC:\PROGRA~1\MATLAB\R2010a\extern\include -IC:\PROGRA~1\MATLAB\R2010a\simulink\include -O2 -DNDEBUG "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_main.c"
--> cl  -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE /FoC:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_delay_load.obj -IC:\PROGRA~1\MATLAB\R2010a\extern\include -IC:\PROGRA~1\MATLAB\R2010a\simulink\include -O2 -DNDEBUG "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_delay_load.c"
--> cl  -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE /FoC:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_mcc_component_data.obj -IC:\PROGRA~1\MATLAB\R2010a\extern\include -IC:\PROGRA~1\MATLAB\R2010a\simulink\include -O2 -DNDEBUG "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_mcc_component_data.c"
    Contents of C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\mbuild_tmp.rsp:
  C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_main.obj  C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_delay_load.obj  C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_mcc_component_data.obj
--> link "/out:C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.exe" kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib oleaut32.lib ole32.lib uuid.lib odbc32.lib odbccp32.lib /LIBPATH:"C:\PROGRA~1\MATLAB\R2010a\extern\lib\win32\microsoft" /nologo /manifest /manifestfile:"C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest" mclmcrrt.lib delayimp.lib /delayload:mclmcrrt713.dll  @C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\MBUILD~1.RSP  
--> "if exist C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\templib.def del C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\templib.def"
--> mt.exe -outputresource:"C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.exe";1 -manifest "C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest"
Microsoft (R) Manifest Tool version 5.2.3790.2075
Copyright (c) Microsoft Corporation 2005. 
All rights reserved.
--> del "C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest"
copy 'C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1.exe' 'C:\Documents and Settings\kevin\DebTest\Untitled1\distrib\Untitled1.exe'
copy 'C:\Documents and Settings\kevin\DebTest\Untitled1\src\readme.txt' 'C:\Documents and Settings\kevin\DebTest\Untitled1\distrib\readme.txt'

Package Log

<ant />

Finally, success!

Alt Text

All you need to remember is that the default project Standalone Windows Application is not what you need and choose Console Application when you first create the project.

Once you have accepted the default, you won't get any console.

Running Compiled MATLAB codes on the SCS/Condor Grid

Around September 2010, ITS rolled out the MATLAB Compiler Runtime (MCR) to SCS machines that participate in the Condor grid that runs across them.

This allows users who have an SCS/Condor grid account to run MATLAB codes across unused machines within the 950 or so that ITS provide as general lab PCs.

(NOTE: the above turned out to be a false hope. Some eighteen months (Feb 2012) after they were supposed to have rolled the MCR out, ITS still had not managed to do it.)

Summary of the Basics

There is a guide, aimed at SECS users, to the SCS/Condor Grid.

It is recommended that anyone wishing to use the SCS/Condor Grid reads it.

It can be found here

Accessing, and copying files to/from, the SCS/Condor Grid

If you are not an ECS user, you will need access to the equivalents of the secure shell (ssh) and secure copy (scp) utilities for your platform.

Files required to run a simple job

There are two ways to run even the simplest of jobs on the SCS/Condor grid.

The first invloves simply specifying the program (.EXE) you wish to run and the environment you need to run it in, in a job submission script (.CMD).

The second involves using the job submission script (.CMD) to run a DOS batch (.BAT) file, which in turn, runs the program (.EXE) and is used to set the environment you need to run it in.

Compiled MATLAB Console Application (.EXE)

We'll assume that you have created a compiled MATLAB executable called GridTest713.exe on a windows box that provides you with access to MATLAB version 2010a/v713 and have copied this file to a directory on the Condor master machine vuwunicondgrd01.

Condor Job Submission Script (.CMD): direct invocation of a .EXE

You'll now need a job submission script that will tell the software system controlling the grid what you want to do with it. We'll assume that this file is called gridtest713.cmd. Note that the extension can be anything you want and may not even be supplied, however it can be useful to differentiate the various files, targetted at the underlying windows boxes the grid jobs will run on, in this way.

A typical job submission script for running compiled MATLAB programs would look like this:

universe = vanilla
environment = "path='c:\WINDOWS\SYSTEM32;C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32'"
executable = GridTest713.exe
output     = gridtest.out.$(Cluster).$(Process)
error      = gridtest.err.$(Cluster).$(Process)
log        = gridtest.log.$(Cluster).$(Process)
Requirements = (OpSys == "WINNT51")
ShouldTransferFiles  = YES
WhenToTransferOutput = ON_EXIT
queue 1

The directives worthy of note are:

Submitting the job to the SCS/Condor grid

We'll assume that you are now logged into the Condor front-end machine and that both the job submission script gridtest713.cmd and program GridTest713.exe are in the current directory

We then submit the job by typing

condor_submit gridtest713.cmd

after which, you will see some acknowledgement from the Condor system

$ condor_submit gridtest713.cmd
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 95452.

and may notice that three files have been created in the current directory.

$ ls -otr
-rw-rw-r--    1 username        0 Aug 27 16:11 gridtest713.out.95452.0
-rw-rw-r--    1 username        0 Aug 27 16:11 gridtest713.err.95452.0
-rw-r--r--    1 username       86 Aug 27 16:11 gridtest713.log.95452.0

Note that the sub tasks (Process) in a Cluster are numbered from 0

Those initial 86 bytes appearing in the log file are the submission details.

We can follow the progress of the job by using the condor_q command and restricting the output to our job (Cluster)

Note: invoking condor_q without any specific job id will show all of the jobs currently in the system

$ condor_q 95452

-- Submitter: : <> :
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
95452.0   username        8/27 16:24   0+00:00:00 I  0   0.1  GridTest713.exe

The status (ST) column shows I at this point, indicating the job is "Idle", or waiting in the queue.

Once the job is running, the ST column will show R and the RUN TIME should be non-zero

$ condor_q 95452

-- Submitter: : <> :
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
95452.0   username        8/27 16:24   0+00:00:02 R  0   0.1  GridTest713.exe   

whilst once it has finished, it will not show up at all in the condor_q output

$ condor_q 95452

-- Submitter: : <> :

After the job has finished, any new or modified files, can now be found in the current directory

$ ls -otr
-rw-------    1 username      129 Aug 27 16:32 MCRtest1b.txt
-rw-------    1 username       78 Aug 27 16:32 MCRtest1a.txt
-rw-rw-r--    1 username        0 Aug 27 16:32 gridtest.out.95452.0
-rw-r--r--    1 username      626 Aug 27 16:32 gridtest.log.95452.0
-rw-rw-r--    1 username        0 Aug 27 16:32 gridtest.err.95452.0

Note that in this case, a single task, the output files created by the job itself MCRtest1a.txt and MCRtest1b.txt give no indication that they came from Cluster 95452, Process 0.

We will look at possible ways to achieve that, later on

Condor Job Submission Script (.CMD): indirect invocation via a .BAT

... job submission file, no need to add MCR path to that given as the environment ... but supplying the .BAT file as the "program" to run ... and then needing to explicitly copy the .EXE over, so it can be found by the .BAT file

environment = "path=c:\WINDOWS\SYSTEM32"
executable  = gridtest713.bat
TransferInputFiles  = GridTest713.exe

... batch file, containing the setting of the path and invoking the program ...

set PATH=C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32;%PATH%

Condor Job Submission Script (.CMD): passing job info to the DOS environment

As mentioned above, it can be useful to give the programs running on the windows machines access to the environmental variables that the Condor system provides and which are used in the example job submission script to differentiate system output between and within jobs.

This may be achieved by adding to the environment= directive, for example:

environment = "path='c:\WINDOWS\SYSTEM32;C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32' CONDOR_CLUSTER=$(Cluster) CONDOR_PROCESS=$(Process)"

Note that the extra environmental variables are within the double quotes

With that information placed into the DOS environment, we might explicitly pass that information as arguments to the executable in our .BAT file

set PATH=C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32;%PATH%

Note: the DOS variable holding the existing path, set in our job submisson script, and to which we add our MCR path, is available to us as %PATH%. Similarly, other variables are available as using the standard %VARNAME% tokens

or use that information to locally modify the known output files, as the last part of the job

set PATH=C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32;%PATH%
ren MCRtest1a.txt MCRtest1a.txt.%CONDOR_CLUSTER%.%CONDOR_PROCESS%
ren MCRtest1b.txt MCRtest1b.txt.%CONDOR_CLUSTER%.%CONDOR_PROCESS%

Note: the DOS command ren is the equivalent of the UNIX mv.

This works because the file transfer occurs at the end of the job, in this case, after the last command line in the .BAT file.

Passing as part of arguments= ??

Running Compiled MATLAB codes on the SGEES multi-core box

Around September 2010, SGEES rolled out the MATLAB Compiler Runtime (MCR) to its multi-core machine.

This allows users who have a SGEES account to run MATLAB codes across its 64 cores and with access to its 128GB of memory

Summary of the Basics

The SGEES multi-core box runs a GNU/Linux (UNIX-like) operating system from a Ubuntu distribution

The scheduling of jobs submitted to it is controlled by the Grid Engine (formerly Sun Grid Engine, SGE) system, from front-end machines within SGEES

The front-end machines give one access to the MATLAB compiler in its command line form

ECS provide a technical note to submitting jobs to their publically accessible SGE Grid which may be useful to a user new to the SGE concepts.

You can find that guide here

Accessing, and copying files to/from, the SGEES multi-core box

If you are not an SGEES user, you will need access to the equivalents of the secure shell (ssh) and secure copy (scp) utilities for your platform.

Files required to run a multi-M-file job that reads data files

This example will demonstrate both the compilation of the source files with the command-line version of the MATLAB Compiler (mcc) and the submission of the job via the Grid Engine's qsub command

Because the SGEES filestore is visible to both the front-end machines and the back-end resource running the jobs, there is no need to stage data between the front- and back-ends

We use as an example a multi-M-file, rainfall-runoff simualtion model that comprises the following six M-files and two input data files

all of which we will assume are in the same directory on one of the SGEES front-end machines

evapinterpolated.mat   performance_multi.m   taupointerpolated.mat
klinggupta.m           outputs.m             soilmodel.m
nashsutcliffe.m        runwaimarino_multi.m

Compiling the multi-M-file execuatble

Note: if you have created a project file (.prj) when compiling the M files on another platform or when using MATLAB's deploytool, you can use that as the driver for the compilation, however we will assume that you only have the M-files and data files

We invoke the MATLAB compiler as follows

mcc -m -R -nojvm -R -nodisplay -v runwaimarino_multi.m klinggupta.m  \
  nashsutcliffe.m  outputs.m performance_multi.m soilmodel.m


It has since been discovered that the default compilation mode tries to grab up to 32 threads, which is obviously not what you want when running in a shared environment

We now invoke the MATLAB compiler as follows

mcc -m -R -singleCompThread -R -nojvm -R -nodisplay -v runwaimarino_multi.m \
  klinggupta.m nashsutcliffe.m outputs.m performance_multi.m soilmodel.m


Note that we put the M-file containing the main routine for the program as a whole, as the first M-file in the list. This will see the generation of an executable file with the same name as that M-file, but without the .m extension.

The following files of interest will be created

a project description file (.prj), the executable file itself runwaimarino_multi and a wrapper shell script, which, whilst specific to this executable, can be used to run the executable against a MATLAB Compiler Runtime environment located along arbitrary paths.

Note: as we are running within a grid where a generic wrapper is provided for us, we will not need the generated wrapper when running our executable.

Submitting the job to the SGEES multi-core box

The Grid Engine job submission script

In order to tell the Grid Engine what we want to run and where to run it from, we create a job submission script

#$ -S /bin/sh
#$ -wd /Users/home/kevin/Projects/GridTest/DebMaxwell
run_mcr runwaimarino_multi

A fuller explantion of the syntax of Grid Engine job submission scripts can be found in the documentation for the system, however, the line

#$ -wd /Users/home/kevin/Projects/GridTest/DebMaxwell

says that the job should be run as though starting from the directory supplied there

The run_mcr is the generic MATLAB Compiler Runtime wrapper for the SGEES multi-core box and it takes as its arguments, the program we wish to run and any other arguments to that program.

Note that because we are running from the directory containing our input (.mat) files, there are no arguments to our compiled execuatble

To submit the job into the grid, we use the qsub command as follows

Your job 44 ("") has been submitted
after which we can inspect the status of our jobs in the system using the qstat command

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
     44 0.55500 runwaimari kevin        r     08/30/2010 11:58:23       1 

where the r denotes a running job.

A q would indicate that our job was waiting in the queue for resources to become available

Once the job is running, the standard output and standard error channels for the executable will be directed, by default, to seperate files taking the name of the job submission script and appending either and e or an o and the job-ID as follows:

$ls -otr
-rw-r--r-- 1 kevin        0 Aug 30 11:58
-rw-r--r-- 1 kevin     1418 Aug 30 12:00

Delayed starting of SCS/Condor Grid jobs

Might as well put this here for now

OK, here's how to make the jobs start at say, 1am, when you think you might get six or seven hours of uniterrupted computing on each machine you can get your hands on overnight.

It's a distillation of Section 2.12.1 in the Condor Manual

Choose the Stable Release, Online HTML to view the relevant stuff.

Basically, you need to add three lines to the submission file, eg

deferral_time      = 1291291200
deferral_window    = 18000
deferral_prep_time = 120

Those values are in seconds.

The reason the time is so large is that it's the number of seconds since midnight GMT on Jan 1, 1970.

What the above says is:

start my job (or jobs if you queue more than one per submission file) at exactly 2010-12-03 01:00:00 or within a window of 5 hours afterwards, however don't try and grab any resources until 2 minutes before the time I'd like it to start.

Note that you need to be specific about the date, not just the time of day to remove possible confusion.

When you submit the job(s) they'll appear as "Idle" in the queue up until 2 minutes before the proposed start time when, assuming there are resources available, they'll start to grab resources.

The reason for having the 2 minutes is that without it, you might grab the resources straight away, ie hog the machine with nothing else running on it until the start time. This potentially wastes resources and defeats the object of the grid.

The reason for having the window is to allow your job to start if it can't get resources at the exact time specified.

Whilst I am sure you will be able to do the math each time you need to calculate the number of seconds since Jan 1, 1970, on most UNIX boxes you should be able to do the following

$ date -d "2010-12-03 01:00:00" +%s

and get back


The manual suggests you'ld type

$ date --date "12/03/2010 01:00:00" +%s

with an American date style, though personally I get confused.

I am not sure how you would achieve that from a windows box but you have access to the Condor master so you could do it on there.