The initial set of input files comprised:
You can fire up MATLAB without the desktop as all we will be doing is typing deploytool, however you do need the JVM here.
The default project, if you are starting a new one, is a Windows Standalone Application, probably not what you want for a batch-processing, non-interactive environment.
so be sure to select Console Application before you start adding files into the project, as there does not seem to be a way to alter the Target afterwards.
On opening the project, you are presented with a tabbed dialog that allows you to specify the fles that will form the project
You can add files into three seperate catergories
1 Main M file
2 Other M files
3 Exteraneous files, eg data that you may wish to package up
Be aware that whilst the packaging of all the files together here may be useful, when you come to run the programs within a grid environment, you may want to have access to the individual files, so there may be no need to package data files just to create the executable.
Note that the execuatble Unititled3.exe takes the name of the project. Changing this can be achieved from within the Settings dialog.
after which
Once you have added the files you want to see in the package, you'll have something like this:
whilst within the Settings dialogs, the following customisations are available
Disabling the JVM, which your console application is unlikely to need, is, reportedly, a "good thing"
ant:
<ant>
<mkdir dir="C:\Documents and Settings\kevin\DebTest\Untitled1\distrib" />
<mkdir dir="C:\Documents and Settings\kevin\DebTest\Untitled1\src" />
</ant>
mcc -o Untitled1 -W WinMain:Untitled1 -T link:exe -d 'C:\Documents and Settings\kevin\DebTest\Untitled1\src' -w enable:specified_file_mismatch -w enable:repeated_file -w enable:switch_ignored -w enable:missing_lib_sentinel -w enable:demo_license -v 'C:\Documents and Settings\kevin\DebTest\runwaimarino_multi.m' -a 'C:\Documents and Settings\kevin\DebTest\klinggupta.m' -a 'C:\Documents and Settings\kevin\DebTest\nashsutcliffe.m' -a 'C:\Documents and Settings\kevin\DebTest\performance_multi.m' -a 'C:\Documents and Settings\kevin\DebTest\soilmodel.m'
Compiler version: 4.13 (R2010a)
Processing C:\Program Files\MATLAB\R2010a\toolbox\nnet\mcc.enc
Processing C:\Program Files\MATLAB\R2010a\toolbox\stats\mcc.enc
Processing include files...
2 item(s) added.
Processing directories installed with MCR...
The file C:\Documents and Settings\kevin\DebTest\Untitled1\src\mccExcludedFiles.log contains a list of functions excluded from the CTF archive.
2 item(s) added.
Generating MATLAB path for the compiled application...
Created 52 path items.
Begin validation of MEX files: Thu Aug 12 15:21:53 2010
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\DNToMatrix.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\MatrixToDN.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\computeDN.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\convertToDouble.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\matrixBitXor.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolIndexed.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolNet.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolPoint.mexw32'.
No conflicting M-file found.
Validating 'C:\Program Files\MATLAB\R2010a\toolbox\stats\@sobolset\private\sobolSequence.mexw32'.
No conflicting M-file found.
End validation of MEX files: Thu Aug 12 15:21:53 2010
Parsing file "C:\Documents and Settings\kevin\DebTest\runwaimarino_multi.m"
(Referenced from: "Compiler Command Line").
Parsing file "C:\Program Files\MATLAB\R2010a\toolbox\compiler\deploy\deployprint.m"
(Referenced from: "Compiler Command Line").
Parsing file "C:\Program Files\MATLAB\R2010a\toolbox\compiler\deploy\printdlg.m"
(Referenced from: "Compiler Command Line").
Deleting 9 temporary MEX authorization files.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_1880_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_29945_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_32656_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_16162_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_18273_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_15372_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_27761_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_10969_10260.auth'.
Removing: 'C:\DOCUME~1\kevin\LOCALS~1\Temp\17\mathworks_tmp_10260_8073_10260.auth'.
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_main.c".
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_delay_load.c".
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\readme.txt".
Generating file "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_mcc_component_data.c".
Executing command: ""C:\Program Files\MATLAB\R2010a\bin\mbuild" -O -v -output "Untitled1" -I"C:\Documents and Settings\kevin\DebTest\Untitled1\src" "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_main.c" "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_delay_load.c" "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_mcc_component_data.c" -link exe -env MCR_DELAYLOAD=/delayload:mclmcrrt713.dll -env MCR_DELAYLIB=delayimp.lib -outdir "C:\Documents and Settings\kevin\DebTest\Untitled1\src""
This is mbuild Copyright 1984-2006 The MathWorks, Inc.
-> Default options filename found in C:\Documents and Settings\kevin\Application Data\MathWorks\MATLAB\R2010a
----------------------------------------------------------------
-> Options file = C:\Documents and Settings\kevin\Application Data\MathWorks\MATLAB\R2010a\compopts.bat
-> COMPILER = cl
-> Compiler flags:
COMPFLAGS = -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE
OPTIMFLAGS = -O2 -DNDEBUG
DEBUGFLAGS = -Z7
arguments = -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC
Name switch = /Fo
-> Pre-linking commands =
-> LINKER = link
-> Link directives:
LINKFLAGS = kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib oleaut32.lib ole32.lib uuid.lib odbc32.lib odbccp32.lib /LIBPATH:"C:\PROGRA~1\MATLAB\R2010a\extern\lib\win32\microsoft" /nologo /manifest /manifestfile:"C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest" mclmcrrt.lib delayimp.lib /delayload:mclmcrrt713.dll
LINKFLAGSPOST =
Name directive = "/out:C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.exe"
File link directive =
Lib. link directive =
Rsp file indicator = @
-> Resource Compiler = rc /fo "C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\.res"
-> Resource Linker =
----------------------------------------------------------------
--> cl -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE /FoC:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_main.obj -IC:\PROGRA~1\MATLAB\R2010a\extern\include -IC:\PROGRA~1\MATLAB\R2010a\simulink\include -O2 -DNDEBUG "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_main.c"
Untitled1_main.c
--> cl -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE /FoC:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_delay_load.obj -IC:\PROGRA~1\MATLAB\R2010a\extern\include -IC:\PROGRA~1\MATLAB\R2010a\simulink\include -O2 -DNDEBUG "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_delay_load.c"
Untitled1_delay_load.c
--> cl -IC:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC -MD -c -Zp8 -GR -W3 -EHsc- -Zc:wchar_t- -nologo /D_CRT_SECURE_NO_DEPRECATE /FoC:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_mcc_component_data.obj -IC:\PROGRA~1\MATLAB\R2010a\extern\include -IC:\PROGRA~1\MATLAB\R2010a\simulink\include -O2 -DNDEBUG "C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1_mcc_component_data.c"
Untitled1_mcc_component_data.c
Contents of C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\mbuild_tmp.rsp:
C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_main.obj C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_delay_load.obj C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\Untitled1_mcc_component_data.obj
--> link "/out:C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.exe" kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib oleaut32.lib ole32.lib uuid.lib odbc32.lib odbccp32.lib /LIBPATH:"C:\PROGRA~1\MATLAB\R2010a\extern\lib\win32\microsoft" /nologo /manifest /manifestfile:"C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest" mclmcrrt.lib delayimp.lib /delayload:mclmcrrt713.dll @C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\MBUILD~1.RSP
--> "if exist C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\templib.def del C:\DOCUME~1\KEVIN\LOCALS~1\TEMP\17\MBUILD~1\templib.def"
--> mt.exe -outputresource:"C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.exe";1 -manifest "C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest"
Microsoft (R) Manifest Tool version 5.2.3790.2075
Copyright (c) Microsoft Corporation 2005.
All rights reserved.
--> del "C:\DOCUME~1\KEVIN\DEBTEST\UNTITL~1\SRC\Untitled1.msvc.manifest"
copy 'C:\Documents and Settings\kevin\DebTest\Untitled1\src\Untitled1.exe' 'C:\Documents and Settings\kevin\DebTest\Untitled1\distrib\Untitled1.exe'
copy 'C:\Documents and Settings\kevin\DebTest\Untitled1\src\readme.txt' 'C:\Documents and Settings\kevin\DebTest\Untitled1\distrib\readme.txt'
ant: <ant /> com.mathworks.mde.deploytool.engine.ZipCommand@bcbc
All you need to remember is that the default project Standalone Windows Application is not what you need and choose Console Application when you first create the project.
Once you have accepted the default, you won't get any console.
Around September 2010, ITS rolled out the MATLAB Compiler Runtime (MCR) to SCS machines that participate in the Condor grid that runs across them.
This allows users who have an SCS/Condor grid account to run MATLAB codes across unused machines within the 950 or so that ITS provide as general lab PCs.
(NOTE: the above turned out to be a false hope. Some eighteen months (Feb 2012) after they were supposed to have rolled the MCR out, ITS still had not managed to do it.)
There is a guide, aimed at SECS users, to the SCS/Condor Grid.
It is recommended that anyone wishing to use the SCS/Condor Grid reads it.
It can be found here
If you are not an ECS user, you will need access to the equivalents of the secure shell (ssh) and secure copy (scp) utilities for your platform.
There are two ways to run even the simplest of jobs on the SCS/Condor grid.
The first invloves simply specifying the program (.EXE) you wish to run and the environment you need to run it in, in a job submission script (.CMD).
The second involves using the job submission script (.CMD) to run a DOS batch (.BAT) file, which in turn, runs the program (.EXE) and is used to set the environment you need to run it in.
We'll assume that you have created a compiled MATLAB executable called GridTest713.exe on a windows box that provides you with access to MATLAB version 2010a/v713 and have copied this file to a directory on the Condor master machine vuwunicondgrd01.
You'll now need a job submission script that will tell the software system controlling the grid what you want to do with it. We'll assume that this file is called gridtest713.cmd. Note that the extension can be anything you want and may not even be supplied, however it can be useful to differentiate the various files, targetted at the underlying windows boxes the grid jobs will run on, in this way.
A typical job submission script for running compiled MATLAB programs would look like this:
universe = vanilla environment = "path='c:\WINDOWS\SYSTEM32;C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32'" executable = GridTest713.exe output = gridtest.out.$(Cluster).$(Process) error = gridtest.err.$(Cluster).$(Process) log = gridtest.log.$(Cluster).$(Process) Requirements = (OpSys == "WINNT51") ShouldTransferFiles = YES WhenToTransferOutput = ON_EXIT queue 1
The directives worthy of note are:
We'll assume that you are now logged into the Condor front-end machine and that both the job submission script gridtest713.cmd and program GridTest713.exe are in the current directory
We then submit the job by typing
condor_submit gridtest713.cmd
after which, you will see some acknowledgement from the Condor system
$ condor_submit gridtest713.cmd Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 95452. $
and may notice that three files have been created in the current directory.
$ ls -otr ... -rw-rw-r-- 1 username 0 Aug 27 16:11 gridtest713.out.95452.0 -rw-rw-r-- 1 username 0 Aug 27 16:11 gridtest713.err.95452.0 -rw-r--r-- 1 username 86 Aug 27 16:11 gridtest713.log.95452.0
Note that the sub tasks (Process) in a Cluster are numbered from 0
Those initial 86 bytes appearing in the log file are the submission details.
We can follow the progress of the job by using the condor_q command and restricting the output to our job (Cluster)
Note: invoking condor_q without any specific job id will show all of the jobs currently in the system
$ condor_q 95452 -- Submitter: vuwunicondgrd01.vuw.ac.nz : <130.195.85.70:50370> : vuwunicondgrd01.vuw.ac.nz ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 95452.0 username 8/27 16:24 0+00:00:00 I 0 0.1 GridTest713.exe
The status (ST) column shows I at this point, indicating the job is "Idle", or waiting in the queue.
Once the job is running, the ST column will show R and the RUN TIME should be non-zero
$ condor_q 95452 -- Submitter: vuwunicondgrd01.vuw.ac.nz : <130.195.85.70:50370> : vuwunicondgrd01.vuw.ac.nz ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 95452.0 username 8/27 16:24 0+00:00:02 R 0 0.1 GridTest713.exe $
whilst once it has finished, it will not show up at all in the condor_q output
$ condor_q 95452 -- Submitter: vuwunicondgrd01.vuw.ac.nz : <130.195.85.70:50370> : vuwunicondgrd01.vuw.ac.nz ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD $
After the job has finished, any new or modified files, can now be found in the current directory
$ ls -otr ... -rw------- 1 username 129 Aug 27 16:32 MCRtest1b.txt -rw------- 1 username 78 Aug 27 16:32 MCRtest1a.txt -rw-rw-r-- 1 username 0 Aug 27 16:32 gridtest.out.95452.0 -rw-r--r-- 1 username 626 Aug 27 16:32 gridtest.log.95452.0 -rw-rw-r-- 1 username 0 Aug 27 16:32 gridtest.err.95452.0 $
Note that in this case, a single task, the output files created by the job itself MCRtest1a.txt and MCRtest1b.txt give no indication that they came from Cluster 95452, Process 0.
We will look at possible ways to achieve that, later on
... job submission file, no need to add MCR path to that given as the environment ... but supplying the .BAT file as the "program" to run ... and then needing to explicitly copy the .EXE over, so it can be found by the .BAT file
environment = "path=c:\WINDOWS\SYSTEM32" executable = gridtest713.bat ... TransferInputFiles = GridTest713.exe
... batch file, containing the setting of the path and invoking the program ...
set PATH=C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32;%PATH% GridTest1.exe
As mentioned above, it can be useful to give the programs running on the windows machines access to the environmental variables that the Condor system provides and which are used in the example job submission script to differentiate system output between and within jobs.
This may be achieved by adding to the environment= directive, for example:
environment = "path='c:\WINDOWS\SYSTEM32;C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32' CONDOR_CLUSTER=$(Cluster) CONDOR_PROCESS=$(Process)"
Note that the extra environmental variables are within the double quotes
With that information placed into the DOS environment, we might explicitly pass that information as arguments to the executable in our .BAT file
set PATH=C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32;%PATH% GridTest1.exe %CONDOR_CLUSTER% %CONDOR_PROCESS%
Note: the DOS variable holding the existing path, set in our job submisson script, and to which we add our MCR path, is available to us as %PATH%. Similarly, other variables are available as using the standard %VARNAME% tokens
or use that information to locally modify the known output files, as the last part of the job
set PATH=C:\Program Files\MATLAB\MATLAB Compiler Runtime\v713\runtime\win32;%PATH% GridTest1.exe ren MCRtest1a.txt MCRtest1a.txt.%CONDOR_CLUSTER%.%CONDOR_PROCESS% ren MCRtest1b.txt MCRtest1b.txt.%CONDOR_CLUSTER%.%CONDOR_PROCESS%
Note: the DOS command ren is the equivalent of the UNIX mv.
This works because the file transfer occurs at the end of the job, in this case, after the last command line in the .BAT file.
Passing as part of arguments= ??
Around September 2010, SGEES rolled out the MATLAB Compiler Runtime (MCR) to its multi-core machine.
This allows users who have a SGEES account to run MATLAB codes across its 64 cores and with access to its 128GB of memory
The SGEES multi-core box runs a GNU/Linux (UNIX-like) operating system from a Ubuntu distribution
The scheduling of jobs submitted to it is controlled by the Grid Engine (formerly Sun Grid Engine, SGE) system, from front-end machines within SGEES
The front-end machines give one access to the MATLAB compiler in its command line form
ECS provide a technical note to submitting jobs to their publically accessible SGE Grid which may be useful to a user new to the SGE concepts.
You can find that guide here
If you are not an SGEES user, you will need access to the equivalents of the secure shell (ssh) and secure copy (scp) utilities for your platform.
This example will demonstrate both the compilation of the source files with the command-line version of the MATLAB Compiler (mcc) and the submission of the job via the Grid Engine's qsub command
Because the SGEES filestore is visible to both the front-end machines and the back-end resource running the jobs, there is no need to stage data between the front- and back-ends
We use as an example a multi-M-file, rainfall-runoff simualtion model that comprises the following six M-files and two input data files
all of which we will assume are in the same directory on one of the SGEES front-end machines sgees010.geo.vuw.ac.nz
$ls evapinterpolated.mat performance_multi.m taupointerpolated.mat klinggupta.m outputs.m soilmodel.m nashsutcliffe.m runwaimarino_multi.m
Note: if you have created a project file (.prj) when compiling the M files on another platform or when using MATLAB's deploytool, you can use that as the driver for the compilation, however we will assume that you only have the M-files and data files
We invoke the MATLAB compiler as follows
mcc -m -R -nojvm -R -nodisplay -v runwaimarino_multi.m klinggupta.m \ nashsutcliffe.m outputs.m performance_multi.m soilmodel.m
--------------UPDATE----------------
It has since been discovered that the default compilation mode tries to grab up to 32 threads, which is obviously not what you want when running in a shared environment
We now invoke the MATLAB compiler as follows
mcc -m -R -singleCompThread -R -nojvm -R -nodisplay -v runwaimarino_multi.m \ klinggupta.m nashsutcliffe.m outputs.m performance_multi.m soilmodel.m
--------------UPDATE----------------
Note that we put the M-file containing the main routine for the program as a whole, as the first M-file in the list. This will see the generation of an executable file with the same name as that M-file, but without the .m extension.
The following files of interest will be created
a project description file (.prj), the executable file itself runwaimarino_multi and a wrapper shell script runwaimarino_multi.sh, which, whilst specific to this executable, can be used to run the executable against a MATLAB Compiler Runtime environment located along arbitrary paths.
Note: as we are running within a grid where a generic wrapper is provided for us, we will not need the generated wrapper when running our executable.
In order to tell the Grid Engine what we want to run and where to run it from, we create a job submission script runwaimarino_multi-qsub.sh
#!/bin/sh # #$ -S /bin/sh #$ -wd /Users/home/kevin/Projects/GridTest/DebMaxwell # run_mcr runwaimarino_multi
A fuller explantion of the syntax of Grid Engine job submission scripts can be found in the documentation for the system, however, the line
#$ -wd /Users/home/kevin/Projects/GridTest/DebMaxwell
says that the job should be run as though starting from the directory supplied there
The run_mcr is the generic MATLAB Compiler Runtime wrapper for the SGEES multi-core box and it takes as its arguments, the program we wish to run and any other arguments to that program.
Note that because we are running from the directory containing our input (.mat) files, there are no arguments to our compiled execuatble
To submit the job into the grid, we use the qsub command as follows
$qsub runwaimarino_multi-qsub.sh
Your job 44 ("runwaimarino_multi-qsub.sh") has been submitted
after which we can inspect the status of our jobs in the system using the qstat command
$qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
44 0.55500 runwaimari kevin r 08/30/2010 11:58:23 all.q@sgees017.geo.vuw.ac.nz 1
where the r denotes a running job.
A q would indicate that our job was waiting in the queue for resources to become available
Once the job is running, the standard output and standard error channels for the executable will be directed, by default, to seperate files taking the name of the job submission script and appending either and e or an o and the job-ID as follows:
$ls -otr ... -rw-r--r-- 1 kevin 0 Aug 30 11:58 runwaimarino_multi-qsub.sh.e44 -rw-r--r-- 1 kevin 1418 Aug 30 12:00 runwaimarino_multi-qsub.sh.o44
Might as well put this here for now
OK, here's how to make the jobs start at say, 1am, when you think you might get six or seven hours of uniterrupted computing on each machine you can get your hands on overnight.
It's a distillation of Section 2.12.1 in the Condor Manual
http://www.cs.wisc.edu/condor/manual/
Choose the Stable Release, Online HTML to view the relevant stuff.
Basically, you need to add three lines to the submission file, eg
deferral_time = 1291291200 deferral_window = 18000 deferral_prep_time = 120
Those values are in seconds.
The reason the time is so large is that it's the number of seconds since midnight GMT on Jan 1, 1970.
What the above says is:
start my job (or jobs if you queue more than one per submission file) at exactly 2010-12-03 01:00:00 or within a window of 5 hours afterwards, however don't try and grab any resources until 2 minutes before the time I'd like it to start.
Note that you need to be specific about the date, not just the time of day to remove possible confusion.
When you submit the job(s) they'll appear as "Idle" in the queue up until 2 minutes before the proposed start time when, assuming there are resources available, they'll start to grab resources.
The reason for having the 2 minutes is that without it, you might grab the resources straight away, ie hog the machine with nothing else running on it until the start time. This potentially wastes resources and defeats the object of the grid.
The reason for having the window is to allow your job to start if it can't get resources at the exact time specified.
Whilst I am sure you will be able to do the math each time you need to calculate the number of seconds since Jan 1, 1970, on most UNIX boxes you should be able to do the following
$ date -d "2010-12-03 01:00:00" +%s
and get back
1291291200
The manual suggests you'ld type
$ date --date "12/03/2010 01:00:00" +%s
with an American date style, though personally I get confused.
I am not sure how you would achieve that from a windows box but you have access to the Condor master so you could do it on there.