Documentation

Wrapper Scripts

The sample plugin scripts use wrapper scripts to simplify the implementation of independentSubmitFcn.m and communicatingSubmitFcn.m. These scripts are not required, however, using them is a good practice to make your code more readable. This table describes these scripts:

independentSubmitFcn

When you submit an independent job to a generic cluster, the independentSubmitFcn.m function executes in the MATLAB client session.

The declaration line of this function must be:

function independentSubmitFcn(cluster,job,environmentProperties)

Each task in a MATLAB independent job corresponds to a single job on your scheduler. The purpose of this function is to submit N jobs to your third-party scheduler, where N is the number of tasks in the independent job. Each of these jobs must:

Configure the worker environment.  This table identifies the five environment variables and values that must be set on the worker MATLAB to run an individual task:

Many schedulers support copying the client environment as part of the submission command. If so, you can set the previous environment variables in the client, so the scheduler can copy them to the worker environment. If not, you must modify your submission command to forward these variables.

Submit scheduler jobs to run MATLAB workers.  Once the five required parameters for a given job and task are defined on a worker, the task is run by calling the MATLAB executable with suitable arguments. The MATLAB executable to call is defined in environmentProperties.MatlabExecutable. The arguments to pass are defined in environmentProperties.MatlabArguments.

Sample script for a SLURM scheduler.  This script shows a basic submit function for a SLURM scheduler with a shared file system. For a more complete example, see Sample Plugin Scripts.

function independentSubmitFcn(cluster,job,environmentProperties)
    % Specify the required environment variables.
    setenv('PARALLEL_SERVER_DECODE_FUNCTION', 'parallel.cluster.generic.independentDecodeFcn');
    setenv('PARALLEL_SERVER_STORAGE_CONSTRUCTOR', environmentProperties.StorageConstructor);
    setenv('PARALLEL_SERVER_STORAGE_LOCATION', environmentProperties.StorageLocation);
    setenv('PARALLEL_SERVER_JOB_LOCATION', environmentProperties.JobLocation);
    
    % Specify the MATLAB executable and arguments to run on the worker.
    % These are used in the independentJobWrapper.sh script.
    setenv('PARALLEL_SERVER_MATLAB_EXE', environmentProperties.MatlabExecutable);
    setenv('PARALLEL_SERVER_MATLAB_ARGS', environmentProperties.MatlabArguments);
    
    for ii = 1:environmentProperties.NumberOfTasks
        % Specify the environment variable required to identify which task to run.
        setenv('PARALLEL_SERVER_TASK_LOCATION', environmentProperties.TaskLocations{ii});
        % Specify the command to submit the job to the SLURM scheduler.
        % SLURM will automatically copy environment variables to workers.
        commandToRun = 'sbatch --ntasks=1 independentJobWrapper.sh';
        [cmdFailed, cmdOut] = system(commandToRun);
    end
end
 

The previous example submits a simple bash script, independentJobWrapper.sh, to the scheduler. The independentJobWrapper.sh script embeds the MATLAB executable and arguments using environment variables:

#!/bin/sh
# PARALLEL_SERVER_MATLAB_EXE - the MATLAB executable to use
# PARALLEL_SERVER_MATLAB_ARGS - the MATLAB args to use
exec "${PARALLEL_SERVER_MATLAB_EXE}" ${PARALLEL_SERVER_MATLAB_ARGS}

communicatingSubmitFcn

When you submit a communicating job to a generic cluster, the communicatingSubmitFcn.m function executes in the MATLAB client session.

The declaration line of this function must be:

function communicatingSubmitFcn(cluster,job,environmentProperties)

The purpose of this function is to submit a single job to your scheduler. This job must:

Configure the worker environment.  This table identifies the four environment variables and values that must be set on the worker MATLAB to run a task of a communicating job:

Many schedulers support copying the client environment as part of the submission command. If so, you can set the previous environment variables in the client, so the scheduler can copy them to the worker environment. If not, you must modify your submission command to forward these variables.

Submit scheduler job to launch MPI process.  After you define the four required parameters for a given job, run your job by launching N worker MATLAB processes using mpiexec. mpiexec is software shipped with the Parallel Computing Toolbox™ that implements the Message Passing Interface (MPI) standard to allow communication between the worker MATLAB processes. For more information about mpiexec, see the MPICH home page.

To run your job, you must submit a job to your scheduler, which executes the following steps. Note that matlabroot refers to the MATLAB installation location on your worker nodes.

Sample script for a SLURM scheduler.  The following script shows a basic submit function for a SLURM scheduler with a shared file system.

The submitted job is contained in a bash script, communicatingJobWrapper.sh. This script implements the relevant steps in Submit scheduler job to launch MPI process for a SLURM scheduler. For a more complete example, see Sample Plugin Scripts.

function communicatingSubmitFcn(cluster,job,environmentProperties)
    % Specify the four required environment variables.
    setenv('PARALLEL_SERVER_DECODE_FUNCTION', 'parallel.cluster.generic.communicatingDecodeFcn');
    setenv('PARALLEL_SERVER_STORAGE_CONSTRUCTOR', environmentProperties.StorageConstructor);
    setenv('PARALLEL_SERVER_STORAGE_LOCATION', environmentProperties.StorageLocation);
    setenv('PARALLEL_SERVER_JOB_LOCATION', environmentProperties.JobLocation);
    
    % Specify the MATLAB executable and arguments to run on the worker.
    % Specify the location of the MATLAB install on the cluster nodes.
    % These are used in the communicatingJobWrapper.sh script.
    setenv('PARALLEL_SERVER_MATLAB_EXE', environmentProperties.MatlabExecutable);
    setenv('PARALLEL_SERVER_MATLAB_ARGS', environmentProperties.MatlabArguments);
    setenv('PARALLEL_SERVER_CMR', cluster.ClusterMatlabRoot);
    
    numberOfTasks = environmentProperties.NumberOfTasks;
    
    % Specify the command to submit a job to the SLURM scheduler which
    % requests as many processes as tasks in the job.
    % SLURM will automatically copy environment variables to workers.
    commandToRun = sprintf('sbatch --ntasks=%d communicatingJobWrapper.sh', numberOfTasks);
    [cmdFailed, cmdOut] = system(commandToRun);
end

getJobStateFcn

When you query the state of a job created with a generic cluster, the getJobStateFcn.m function executes in the MATLAB client session. The declaration line of this function must be:

function state = getJobStateFcn(cluster,job,state)

When using a third-party scheduler, it is possible that the scheduler can have more up-to-date information about your jobs than what is available to the toolbox from the local job storage location. This situation is especially true if using a nonshared file system, where the remote file system could be slow in propagating large data files back to your local data location.

To retrieve that information from the scheduler, add a function called getJobStateFcn.m to the PluginScriptsLocation of your cluster.

The state passed into this function is the state derived from the local job storage. The body of this function can then query the scheduler to determine a more accurate state for the job and return it in place of the stored state. The function you write for this purpose must return a valid value for the state of a job object. Allowed values are ‘pending’, ‘queued’, ‘running’, ‘finished’, or ‘failed’.

For instructions on pairing MATLAB tasks with their corresponding scheduler job ID, see Managing Jobs with Generic Scheduler.

cancelJobFcn

When you cancel a job created with a generic cluster, the cancelJobFcn.m function executes in the MATLAB client session. The declaration line of this function must be:

function OK = cancelJobFcn(cluster,job)

When you cancel a job created using the generic scheduler interface, by default this action affects only the job data in storage. To cancel the corresponding jobs on your scheduler, you must provide instructions on what to do and when to do it to the scheduler. To achieve this, add a function called cancelJobFcn.m to the PluginScriptsLocation of your cluster.

The body of this function can then send a command to the scheduler, for example, to remove the corresponding jobs from the queue. The function must return a logical scalar indicating the success or failure of canceling the jobs on the scheduler.

For instructions on pairing MATLAB tasks with their corresponding scheduler job ID, see Managing Jobs with Generic Scheduler.

cancelTaskFcn

When you cancel a task created with a generic cluster, the cancelTaskFcn.m function executes in the MATLAB client session. The declaration line of this function must be:

function OK = cancelTaskFcn(cluster,task)

When you cancel a task created using the generic scheduler interface, by default, this affects only the task data in storage. To cancel the corresponding job on your scheduler, you must provide instructions on what to do and when to do it to the scheduler. To achieve this, add a function called cancelTaskFcn.m to the PluginScriptsLocation of your cluster.

The body of this function can then send a command to the scheduler, for example, to remove the corresponding job from the scheduler queue. The function must return a logical scalar indicating the success or failure of canceling the job on the scheduler.

For instructions on pairing MATLAB tasks with their corresponding scheduler job ID, see Managing Jobs with Generic Scheduler.

deleteJobFcn

When you delete a job created with a generic cluster, the deleteJobFcn.m function executes in the MATLAB client session. The declaration line of this function must be:

function deleteTaskFcn(cluster,task)

When you delete a job created using the generic scheduler interface, by default, this affects only the job data in storage. To remove the corresponding jobs on your scheduler, you must provide instructions on what to do and when to do it to the scheduler. To achieve this, add a function called deleteJobFcn.m to the PluginScriptsLocation of your cluster.

The body of this function can then send a command to the scheduler, for example, to remove the corresponding jobs from the scheduler queue.

For instructions on pairing MATLAB tasks with their corresponding scheduler job ID, see Managing Jobs with Generic Scheduler.

deleteTaskFcn

When you delete a task created with a generic cluster, the deleteTaskFcn.m function executes in the MATLAB client session. The declaration line of this function must be:

function deleteTaskFcn(cluster,task)

When you delete a task created using the generic scheduler interface, by default, this affects only the task data in storage. To remove the corresponding job on your scheduler, you must provide instructions on what to do and when to do it to the scheduler. To achieve this, add a function called deleteTaskFcn.m to the PluginScriptsLocation of your cluster.

The body of this function can then send a command to the scheduler, for example, to remove the corresponding job from the scheduler queue.

For instructions on pairing MATLAB tasks with their corresponding scheduler job ID, see Managing Jobs with Generic Scheduler.

postConstructFcn

After you create an instance of your cluster in MATLAB, the postConstructFcn.m function executes in the MATLAB client session. For example, the following line of code creates an instance of your cluster and runs the postConstructFcn function associated with the ‘myProfile’ cluster profile:

c = parcluster('myProfile');

The declaration line of the postConstructFcn function must be:

function postConstructFcn(cluster)

If you need to perform custom configuration of your cluster before its use, add a function called postConstructFcn.m to the PluginScriptsLocation of your cluster. The body of this function can contain any extra setup steps you require.

Setting AdditionalProperties from the Cluster Profile Manager

With the modification to your scripts in the previous example, you can add an AdditionalProperties entry to your generic cluster profile to specify a list of nodes to use. This provides a method of documenting customization added to your plugin scripts for anyone you share the cluster profile with.

To add the NodeList property to your cluster profile:

Setting AdditionalProperties from the MATLAB Command Line

With the modification to your scripts in Adding User Customization, you can edit the list of nodes from the MATLAB command line by setting the appropriate property of the cluster object before submitting a job:

c = parcluster;
c.AdditionalProperties.NodeList = 'gpuNodeName';
j = c.batch('myScript'); 

Display the AdditionalProperties object to see all currently defined properties and their values:

>> c.AdditionalProperties
ans = 
  AdditionalProperties with properties:
                 ClusterHost: 'myClusterHost'
                    NodeList: 'gpuNodeName'
    RemoteJobStorageLocation: '/tmp/jobs'

Save Job Scheduler Data

This example shows how to modify the independentSubmitFcn.m function to parse the output of each command submitted to a SLURM scheduler. You can use regular expressions to extract the scheduler job ID for each task and then store it using setJobClusterData.

% Pattern to extract scheduler job ID from SLURM sbatch output
searchPattern = '.*Submitted batch job ([0-9]+).*';
 
jobIDs = cell(numberOfTasks, 1);
for ii = 1:numberOfTasks
    setenv('PARALLEL_SERVER_TASK_LOCATION', environmentProperties.TaskLocations{ii});
    commandToRun = 'sbatch --ntasks=1 independentJobWrapper.sh';
    [cmdFailed, cmdOut] = system(commandToRun);
    jobIDs{ii} = regexp(cmdOut, searchPattern, 'tokens', 'once');
end
 
% set the job IDs on the job cluster data
cluster.setJobClusterData(job, struct('ClusterJobIDs', {jobIDs}));

Retrieve Job Scheduler Data

This example modifies the cancelJobFcn.m to cancel the corresponding jobs on the SLURM scheduler. The example uses getJobClusterData to retrieve job scheduler data.

function OK = cancelJobFcn(cluster, job)

% Get the scheduler information for this job
data = cluster.getJobClusterData(job);
jobIDs = data.ClusterJobIDs;

for ii = 1:length(jobIDs)
    % Tell the SLURM scheduler to cancel the job
    commandToRun = sprintf('scancel ''%s''', jobIDs{ii});
    [cmdFailed, cmdOut] = system(commandToRun);
end

OK = true;