parallel.pool.DataQueue

Class that enables sending and listening for data between client and workers

Description

A data queue enables sending data or messages from workers back to the client in a parallel pool while a computation is carried out. For example, you can get intermediate values and an indication of the progress of the computation.

To send data from a parallel pool worker back to the client, first construct a DataQueue in the client. Pass this DataQueue into a parfor-loop or other parallel language construct, such as spmd. From the workers, call send to send data back to the client. At the client, register a function to be called each time data is received by using afterEach.

  • You can call send from the process that calls the constructor, if required.

  • You can construct the queue on the workers and send it back to the client to enable communication in the reverse direction. However, you cannot send a queue from one worker to another. Use spmd, labSend, or labReceive instead.

  • Unlike all other handle objects, DataQueue instances do remain connected when they are sent to workers.

Creation

Description

example

q = parallel.pool.DataQueue takes no arguments and returns an object that can be used to send or listen for messages (or data) from different workers. You call the constructor only in the process where you want to receive the data. In the usual workflow, the workers should not be calling the constructor, but should be handed an existing DataQueue instance instead.

Properties

expand all

Read-only property that indicates how many items of data are waiting to be removed from the queue. The value is 0 or a positive integer on the process that created the DataQueue instance. The value is 0 on all other processes.

Object Functions

A parallel.pool.DataQueue object has the following methods.

afterEachDefine a function to call when new data is received on a DataQueue
sendSend data from worker to client using a data queue

Examples

collapse all

When you send a message to a DataQueue object, the message waits in the queue until it is processed by a listener. Each message adds 1 to the queue length. In this example, you use the QueueLength property to find the length of a DataQueue object.

When a MATLAB process creates a DataQueue object, any messages that are sent to the queue are held in the memory of that process. Therefore, the QueueLength property on all other processes is 0. In this example, you create a DataQueue object on the client, and send data from a worker.

First, create a parallel pool with one worker.

parpool(1);
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 1).

Then, create a DataQueue.

q = parallel.pool.DataQueue
q = 
  DataQueue with properties:

    QueueLength: 0

A newly created DataQueue has an empty queue. You can use parfor to find q.QueueLength on the worker. Find the queue length on the client, and the queue length on the worker.

fprintf('On the client: %i\n', q.QueueLength)
On the client: 0
parfor i = 1
    fprintf('On the worker: %i\n', q.QueueLength)
end
On the worker: 0

As the queue is empty, the QueueLength is 0 for both the client and the worker. Next, send a message to the queue from the worker. Then, use the QueueLength property to find the length of the queue.

% Send a message first
parfor i = 1
    send(q, 'A message');
end

% Find the length
fprintf('On the client: %i\n', q.QueueLength)
On the client: 1
parfor i = 1
    fprintf('On the worker: %i\n', q.QueueLength)
end
On the worker: 0

The QueueLength property is 1 on the client, and 0 on the worker. Create a listener to process the queue by immediately displaying the data.

el = afterEach(q, @disp);

Wait until the queue is empty, then delete the listener.

while q.QueueLength > 0
    pause(0.1);
end
delete(el);

Use the QueueLength property to find the length of the queue.

fprintf('On the client: %i\n', q.QueueLength)
On the client: 0

QueueLength is 0 because the queue processing is complete.

Construct a DataQueue, and call afterEach.

q = parallel.pool.DataQueue;
afterEach(q, @disp);
Start a parfor-loop, and send a message. The pending message is passed to the afterEach function, in this example @disp.

parfor i = 1:3
    send(q, i); 
end;
     1

     2

     3

For more details on listening for data using a DataQueue, see afterEach.

Create a DataQueue, and use afterEach to specify the function to execute each time the queue receives data. This example calls a subfunction that updates the wait bar.

Create a parfor-loop to carry out a computationally demanding task in MATLAB®. Use send to send some dummy data on each iteration of the parfor-loop. When the queue receives the data, afterEach calls nUpdateWaitbar in the client MATLAB, and you can observe the wait bar progress.

function a = parforWaitbar

D = parallel.pool.DataQueue;
h = waitbar(0, 'Please wait ...');
afterEach(D, @nUpdateWaitbar);

N = 200;
p = 1;

parfor i = 1:N
    a(i) = max(abs(eig(rand(400))));
    send(D, i);
end

    function nUpdateWaitbar(~)
        waitbar(p/N, h);
        p = p + 1;
    end
end

If you call afterEach and there are items on the queue waiting to be dispatched, these items are immediately dispatched to the function handle specified by afterEach. Call afterEach before sending data to the queue, to ensure that on send, the function handle @disp is called.

Construct a DataQueue and call afterEach.

q = parallel.pool.DataQueue;
afterEach(q, @disp);
If you then send messages to the queue, each message is passed to the function handle specified by afterEach immediately.

parfor i = 1:3
    send(q, i); 
end
send(q, 0);
     1

     3

     2

     0

If you send the data to the queue and then call afterEach, each of the pending messages are passed to the function handle specified by afterEach.

q = parallel.pool.DataQueue;
parfor i = 1:3
    send(q, i); 
end
afterEach(q, @disp);
       3

       1

       2
Introduced in R2017a