partition

Partition a minibatchqueue

collapse all in page

    Syntax

    submbq = partition(mbq,numParts,indx)

    Description

    example

    submbq = partition(mbq,numParts,indx) partitions minibatchqueue mbq into numParts parts and returns the partition corresponding to the index indx. The properties of submbq are the same as the properties of mbq.

    The output minibatchqueue only has access to the partition of data it is given when it is created. Using reset with submbq resets the minibatchqueue to the start of the data partition. Using shuffle with submbq shuffles only the partitioned data. If you want to shuffle the data across multiple partitions, you must shuffle the original minibatchqueue and then re-partition.

    Examples

    collapse all

    Use the partition function to divide a minibatchqueue into three parts.

    Create a minibatchqueue from a datastore.

    ds = digitDatastore;
mbq = minibatchqueue(ds)
    mbq = 
minibatchqueue with 1 output and properties:

   Mini-batch creation:
           MiniBatchSize: 128
        PartialMiniBatch: 'return'
            MiniBatchFcn: 'collate'
    DispatchInBackground: 0

   Outputs:
              OutputCast: {'single'}
         OutputAsDlarray: 1
         MiniBatchFormat: {''}
       OutputEnvironment: {'auto'}

    Partition the minibatchqueue into three parts and return the first partition.

    sub1 = partition(mbq,3,1)
    sub1 = 
minibatchqueue with 1 output and properties:

   Mini-batch creation:
           MiniBatchSize: 128
        PartialMiniBatch: 'return'
            MiniBatchFcn: 'collate'
    DispatchInBackground: 0

   Outputs:
              OutputCast: {'single'}
         OutputAsDlarray: 1
         MiniBatchFormat: {''}
       OutputEnvironment: {'auto'}

    sub1 contains approximately the first third of the data in mbq.

    Use the partition function to divide a minibatchqueue into three parts.

    Create a minibatchqueue from a datastore.

    ds = digitDatastore;
mbq = minibatchqueue(ds)
    mbq = 
minibatchqueue with 1 output and properties:

   Mini-batch creation:
           MiniBatchSize: 128
        PartialMiniBatch: 'return'
            MiniBatchFcn: 'collate'
    DispatchInBackground: 0

   Outputs:
              OutputCast: {'single'}
         OutputAsDlarray: 1
         MiniBatchFormat: {''}
       OutputEnvironment: {'auto'}

    Partition the minibatchqueue into three parts on three workers in a parallel pool. Iterate over the data on each worker.

    numWorkers = 3;
p = parpool('local',numWorkers);
parfor i=1:3
    submbq = partition(mbq,3,i);
    while hasdata(submbq)
        data = next(submbq);
    end
end

    Each worker has access to a subset of the data in the original minibatchqueue.

    Input Arguments

    collapse all

    Queue of mini-batches, specified as a minibatchqueue object.

    Number of partitions, specified as a numeric scalar.

    Partition index, specified as a numeric scalar.

    Output Arguments

    collapse all

    Output minibatchqueue. submbq contains subset of the data in mbq The properties of submbq are the same as the properties of mbq.

    See Also

    | | |

    Topics

    Introduced in R2020b

    Deep Learning Toolbox Documentation

    Support