Parallel Evaluation

Sending Commands to Remote Kernels

Recall that connections to remote kernels, as opened by (RefLink[LaunchKernels, paclet:ref/LaunchKernels]), are represented as kernel objects. See Connection Methods for details. The commands in this section take parallel kernels as arguments and use them to carry out computations.

Low-Level Parallel Evaluation

Sending and receiving commands to and from remote kernels.

(RefLink[ParallelEvaluate, paclet:ref/ParallelEvaluate]) has the attribute (RefLink[HoldFirst, paclet:ref/HoldFirst]).

You cannot use (RefLink[ParallelEvaluate, paclet:ref/ParallelEvaluate]) while a concurrent computation involving (RefLink[ParallelSubmit, paclet:ref/ParallelSubmit]) or (RefLink[WaitAll, paclet:ref/WaitAll]) is in progress. See "Concurrency: Managing Parallel Processes" for details.

Values of Variables

Values of variables defined on the local master kernel are usually not available to remote kernels. If a command you send for evaluation refers to a variable, it usually will not work as expected.

In these examples, a command is always sent to a single kernel.

mykernel=First[Kernels[]]

The following program will return (RefLink[False, paclet:ref/False]) because the symbol a will most likely not have any value at all on the remote kernel.

a=2;
ParallelEvaluate[a===2,mykernel]

A convenient way to insert variable values into unevaluated commands is to use (RefLink[With, paclet:ref/With]), as demonstrated in the following command. The symbol a is replaced by 2, then the expression 2 === 2 is sent to the remote kernel, where it evaluates to (RefLink[True, paclet:ref/True]).

With[{a=2},ParallelEvaluate[a===2,mykernel] ]

If you need variable values and definitions carried over to the remote kernels, use (RefLink[DistributeDefinitions, paclet:ref/DistributeDefinitions]) or shared variables.

Iterators, such as (RefLink[Table, paclet:ref/Table]) and (RefLink[Do, paclet:ref/Do]), work in the same way with respect to the iterator variable. Therefore, a statement like the following will not do the expected thing.

The variable i will not have a value on the remote kernel.

Table[ParallelEvaluate[IntegerQ[i],mykernel],{i,1,10}]

You can use the following command to accomplish the intended iteration on the remote kernel. This substitutes the value of i into the argument of (RefLink[ParallelEvaluate, paclet:ref/ParallelEvaluate]).

Table[With[{i=i},ParallelEvaluate[IntegerQ[i],mykernel]],{i,1,10}]

Pattern variables, constants, and pure function variables will work as expected on the remote kernel. Each of the following three examples will produce the expected result.

Formal parameters of pure functions are inserted into the body before the expression is sent to the parallel kernels.

Function[i,ParallelEvaluate[IntegerQ[i]]][5]

Pattern variables are also inserted on the right side of the definition.

f[i_]:=ParallelEvaluate[IntegerQ[i]]

f[5]

Constants are also inserted.

With[{i=5},ParallelEvaluate[IntegerQ[i]]]

Parallel Evaluation of Expressions

({{(RefLink[ParallelCombine, paclet:ref/ParallelCombine])[f,h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])],comb], evaluates f[h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])]] in parallel by distributing chunks f[h[(Subscript[e, i]),(Subscript[e, i+1]),…,(Subscript[e, i+k])]] to all kernels and combining the results with comb[]}, {(RefLink[ParallelCombine, paclet:ref/ParallelCombine])[f,h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])]], the default combiner comb is h, if h has attribute (RefLink[Flat, paclet:ref/Flat]), and (RefLink[Join, paclet:ref/Join]) otherwise}})Basic parallel dispatch of evaluations.

(RefLink[ParallelCombine, paclet:ref/ParallelCombine])[f,h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])],comb] breaks h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])] into pieces h[(Subscript[e, i]),(Subscript[e, i+1]),…,(Subscript[e, i+k])], evaluates f[h[(Subscript[e, i]),(Subscript[e, i+1]),…,(Subscript[e, i+k])]] in parallel, then combines the results (Subscript[r, i]) using comb[(Subscript[r, 1]),(Subscript[r, 2]),…,(Subscript[r, m])]. (RefLink[ParallelCombine, paclet:ref/ParallelCombine]) has the attribute (RefLink[HoldFirst, paclet:ref/HoldFirst]), so that h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])] is not evaluated on the master kernel before the parallelization.

ParallelCombine

(RefLink[ParallelCombine, paclet:ref/ParallelCombine]) is a general and powerful command with default values for its arguments that are suitable for evaluating elements of containers, such as lists and associative functions.

Evaluating List-like Containers

If the result of applying the function f to a list is again a list, (RefLink[ParallelCombine, paclet:ref/ParallelCombine])[f,h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])],comb] simply applies f to pieces of the input list and joins the partial results together.

ParallelCombine[Prime,{1,2,3,4,5,6,7,8,9}]

The result is the same as that of (RefLink[Prime, paclet:ref/Prime])[{1,2,3,4,5,6,7,8,9}], but the computation is done in parallel.

If the function is (RefLink[Identity, paclet:ref/Identity]), (RefLink[ParallelCombine, paclet:ref/ParallelCombine]) simply evaluates the elements (Subscript[e, i]) in parallel.

ParallelCombine[Identity,{1+2,2+3,3+4,4+5,5+6}]

If the result of applying the function f to a list is not a list, a custom combiner has to be chosen.

The function (RefLink[Function, paclet:ref/Function])[li,(RefLink[Count, paclet:ref/Count])[li,_?(RefLink[OddQ, paclet:ref/OddQ])]] counts the number of odd elements in a list. To find the total number of odd elements, add the partial results together.

ParallelCombine[Function[li,Count[li,_?OddQ]],{1,2,3,4,5,6,7,8,9},Plus]

Evaluating Associative Operations

If the operation h in h[(Subscript[e, 1]),(Subscript[e, 2]),…,(Subscript[e, n])] is associative (has attribute (RefLink[Flat, paclet:ref/Flat])), the identity

holds; with the default combiner being h itself, the operation is parallelized in a natural way. Here all numbers are added in parallel.

ParallelCombine[Identity, Unevaluated[1+2+3+4+5+6+7+8+9]]

ParallelCombine[GCD, Unevaluated[GCD[4,6,8,10]]]

Parallel Mapping and Iterators

The commands in this section are fundamental to parallel programming in the Wolfram Language.

Parallel evaluation, mapping, and tables.

(RefLink[ParallelMap, paclet:ref/ParallelMap])[f,h[(Subscript[e, 1]),(Subscript[e, 2]),…]] is a parallel version of f/@h[(Subscript[e, 1]),(Subscript[e, 2]),…] evaluating the individual f[(Subscript[e, i])] in parallel rather than sequentially.

Side Effects

Unless you use shared variables, the parallel evaluations performed are completely independent and cannot influence each other. Furthermore, any side effects, such as assignments to variables, that happen as part of evaluations will be lost. The only effect of a parallel evaluation is that its result is returned at the end.

Examples

First, start several remote kernels.

ParallelEvaluate[$ProcessID]

The sine function is applied to the given arguments. Each computation takes place on a remote kernel.

ParallelMap[Sin,{0,π,1.0}]

This particular computation is almost certainly too trivial to benefit from parallel evaluation. The overhead required to send the expressions (RefLink[Sin, paclet:ref/Sin])[0], (RefLink[Sin, paclet:ref/Sin])[π], and so on to the remote kernels and to collect the results will be larger than the gain obtained from parallelization.

Factoring integers of the form 111…1 takes more time, so this computation can benefit from parallelization.

ParallelMap[FactorInteger,((10^Range[20,30]-1)/9)]

Alternatively, you can use (RefLink[ParallelTable, paclet:ref/ParallelTable]). Here a list of the number of factors in 11…1/i  is generated.

ParallelTable[{i,Plus@@(#[[2]]&)/@FactorInteger[(((10^i)-1)/9)]},{i,20,30}]//TableForm

Automatic Distribution of Definitions

Parallel commands such as (RefLink[ParallelTable, paclet:ref/ParallelTable]) will automatically distribute the values and functions needed, using effectively (RefLink[DistributeDefinitions, paclet:ref/DistributeDefinitions]).

For this parallel table, the function f and the iterator bound n will evaluate on the subkernels, so their definitions need to be distributed to make it work.

f[x_,y_]:=x^y
{m,n}={4,2};

ParallelTable[f[i,j],{i,m},{j,n}]

This automatic distribution happens for any functions and variables you define interactively, within the same notebook (technically, for all symbols in the default context). Definitions from other contexts, such as functions from packages, are not distributed automatically.

Lower-level functions, such as (RefLink[ParallelEvaluate, paclet:ref/ParallelEvaluate]), do not perform any automatic distribution of values.

The symbol kid will return the value of (RefLink[$KernelID, paclet:ref/$KernelID]) when evaluated.

kid:=$KernelID

On the remote kernels, (RefLink[$KernelID, paclet:ref/$KernelID]) is nonzero, but the symbol is not evaluated there, only later on the master kernel, where the value is zero.

ParallelEvaluate[kid]

After the definition of kid is distributed, it is evaluated on the remote kernels.

DistributeDefinitions[kid];

ParallelEvaluate[kid]

Automatic Parallelization

(RefLink[Parallelize, paclet:ref/Parallelize])[cmd[list,arguments…]] recognizes if cmd is a Wolfram Language function that operates on a list or other long expression in a way that can be easily parallelized and performs the parallelization automatically.

Parallelize[Count[{1,2,3,4,5,6,7},_?PrimeQ]]

Parallelize[Map[f,{a,b,c,d,e,f}]]

Parallelize[{a,b,c,d}.(({{w1, w2}, {x1, x2}, {y1, y2}, {z1, z2}}))]

Not all uses of these commands can be parallelized. A message is generated and the evaluation is performed sequentially on the master kernel if necessary.

Parallelize[Apply[f,g[a,b,c,d]]]