Task interface

The task interface to the cluster presents the engines as a fault-tolerant, dynamic load balancing of Workers. In contrast to the direct interface, the task interface does not have direct access to individual engines. As the IPython scheduler assigns the workers, the interface becomes simpler and more powerful at the same time.

The best part, however, is that both interfaces can be used at the same time to leverage their respective strengths. If calculations do not depend on previous results, the task interface is ideal:

Create an LoadBalancedView instance

import ipyparallel as ipp

rc = ipp.Client()

rc = ipp.Client(url_file='/Users/veit/.ipython/profile_mpi/security/ipcontroller-client.json')

rc = ipp.Client(profile='mpi')

lview = rc.load_balanced_view()

load_balanced_view is the default view.

See also:

• Views

Fast and easy parallelism

map()-LoadBalancedView

lview.block = True
serial_result = map(lambda x:x**10, range(32))
parallel_result = lview.map(lambda x:x**10, range(32))
serial_result==parallel_result

True

@lview.parallel() decorator

@lview.parallel()
def f(x):
    return 10.0*x**4

f.map(range(32))

[0.0,10.0,160.0,…]

Dependencies

Note:

Please note that the pure ZeroMQ scheduler does not support any dependencies.

Function dependencies

UnmetDependency

@ipp.require decorator

@ipp.depend decorator

dependent object

Dependency

client.block=False

ar = lview.apply(f, args, kwargs)
ar2 = lview.apply(f2)

with lview.temp_flags(after=[ar,ar2]):
    ar3 = lview.apply(f3)

with lview.temp_flags(follow=[ar], timeout=2.5)
    ar4 = lview.apply(f3)

See also: Some parallel workloads can be described as Directed acyclic graph (DAG). In DAG Dependencies we describe using an example how NetworkX is used to represent the task dependencies as DAG.

ImpossibleDependency

retries and resubmit

Schedulers

ipcontroller --scheme=lru

Scheme	Description
lru	Least Recently Used: Always assigns the workers to the last used engine. Similar to round robin, however, it does not take into account the runtime of each individual task.
plainrandom	Plain Random: Randomly selects the engine to be run.
twobin	Two-Bin Random: Requires numpy. Randomly select two engines and use lru. This is often better than the purely random distribution, but requires more computational effort.
leastload	Least Load: Standard scheme that the engine always assigns tasks with the fewest outstanding tasks.
weighted	Weighted Two-Bin Random: Weighted Two-Bin Random scheme.