Running parallel simulations¶
Where the underlying simulator supports distributed simulations, in which the computations are spread over multiple processors using MPI (this is the case for NEURON and NEST), PyNN also supports this. To run a distributed simulation on eight nodes, the command will be something like:
$ mpirun -np 8 -machinefile ~/mpi_hosts python myscript.py
Depending on the implementation of MPI you have, mpirun
could be replaced by
mpiexec
or another command, and the options may also be somewhat different.
For NEURON only, you can also run distributed simulations using nrniv
instead of the python
executable:
$ mpirun -np 8 -machinefile ~/mpi_hosts nrniv -python -mpi myscript.py
Additional requirements¶
First, make sure you have compiled the simulators you wish to use with MPI enabled.
There is usually a configure flag called something like “--with-mpi
” to do this,
but see the installation documentation for each simulator for details.
If you wish to use the default “gather” feature (see below), which
automatically gathers output data from all the nodes to the master node (the one
on which you launched the simulation), you will need to install the mpi4py
module (see http://mpi4py.scipy.org/ for downloads and documentation).
Installation is usually very straightforward, although, if you have more than
one MPI implementation installed on your system (e.g. OpenMPI and MPICH2), you
must be sure to build mpi4py
with the same MPI implementation that you used to
build the simulator.
Code modifications¶
In most cases, no modifications to your code should be necessary to run in parallel. PyNN, and the simulator, take care of distributing the computations between nodes. Furthermore, the default settings should give results that are independent of the number of processors used, even when using random numbers.
Gathering data to the master node¶
The various methods of the Population
and Assembly
classes that deal with
accessing recorded data or writing it to disk, such as get_data()
, write_data()
,
etc., have an optional argument gather, which is True
by default.
If gather is True
, then data generated on other nodes is sent to the master node.
This means that, for example, write_data()
will create only a single file,
on the filesystem of the master node. If gather is False
, each node will
write a file on its local filesystem. This option is often desirable if you wish
to do distributed post-processing of the data. (Don’t worry, by the way, if you
are using a shared filesystem such as NFS. If gather is False
then the MPI
rank is appended to the filename, so there is no chance of conflict between the
different nodes).
Random number generators¶
In general, we expect that our results should not depend on the number of processors used to produce them. If our simulations use random numbers in setting-up or running the network, this means that each object that uses random numbers should receive the same sequence independent of which node it is on or how many nodes there are. PyNN achieves this by ensuring the generator seed is the same on all nodes, and then generating as many random numbers as would be used in the single-processor case and throwing away those that are not needed.
This obviously has a potential impact on performance, and so it is possible to turn it off by passing parallel_safe=False as argument when creating the random number generator, e.g.:
>>> from pyNN.random import NumpyRNG
>>> rng = NumpyRNG(seed=249856, parallel_safe=False)
Now, PyNN will ensure the seed is different on each node, and will generate only as many numbers as are actually needed on each node.
Note that the above applies only to the random number generators provided by the
pyNN.random
module, not to the native RNGs used internally by each simulator.
This means that, for example, you should prefer SpikeSourceArray
(for which
you can generate Poisson spike times using a parallel-safe RNG) to
SpikeSourcePoisson
, which uses the simulator’s internal RNG, if you care about
being independent of the number of processors.