simulator.shared¶

The large arrays of a chunk result in shared memory, from a worker to the parent.

A worker of a pool returns the result of a chunk through a pipe of the pool, which pickles its arrays in the worker and unpickles them in the parent: 8 results of 214 MB take 1.44 s on linux with 4 workers, 0.11 s written into shared memory. A worker therefore writes an array of at least SHARED_MEMORY_THRESHOLD bytes into a segment of multiprocessing.shared_memory which it creates (share) and returns its SharedArray, the name, the shape and the dtype, instead of the array; the parent maps the segment, removes its name and reads the array from the mapping, without a copy, which is released with the array (receive). A smaller array keeps pickling, a segment costs more than it saves.

The parent owns the segments of a run: their names are a prefix of the run (run_prefix) and the position of the chunk (segment_stem), and a worker creates a segment without the resource tracker (track=False), which would remove it when the worker ends, before the parent read it. The parent removes the name of every segment it reads and, at the end of the run, also after an error, Ctrl-C or a dead worker, every name of the run which may exist (unlink), so no segment outlives its run. A run in the calling process never uses segments.

Shared memory is used on POSIX only (linux, macOS): on windows a named mapping disappears when its last handle closes, so a worker cannot hand one over after it returned, and the arrays keep pickling there. An array keeps pickling as well when its segment cannot be created or its memory reserved, e.g. a /dev/shm too small for it.

SharedArray(name, shape, dtype)

supported()

run_prefix()

segment_stem(prefix, chunk)

segment_names(stem)

share(array, name)

receive(shared)

unlink(names)