Suppose I have a large in memory numpy array, I have a function func that takes in this giant array as input (together with some other parameters). func with different parameters can be run in parallel. For example:
I have three large lists. First contains bitarrays (module bitarray 0.8.0) and the other two contain arrays of integers.
I have a very large (read only) array of data that I want to be processed by multiple processes in parallel.
I have a 60GB SciPy Array (Matrix) I must share between 5+ multiprocessing Process objects. I’ve seen numpy-sharedmem and read this discussion on the SciPy list. There seem to be two approaches–numpy-sharedmem and using a multiprocessing.RawArray() and mapping NumPy dtypes to ctypes. Now, numpy-sharedmem seems to be the way to go, but I’ve yet to see a good reference example. I don’t need any kind of locks, since the array (actually a matrix) will be read-only. Now, due to its size, I’d like to avoid a copy. It sounds like the correct method is to create the only copy of the array as a sharedmem array, and then pass it to the Process objects? A couple of specific questions: