multiprocessing

Share Large, Read-Only Numpy Array Between Multiprocessing Processes

by

I have a 60GB SciPy Array (Matrix) I must share between 5+ multiprocessing Process objects. I’ve seen numpy-sharedmem and read this discussion on the SciPy list. There seem to be two approaches–numpy-sharedmem and using a multiprocessing.RawArray() and mapping NumPy dtypes to ctypes. Now, numpy-sharedmem seems to be the way to go, but I’ve yet to see a good reference example. I don’t need any kind of locks, since the array (actually a matrix) will be read-only. Now, due to its size, I’d like to avoid a copy. It sounds like the correct method is to create the only copy of the array as a sharedmem array, and then pass it to the Process objects? A couple of specific questions:

How to use multiprocessing queue in Python?

by

I’m having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let’s call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such.

Sharing a complex object between processes?

by

I have a fairly complex Python object that I need to share between multiple processes. I launch these processes using multiprocessing.Process. When I share an object with multiprocessing.Queue and multiprocessing.Pipe in it, they are shared just fine. But when I try to share an object with other non-multiprocessing-module objects, it seems like Python forks these objects. Is that true?

Processing single file from multiple processes

by

I have a single big text file in which I want to process each line ( do some operations ) and store them in a database. Since a single simple program is taking too long, I want it to be done via multiple processes or threads.
Each thread/process should read the DIFFERENT data(different lines) from that single file and do some operations on their piece of data(lines) and put them in the database so that in the end, I have whole of the data processed and my database is dumped with the data I need.