Am 11/02/2016 um 15:53 schrieb Andreas Ringlstetter:
Am 11.02.2016 um 14:53 schrieb Wolfgang Mauerer:
Am 11/02/2016 um 14:36 schrieb Andreas Ringlstetter:
Managed and synchronized structures can only be passed during processthe use of a JoinableQueue, which requires inheritance.I have evaluated the standard solution of python itself,
util.py uses pool, and this needs pickling.
These things snarl the job queue of the util.py.
The sync-structures need to be transferred directly when creating
the
processes. I have not found a way to inject them afterwards into
the
running thread pool.
why do standard IPC mechanisms not work? Please explain in a bit
more detail why the result is negative. Scalability problems?
Technical issues? Inconveniences?
to clarify my question: I assume that you have investigated python's
standard solutions to manage shared states, and none of them
worked. So it would be important to learn why they did not work. --WM
multiprocessing, that one works.
The util.py causes pickling issues, which breaks synchronised data
structures.
sure, but what are the pickling issues? Why can't you, for instance,
modify any shared state via a manager object? Or use a queue for
joint modifications? Without knowing what you did in detail, it's a bit
hard to understand what a pickling issue on a synchronised data
structure is...
fork, it's not possible to pickle them and then pass them via a
previously shared data structure. Attempting to do so will result in a
"can only be passed via inheritance" run time error message.
thanks for clarifying. What about creating a generic communication
channel during process creation that can be used to establish shared
states later on?
I think I mixed something up - proxies for managed objects are
pickleable, only the synchronized datastructures directly provided by
multiprocessing (e.g. multiprocessing.JoinableQueue) are not.
I get what you mean, one could add the behaviour which is provided by
JoinableQueue to BatchJobPool, by passing a proxy to BatchJobPool.add()
to the BatchJob. That would require one pipe/queue per worker process
which needs to be monitored by the master and contains BatchJobTuple
instances. Hidden behind a proxy object which mimics the signature of
BatchJobPool.add() for consistent access.
That would bring the functionality partially on par with
multiprocessing.JoinableQueue (even though the implementation differs
fundamentally), enabling running jobs to issue follow up work dynamically.
