Module threadpool
Easy to use object-oriented thread pool framework.
A thread pool is a class that maintains a pool of worker threads to
perform time consuming operations in parallel. It assigns jobs to the
threads by putting them in a work request queue, where they are picked up
by the next available thread. This then performs the requested operation
in the background and puts the results in a another queue.
The thread pool class can then collect the results from all threads
from this queue as soon as they become available or after all threads
have finished their work. It's also possible, to define callbacks to
handle each result as it comes in.
The basic concept and some code was taken from the book "Python
in a Nutshell" by Alex Martelli, copyright 2003, ISBN 0-596-00188-6,
from section 14.5 "Threaded Program Architecture". I wrapped
the main program logic in the ThreadPool class, added the WorkRequest
class and the callback system and tweaked the code here and there.
Basic usage:
>>> main = TreadPool(poolsize)
>>> requests = makeRequests(some_callable, list_of_args, callback)
>>> [main.putRequests(req) for req in requests]
>>> main.wait()
See the end of the module code for a brief, annotated usage
example.
|
NoResultsPending
All work requests have been processed.
|
|
NoWorkersAvailable
No worker threads available to process remaining requests.
|
|
WorkerThread
Background thread connected to the requests/results queues.
|
|
WorkRequest
A request to execute a callable for putting in the request queue
later.
|
|
ThreadPool
A thread pool, distributing work requests and collecting
results.
|
|
makeRequests(callable,
args_list,
callback=None)
Convenience function for building several work requests for the
same callable with different arguments for each call.
|
|
do_something(data)
|
|
print_result(request,
result)
|
makeRequests(callable,
args_list,
callback=None)
| |
Convenience function for building several work requests for the same
callable with different arguments for each call.
args_list contains the parameters for each invocation of callable.
Each item in 'argslist' should be either a 2-item tuple of the list of
positional arguments and a dictionary of keyword arguments or a single,
non-tuple argument.
callback is called when the results arrive in the result queue.
-
|
print_result(request,
result)
| |
None
-
|
data
None
-
- Value:
[random.randint(1, 10) for i in range(20)]
|
|
requests
f00 = []
for i in xrange(1000):
print i
f00.append( ThreadPool(1) )
-
- Value:
makeRequests(do_something, data, print_result)
|
|