Parallelism

Parallelism support, including asyncio, threading, multiprocessing, etc.

asyncio utilities

---

source

async_wrap

 async_wrap (func:Callable)

Wrap a synchronous function func into an asynchronous function.

Timeout

Based on asyncio

---

source

seq_consume_preset_queue_w_each_timeout

 seq_consume_preset_queue_w_each_timeout (consumer:Callable, idxed_kwargs_
                                          queue:_queue.SimpleQueue|multipr
                                          ocessing.queues.SimpleQueue|list
                                          , timeout:int=5,
                                          pbar:tqdm.std.tqdm=None)

Sequentially run computation-intensive consumer along a preset (no more input) indexed task idxed_kwargs_queue with each task having timeout. queue.SimpleQueue is not thread-safe, don’t run multiple consumers in the same process. However, multiprocessing.SimpleQueue is process-safe based on pipe, you can run multiple consumers in the same number of processes. NOTE: co-routine would get stuck with some consumer like creating and running new interactive IPython shells even with timeout set.

	Type	Default	Details
consumer	Callable		An Awaitable coroutine function.
idxed_kwargs_queue	_queue.SimpleQueue | multiprocessing.queues.SimpleQueue | list		Indexed kwargs queue, comprising elements like (idx, kwargs). For the weird type hint, refer to https://github.com/python/cpython/issues/99509
timeout	int	5	Timeout for each task.
pbar	tqdm	None	Progress bar to update. None means no progress bar.
Returns	list		Indexed return values, comprising elements like (idx, retval).