Worker for Python Dask Parallel Computation Framework
Dask is a python library to achieve advanced parallelism for analytics and this app converts a device into a dask worker.
If deployed on many devices, all devices act as a dask worker and the compulational load is distributed among them.
Parameters
| Parameter | Meaning | Default |
|---|---|---|
| NUM_WORKERS | The number of processes to use in this worker | 1 |
| NUM_THREADS | The number of threads to use in this worker | 1 |
| MEM_LIMIT | The Memory Limit for one Process | 2GB |
| SCHEDULER_ADDRESS | The scheduler instance the worker should connect to and get his work from | localhost:8786 |
Data volumes
Every app running on a device has a private folder /data that can be used to persist data for the case the device resets or the app is restarted.
The folder /shared is accessible by all apps running on the same device and can be used to share data between apps on one device.
License
Copryright © 2019 Dask core developers. New-BSD Licensed.
Modified Copryright © 2020 Record Evolution GmbH
- Modification: Binaries wrapped as a microservice
See license file on the source code