18 Distributed Computing and Machine Learning Frameworks: Empowering Scalable and Efficient Data Processing and Model Training

Open Source Distributed Computing Frameworks

Analytics Zoo
A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray.
License: Unknown
GitHub
Website: Unknown
Bagua
Bagua is a performant and flexible distributed training framework for PyTorch, providing a faster alternative to PyTorch DDP and Horovod. It supports advanced distributed training algorithms such as quantization and decentralization.
License: MIT License
GitHub
Website: https://tutorials.baguasys.com/
BigDL
Deep learning framework on top of Spark/Hadoop to distribute data and computations across a HDFS system.
License: Apache License 2.0
GitHub
Website: https://bigdl.readthedocs.io
Colossal-AI
A unified deep learning system for big model era, which helps users to efficiently and quickly deploy large AI model training and inference.
License: Apache License 2.0
GitHub
Website: https://www.colossalai.org
Dask
Distributed parallel processing framework for Pandas and NumPy computations - (Video) .
License: BSD 3-Clause "New" or "Revised" License
GitHub
Website: https://dask.org
DEAP
A novel evolutionary computation framework for rapid prototyping and testing of ideas. It seeks to make algorithms explicit and data structures transparent. It works in perfect harmony with parallelisation mechanisms such as multiprocessing and SCOOP.
License: GNU Lesser General Public License v3.0
GitHub
Website: http://deap.readthedocs.org/
DeepSpeed
A deep learning optimization library (lightweight PyTorch wrapper) that makes distributed training easy, efficient, and effective.
License: Apache License 2.0
GitHub
Website: https://www.deepspeed.ai/
einops
Flexible and powerful tensor operations for readable and reliable code.
License: MIT License
GitHub
Website: https://einops.rocks
Fiber
Distributed computing library for modern computer clusters from Uber.
License: Apache License 2.0
GitHub
Website: https://uber.github.io/fiber/
Flashlight
A fast, flexible machine learning library written entirely in C++ from the Facebook AI Research and the creators of Torch, TensorFlow, Eigen and Deep Speech.
License: MIT License
GitHub
Website: https://fl.readthedocs.io/en/latest/
Hivemind
Decentralized deep learning in PyTorch.
License: MIT License
GitHub
Horovod
Uber’s distributed training framework for TensorFlow, Keras, and PyTorch.
License: Other
GitHub
Website: http://horovod.ai
LightGBM
LightGBM is a gradient boosting framework that uses tree based learning algorithms.
License: MIT License
GitHub
Website: https://lightgbm.readthedocs.io/en/latest/
NumPyWren
Scientific computing framework build on top of pywren to enable numpy-like distributed computations.
License: No License
GitHub
PyWren
Answer the question of the “cloud button” for python function execution. It’s a framework that abstracts AWS Lambda to enable data scientists to execute any Python function - (Video) .
License: Apache License 2.0
GitHub
PyTorch Lightning
Lightweight PyTorch research framework that allows you to easily scale your models to GPUs and TPUs and use all the latest best practices, without the engineering boilerplate - (Video) .
License: Apache License 2.0
GitHub
Website: https://lightning.ai
Ray
Ray is a flexible, high-performance distributed execution framework for machine learning (VIDEO ).
License: Apache License 2.0
GitHub
Website: https://ray.io
TensorFlowOnSpark
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
License: Apache License 2.0
GitHub