Open Source Distributed Computing Frameworks
A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray.
License: Unknown
Bagua is a performant and flexible distributed training framework for PyTorch, providing a faster alternative to PyTorch DDP and Horovod. It supports advanced distributed training algorithms such as quantization and decentralization.
License: MIT License
Deep learning framework on top of Spark/Hadoop to distribute data and computations across a HDFS system.
License: Apache License 2.0
A unified deep learning system for big model era, which helps users to efficiently and quickly deploy large AI model training and inference.
License: Apache License 2.0
Distributed parallel processing framework for Pandas and NumPy computations - (Video) .
License: BSD 3-Clause "New" or "Revised" License
A novel evolutionary computation framework for rapid prototyping and testing of ideas. It seeks to make algorithms explicit and data structures transparent. It works in perfect harmony with parallelisation mechanisms such as multiprocessing and SCOOP.
License: GNU Lesser General Public License v3.0
A deep learning optimization library (lightweight PyTorch wrapper) that makes distributed training easy, efficient, and effective.
License: Apache License 2.0
Flexible and powerful tensor operations for readable and reliable code.
License: MIT License
Distributed computing library for modern computer clusters from Uber.
License: Apache License 2.0
A fast, flexible machine learning library written entirely in C++ from the Facebook AI Research and the creators of Torch, TensorFlow, Eigen and Deep Speech.
License: MIT License
Uber’s distributed training framework for TensorFlow, Keras, and PyTorch.
License: Other
LightGBM is a gradient boosting framework that uses tree based learning algorithms.
License: MIT License
Lightweight PyTorch research framework that allows you to easily scale your models to GPUs and TPUs and use all the latest best practices, without the engineering boilerplate - (Video) .
License: Apache License 2.0
Ray is a flexible, high-performance distributed execution framework for machine learning (VIDEO ).
License: Apache License 2.0
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
License: Apache License 2.0
Last Updated: Dec 26, 2023