Open Source Model Serving & Monitoring
Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.
License: Other
BentoML is an open source framework for high performance ML model serving.
License: Apache License 2.0
Cortex is an open source platform for deploying machine learning models—trained with any framework—as production web services. No DevOps required.
License: Apache License 2.0
Deepchecks is an open source package for comprehensively validating your machine learning models and data with minimal effort during development, deployment or in production.
License: Other
Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.
License: Other
Cloud-native machine learning model server.
License: Apache License 2.0
Quality Assurance for AI models. Open-source platform to help organizations increase the efficiency of their AI development workflow, eliminate risks of AI biases and ensure robust, reliable & ethical AI models.
License: Other
Helicone is an observability platform for LLMs.
License: Apache License 2.0
Open source model management cluster for deploying, serving and monitoring machine learning models and ad-hoc algorithms with a FaaS architecture.
License: Apache License 2.0
Cloud native search framework that supports to use deep learning/state of the art AI models for search.
License: Apache License 2.0
Serverless framework to deploy and monitor machine learning models in Kubernetes - (Video) .
License: Apache License 2.0
LocalAI is a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing.
License: MIT License
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more.
License: Apache License 2.0
a lightweight, open-source Python tool to get “bolt-on” observability in ML pipelines.
License: Apache License 2.0
A model server for Apache MXNet from Amazon Web Services that is able to run MXNet models as well as Gluon models (Amazon’s SageMaker runs a custom version of MMS under the hood).
License: Apache License 2.0
An open source library to estimate post-deployment model performance (without access to targets). Capable of fully capturing the impact of data drift on performance.
License: Apache License 2.0
A rust-powered and multi-stage pipelined model server which offers dynamic batching and more. Super easy to implement and deploy as micro-services.
License: Apache License 2.0
A high-performance “serverless” framework focused on data, I/O, and compute intensive workloads. It is well integrated with popular data science tools, such as Jupyter and Kubeflow; supports a variety of data and streaming sources; and supports execution over CPUs and GPUs.
License: Apache License 2.0
REST web service for the true real-time scoring (< 1 ms) of Scikit-Learn, R and Apache Spark models.
License: GNU Affero General Public License v3.0
Creates HTML profiling reports from pandas DataFrame objects. It extends the pandas DataFrame with df.profile_report() for quick data analysis.
License: MIT License
Phoenix is an open source ML observability in a notebook to validate, monitor and fine tune your generative LLM, CV and tabular models.
License: Other
An open source Machine Learning Server built on top of a state-of-the-art open source stack for developers and data scientists to create predictive engines for any machine learning task.
License: Apache License 2.0
A Redis module for serving tensors and executing deep learning models. Expect changes in the API and internals.
License: Other
Open source platform for deploying and monitoring machine learning models in kubernetes - (Video) .
License: Apache License 2.0
skops is a Python library helping you share your scikit-learn based models and put them in production.
License: MIT License
Open source SDK that provides a unified interface to multiple MLOps projects that enable data scientists to deploy and productionise machine learning systems.
License: Apache License 2.0
High-performant framework to serve Tensorflow models via grpc protocol able to handle 100k requests per second per core.
License: Apache License 2.0
TorchServe is a flexible and easy to use tool for serving PyTorch models.
License: Apache License 2.0
Transformer-deploy is an efficient, scalable and enterprise-grade CPU/GPU inference server for Hugging Face transformer models.
License: Unknown
Triton is a high performance open source serving software to deploy AI models from any framework on GPU & CPU while maximizing utilization.
License: BSD 3-Clause "New" or "Revised" License
UnionML is an open source MLOps framework that aims to reduce the boilerplate and friction that comes with building models and deploying them to production.
License: Apache License 2.0
vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs
License: Apache License 2.0
Lightweight solution for profiling and monitoring your ML data pipeline end-to-end
License: Apache License 2.0
Last Updated: Dec 26, 2023