Open Source Model Serving & Monitoring
-
Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.
License: Other
-
BentoML is an open source framework for high performance ML model serving.
License: Apache License 2.0
-
Cortex is an open source platform for deploying machine learning models—trained with any framework—as production web services. No DevOps required.
License: Apache License 2.0
-
Deepchecks is an open source package for comprehensively validating your machine learning models and data with minimal effort during development, deployment or in production.
License: Other
-
Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.
License: Other
-
-
Cloud-native machine learning model server.
License: Apache License 2.0
-
Quality Assurance for AI models. Open-source platform to help organizations increase the efficiency of their AI development workflow, eliminate risks of AI biases and ensure robust, reliable & ethical AI models.
License: Other
-
Helicone is an observability platform for LLMs.
License: Apache License 2.0
-
Open source model management cluster for deploying, serving and monitoring machine learning models and ad-hoc algorithms with a FaaS architecture.
License: Apache License 2.0
-
Cloud native search framework that supports to use deep learning/state of the art AI models for search.
License: Apache License 2.0
-
Serverless framework to deploy and monitor machine learning models in Kubernetes - (Video) .
License: Apache License 2.0
-
LocalAI is a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing.
License: MIT License
-
-
-
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more.
License: Apache License 2.0
-
a lightweight, open-source Python tool to get “bolt-on” observability in ML pipelines.
License: Apache License 2.0
-
-
A model server for Apache MXNet from Amazon Web Services that is able to run MXNet models as well as Gluon models (Amazon’s SageMaker runs a custom version of MMS under the hood).
License: Apache License 2.0
-
An open source library to estimate post-deployment model performance (without access to targets). Capable of fully capturing the impact of data drift on performance.
License: Apache License 2.0
-
A rust-powered and multi-stage pipelined model server which offers dynamic batching and more. Super easy to implement and deploy as micro-services.
License: Apache License 2.0
-
A high-performance “serverless” framework focused on data, I/O, and compute intensive workloads. It is well integrated with popular data science tools, such as Jupyter and Kubeflow; supports a variety of data and streaming sources; and supports execution over CPUs and GPUs.
License: Apache License 2.0
-
REST web service for the true real-time scoring (< 1 ms) of Scikit-Learn, R and Apache Spark models.
License: GNU Affero General Public License v3.0
-
Creates HTML profiling reports from pandas DataFrame objects. It extends the pandas DataFrame with df.profile_report() for quick data analysis.
License: MIT License
-
Phoenix is an open source ML observability in a notebook to validate, monitor and fine tune your generative LLM, CV and tabular models.
License: Other
-
An open source Machine Learning Server built on top of a state-of-the-art open source stack for developers and data scientists to create predictive engines for any machine learning task.
License: Apache License 2.0
-
A Redis module for serving tensors and executing deep learning models. Expect changes in the API and internals.
License: Other
-
Open source platform for deploying and monitoring machine learning models in kubernetes - (Video) .
License: Apache License 2.0
-
skops is a Python library helping you share your scikit-learn based models and put them in production.
License: MIT License
-
Open source SDK that provides a unified interface to multiple MLOps projects that enable data scientists to deploy and productionise machine learning systems.
License: Apache License 2.0
-
High-performant framework to serve Tensorflow models via grpc protocol able to handle 100k requests per second per core.
License: Apache License 2.0
-
TorchServe is a flexible and easy to use tool for serving PyTorch models.
License: Apache License 2.0
-
Transformer-deploy is an efficient, scalable and enterprise-grade CPU/GPU inference server for Hugging Face transformer models.
License: Unknown
-
Triton is a high performance open source serving software to deploy AI models from any framework on GPU & CPU while maximizing utilization.
License: BSD 3-Clause "New" or "Revised" License
-
UnionML is an open source MLOps framework that aims to reduce the boilerplate and friction that comes with building models and deploying them to production.
License: Apache License 2.0
-
vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs
License: Apache License 2.0
-
Lightweight solution for profiling and monitoring your ML data pipeline end-to-end
License: Apache License 2.0
Last Updated: Dec 26, 2023