7 Stream Processing Frameworks for Real-Time Data Processing

Open Source Stream Processing Frameworks

• 

  Apache Flink

  

  Open source stream processing framework with powerful stream and batch processing capabilities.

  License: Apache License 2.0

  GitHub
  
• 

  Apache Samza

  

  Distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.

  License: Apache License 2.0

  GitHub
  
• 

  Brooklin

  

  Distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.

  License: Unknown

  GitHub
  Website: Unknown
• 

  Bytewax

  

  Flexible Python-centric stateful stream processing framework built on top of Rust engine.

  License: Apache License 2.0

  GitHub
  Website: https://docs.bytewax.io/
• 

  FastStream

  

  A modern broker-agnostic streaming Python framework supporting Apache Kafka, RabbitMQ and NATS protocols, inspired by FastAPI and easily integratable with other web frameworks.

  License: Apache License 2.0

  GitHub
  Website: https://faststream.airt.ai/latest/
• 

  Faust

  

  Streaming library built on top of Python’s Asyncio library using the async kafka client inspired by the kafka streaming library.

  License: Other

  GitHub
  
• 

  Apache Kafka

  

  Kafka client library for buliding applications and microservices where the input and output are stored in kafka clusters.

  License: Apache License 2.0

  GitHub
  

Last Updated: Dec 26, 2023