Release v0.3.1 (What’s new?).
Welcome to abstract_producer Documentation#
📔 See Full Documentation HERE.
This library provides the abstraction of data producer, which is a common client sending data to stream processing system, such as Apache Kafka, Apache Pulsar, AWS Kinesis, AWS SQS, AWS CloudWatch logs, etc.
It has the following business critical features out-of-box:
group records into micro batch to maximize the utilization of network bandwidth.
use local write-ahead-log to prevent unexpected error and data loss.
auto-retry using exponential backoff strategy to handle transient error.
With this library, it is easy to create data producer client library for any stream processing system.
Usage Examples#
A POC producer: This is a very simple producer that write data to a local, append-only file.
A POC consumer: This is a very simple consumer that work closely with the POC producer for demonstration purpose. You can follow these two example to implement your own producer and consumer library for any stream processing system.
Install#
abstract_producer is released on PyPI, so all you need is to:
$ pip install abstract-producer
To upgrade to latest version:
$ pip install --upgrade abstract-producer