Collaborating on a Kafka Streams-like library equivalent built on confluent-kafka-python

[truthordata]

Hello those at confluent, and community at large!

I was hoping to start a line of communication with the confluent team, but also the community at large, with respect to our efforts on a (soon-to-be) open source library we have been actively working on, which is intended to function like Kafka Streams, but built on confluent-kafka-python. For us, this is actively replacing Faust as our primary client library.

With that in mind, some background on who "we" are: I am a developer at Red Hat on a small technical team in Red Hat marketing. We are primarily a python shop, and we have been using Kafka + Faust + confluent-kafka-python for a few years now in our stack.

As some of you may be aware, Faust is another Kafka client python library that largely attempts to replicate Kafka Streams. These days, it seems like a dead project that has some forks, but the original contributors/visionaries seem long gone. Despite that, Faust is fairly heavy/complex for our more microservice-focused distributed deployment use-case; we use Openshift with apps that can be dynamically deployed and scaled as needed, whereas it seems like Faust was designed to be more monolithic and self-contained in operation with agents and workers (not to say you can't run/treat Faust as the former, we did for a while...but it's not how it was designed).

All this is to say, we needed something different, but we still needed the tabling and exactly-once processing that Faust provided, and more recent iterations of confluent-kafka-python make this possible!

Right now we are still very actively developing and debugging the core functionality, but we have something that now does tabling + changelogging + recovery with RocksDB, with exactly-once semantics, and it automatically handles all the topic committing correctly! All you do is instantiate a simple class that you pass your business logic, and a few other setup details, to.

I'm happy to go into more details and expand on how it works, but I wanted to extend a line of communication first and see how best to get involved with this community. Also, there are a few more things we are trying to get in order first before we feel it is ready to release at large (removing proprietary logic, stability enhancements, etc).

I myself am new to open sourcing something like this, and so I am not sure the best way to go about this. I would definitely love to interact and see if there are sub-optimal patterns in our logic flow, get the word out for people who might find our library useful, and of course, open it up for people to contribute as a whole and hopefully make people's lives easier! =)

Please reach out to me here, or (if it is allowed), I can add contact info as well. Thanks!!

[truthordata]

Hey everyone!

As an update, you can find our library here