Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE_REQUEST]: Distributed processing #69

Open
miguelaeh opened this issue Nov 14, 2023 · 0 comments
Open

[FEATURE_REQUEST]: Distributed processing #69

miguelaeh opened this issue Nov 14, 2023 · 0 comments

Comments

@miguelaeh
Copy link
Collaborator

miguelaeh commented Nov 14, 2023

Pipeless 1.x does not yet include distributed processing as its predecessor (0.x) which had it enabled by default.

Implementing distributed processing should be simple. It just requires to provide an option (--distributed) that replaces the local bus by a message broker like Kafka, Redis or RabbitMQ. Supporting different brokers would be ideal.

Also, the Pipeless node in distributed mode should behave as a worker node, simply taking events from the broker and processing them. Only of frame change events should be moved to the broker and ideally only when the node receiving the stream has not enough capacity.

This includes the implementation of a distributed KV store. See #71

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant