Best practices for consumers in production

[rikvanderkemp]

So I have successfully set up RabbitMQ (in Docker) and have sent messages to RabbitMq and successfully consumed test messages. All in a safe development environment.

What I'm missing in the documentation are some pointers when moving to production.

My idea is to run consumers using supervisord, currently using this for the messenger from Symfony. The disadvantage is, however, code changes mean restarting the consumer(s).

So when I see the following in the documentation:

$ ./app/console rabbitmq:consumer -m 50 upload_picture

It got me thinking, is this the preferred way of setting it up in production? Firing up consumers for limited amounts of messages and then starting new ones when needed? Not sure how that would work.

Or, just multiple consumers for multiple exchanges?

A single consumer handles incoming messages sequentially. So how would you deal with large bulks? An idea would be firing up multiple consumers for a single exchange channel. But again, is this the "right" way?

So I'd like to hear some pointers from people who are using this in production. I'd like to use this in a setup where we could potentially have big bulks of messages that need to be consumed.

[beniamin]

As you said, we at eMAG are running consumers using supervisord.
Usually we do 2 things:

• set graceful_max_execution.timeout - this helps also on long running processes memory leaks(php it's known for not being nice with memory consumption on long running processes)
• restart all supervisord processes right after deploy on production has ended.

Depending on your application, if have both workers and www part, or maybe have multiple workers that impacts the same area of the application, I will advise you to restart your workers immediately after deploy. Otherwise you risk data corruption, or another unwanted behaviour, because some of the workers run the new version and other the old version.

Going on that path, imagine that you set some fixed value for number of messages after the workers should restart. You may end up in a situation in which some of the workers are restarted very soon(because they receive a lot of messages) and others will restart really late, because the amount of messages is really small.

[rikvanderkemp]

Thank you @beniamin. How many supervisord processes are you running if I might ask?

Going on that path, imagine that you set some fixed value for number of messages after the workers should restart. You may end up in a situation in which some of the workers are restarted very soon(because they receive a lot of messages) and others will restart really late, because the amount of messages is really small.

Good point, this is an interesting puzzle.

[beniamin]

@rikvanderkemp supervisord is widely used in eMAG environment. In my applications, we have an average of 90-100 php processes "supervised" by supervisord.
As I said, we restart all these processes on every release. It usually takes around few tens of seconds.

[mihaileu]

@rikvanderkemp I'll add some extra tips, maybe it will help you

It got me thinking, is this the preferred way of setting it up in production? Firing up consumers for limited amounts of messages and then starting new ones when needed? Not sure how that would work.

• use a smaller prefetch_count in order to balance the messages on all machines/containers
• if you are using AWS ECS (cloud) or something similar you ca set scale alarms based on cpu usage or number of messages that are present on queues and scale the service up or down.
• if you are on your own machines with docket stack you can setup a container with supervisor as input command: ["/usr/bin/supervisord", "-n"] and deploy: replicas: x and you will have a maximum rate obtained by number of supervisor workers and replicas nodes. You will have stale workers which will wait for messages and a maximum processing rate when the queue will have a lots of messages.
• If your application is not dockerized you can have multiple machines with supervisor
• there's also another service that might offer you additional benefits on prem: https://www.nomadproject.io/

We are using all of the above, it depends where you will host your code and what you want to obtain:

• If you are using a cloud infrastructure, you can try AWS ECS or something similar from other providers
• if you are at the beginning with your project and you on prem or some virtual machines somewhere, you can start with supervisor. This might be a starting point.
• if you want to spread you workers on prem/cloud and have additional benefits, try nomad

Or, just multiple consumers for multiple exchanges?

you should have consumer based on what queues are you creating and what you want to accomplish.
i.e.: if you have a picture defined as topic where you publish multiple message with different routing keys like:
resized , uploaded, whatever, you can set consumers for each operation which it will decouple you code.

A single consumer handles incoming messages sequentially. So how would you deal with large bulks? An idea would be firing up multiple consumers for a single exchange channel. But again, is this the "right" way?

Depending on what orchestration you choose you can obtain a stable processing rate and if is acceptable you can got with it. If you choose a managed system you can scale in/out base on different parameters in order to cut cost/improve performance.

lmk if it was helpful or not