Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Girder 5 #1553

Open
wants to merge 42 commits into
base: master
Choose a base branch
from
Open

Girder 5 #1553

wants to merge 42 commits into from

Conversation

zachmullen
Copy link
Member

No description provided.

@zachmullen zachmullen force-pushed the girder-5 branch 4 times, most recently from 2b9545d to 2004e08 Compare June 20, 2024 22:51
@manthey
Copy link
Member

manthey commented Jul 29, 2024

There are some jobs that are run locally (so that they share the python environment of the girder server), and some that are run remotely (so that they use the environment of girder_worker). This has been a virtue for, example, converting images to a more efficient format without needing to have a girder_worker node spun up. If we convert all of these jobs to celery jobs, we might want to retain a deployment mode with a local queue that is distinct from the girder_worker queue.

@zachmullen
Copy link
Member Author

In Girder 5, anything that needs to be done asynchronously will be offloaded from the Girder HTTP server, with celery being the built-in choice for doing so. To accommodate that change, the idea is to make the celery environment as similar as possible to the server environment, chiefly by giving it access to the mongo database.

we might want to retain a deployment mode with a local queue that is distinct from the girder_worker queue.

Indeed, this is the goal; I've started replacing these tasks with raw celery tasks rather than girder_worker tasks. The deployment mode now uses multiple processes and eschews background threads. See https://github.com/girder/histomics-dev-env/blob/main/docker-compose.yml for an example of a dev environment that reflects this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants