Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backend: Refresh job status #451

Merged
merged 2 commits into from
Nov 29, 2024
Merged

backend: Refresh job status #451

merged 2 commits into from
Nov 29, 2024

Conversation

dpoulopoulos
Copy link
Contributor

@dpoulopoulos dpoulopoulos commented Nov 28, 2024 •
edited
Loading

What's changing

The /jobs endpoint can return the status of a specific Ray job, but this status is not updated beyond "created".

Optionally refresh the database record before returning:

  • Immediately return statuses "Succeeded" and "Failed" because they cannot change.
  • Otherwise, consult Ray first, and if the status differs from the one in the DB, update the DB and return the updated record.

Closes #382

How to test it

  1. Upload a dataset
  2. Launch a job (e.g. evaluation with BART)
  3. List jobs, find the job_id of one (e.g., the one you just created)
  4. GET /jobs/{job_id} --> created
  5. Check on the ray dashboard until that job has finished
  6. GET /jobs/{job_id} --> finished

I already...

  • added some tests for any new functionality
  • [NA] updated the documentation
  • [NA] checked if a (backend) DB migration step was required and included it if required

@dpoulopoulos dpoulopoulos linked an issue Nov 28, 2024 that may be closed by this pull request
[BUG] The /jobs endpoint should be able to return an up to date job status #382
Closed
@dpoulopoulos dpoulopoulos force-pushed the 382-job-status-endpoint branch from baf16c3 to 92ed207 Compare November 28, 2024 15:15
@github-actions github-actions bot added the schemas Changes to schemas (which may be public facing) label Nov 28, 2024
@dpoulopoulos dpoulopoulos force-pushed the 382-job-status-endpoint branch 2 times, most recently from 76f2d15 to f94fb20 Compare November 29, 2024 12:15
@dpoulopoulos dpoulopoulos marked this pull request as ready for review November 29, 2024 12:15
@dpoulopoulos dpoulopoulos added bug Something isn't working api Changes which impact API/presentation layer and removed schemas Changes to schemas (which may be public facing) labels Nov 29, 2024
@dpoulopoulos dpoulopoulos marked this pull request as draft November 29, 2024 12:24
@dpoulopoulos dpoulopoulos force-pushed the 382-job-status-endpoint branch from f94fb20 to 00f483a Compare November 29, 2024 13:29
@github-actions github-actions bot added the schemas Changes to schemas (which may be public facing) label Nov 29, 2024
@dpoulopoulos dpoulopoulos force-pushed the 382-job-status-endpoint branch from 00f483a to 9d2e26f Compare November 29, 2024 13:31
@dpoulopoulos dpoulopoulos marked this pull request as ready for review November 29, 2024 13:31
aittalam
aittalam approved these changes Nov 29, 2024
View reviewed changes
Copy link
Member

@aittalam aittalam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Dimitris! I added a few comments but after addressing them this LGTM 👍

@dpoulopoulos dpoulopoulos force-pushed the 382-job-status-endpoint branch 2 times, most recently from cbf856f to edd5051 Compare November 29, 2024 15:39
@dpoulopoulos
backend: Refresh job status
fc964c3 
The `/jobs` endpoint can return the status of a specific Ray job, but
this status is not updated beyond "created".

Optionally refresh the database record before returning:

* Immediately return statuses "Succeeded" and "Failed" because they
  cannot change.
* Otherwise, consult Ray first, and if the status differs from the one
  in the DB, update the DB and return the updated record.

Refs #382

Signed-off-by: Dimitris Poulopoulos <[email protected]>
@dpoulopoulos
tests: Test that /jobs/{job_id} refreshes status
53c2ccb 
When a new job is created, the initial status in the database is set to
"created."

Verify that the status stored in the database matches the status of the
job returned by Ray.

Closes #382

Signed-off-by: Dimitris Poulopoulos <[email protected]>
@dpoulopoulos dpoulopoulos force-pushed the 382-job-status-endpoint branch from edd5051 to 53c2ccb Compare November 29, 2024 15:40
@dpoulopoulos dpoulopoulos merged commit 61e1589 into main Nov 29, 2024
9 checks passed
@dpoulopoulos dpoulopoulos deleted the 382-job-status-endpoint branch November 29, 2024 15:54
@ividal ividal mentioned this pull request Dec 9, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aittalam aittalam aittalam approved these changes

Assignees
No one assigned
Labels
api Changes which impact API/presentation layer backend bug Something isn't working schemas Changes to schemas (which may be public facing)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] The /jobs endpoint should be able to return an up to date job status
2 participants
@dpoulopoulos @aittalam