Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't capture batch name for jobs that have left the queue #6

Open
JoshKarpel opened this issue Jan 30, 2020 · 2 comments
Open

Can't capture batch name for jobs that have left the queue #6

JoshKarpel opened this issue Jan 30, 2020 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@JoshKarpel
Copy link
Contributor

The only place we can get guaranteed batch name information is when jobs are still in the queue, via a query. If you ask us to parse an event log whose jobs have all left the queue, we have to go with the default batch name. Although this is probably not a common use case, it will probably be surprising when it happens.

@JoshKarpel JoshKarpel added the bug Something isn't working label Jan 30, 2020
@JoshKarpel JoshKarpel self-assigned this Jan 30, 2020
@JoshKarpel
Copy link
Contributor Author

For now, we should probably just manage to emit a warning when this happens.

@JoshKarpel
Copy link
Contributor Author

This gets particularly nasty when you're running a DAG with many completed jobs in it. You get thousands of rows, because we don't understand how to group them after they've the queue, and unlike a normal submit, they probably all have different cluster ids.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant