-
Notifications
You must be signed in to change notification settings - Fork 303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Directory artifact paths #1571
base: main
Are you sure you want to change the base?
Directory artifact paths #1571
Conversation
3b56f7f
to
bd82ef5
Compare
After some more looking at this, I think it is slightly more complicated than this (but not much). |
61f4a86
to
7a5508d
Compare
Ok I think this is a start now. I see that maybe in the past |
7a5508d
to
d0eac4e
Compare
A brief test on our CI confirms this does what I want. Though the symlinks need some work and there could be cases of which I am unaware. |
6ba2618
to
77dbc24
Compare
Ok tests pass and I think are actually correct (and symlinks are handled). |
d98250e
to
04aa7f8
Compare
Hi @jsoo1! This is an interesting suggestion, and thank you for the contribution, but If you want a shared value for commands and for artifacts then perhaps you want to abstract that. I would suggest doing something like this instead, for example: steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- ${PREFIX}/bin/*
command: make PREFIX="${PREFIX}" |
Hi @sj26 thank you for your feedback!
Hm. Why is that? I would assume that almost any env var in a shell would be subject to some use by a command.
I can see this being useful, thank you! I would like to push back on this decision a bit, though. While I could probably define my own env var for the purpose of My problems are with glob semantics and violation of the principle of least surprise.
Thus my expectations are doubly violated when a directory I list in If we modify the example slightly: steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- ${PREFIX}/bin
command: make PREFIX="${PREFIX}" Then no contents of The primary benefit of supporting directories would be to increase the usability of pipeline definitions. The use of Otherwise I'm not sure I see the downsides. Is there extra maintenance that would be required due to this change I am not aware of? In any case, we love Buildkite and use it heavily. Thank you for your work! |
The agent puts all the parameters required to run the job into the environment and then invokes a bootstrap process which consumes a lot of those variables to run hooks etc around your command to provide the surrounding behaviour. You can upload the binaries produced by make with something like this: steps:
- env:
- PREFIX=the/place/to/build
artifact_paths: ${PREFIX}/bin/*
command: make PREFIX="${PREFIX}" or you could be explicit and handle the upload yourself if you want more control: steps:
- env:
- PREFIX=the/place/to/build
command: |
make PREFIX="${PREFIX}"
buildkite-agent artifact upload "${PREFIX}/bin/*" I reckon we'll add a recursive mode for artifact upload at some point. Those commands need a bit of a rethink for better ergonomics. Maybe something like: steps:
- env:
- PREFIX=the/place/to/build
command: |
make PREFIX="${PREFIX}"
# Doesn't work right now, but might at some point in the future
buildkite-agent artifact upload -r "${PREFIX}/bin" |
Ok... can you expand a bit on how that relates to this conversation? Is that an additional maintenance burden? Sorry I'm having trouble seeing the relation.
... but importantly not:
That's what this pr does, no? The artifact uploader is used here: agent/clicommand/artifact_upload.go Line 149 in c7abbce
And this this pr takes care of the hypothetical
I'm still not seeing the downsides (though I'm open to hearing them!). Note the current state of affairs actually thwarts the glob matching by explicitly ignoring directories! See here: agent/agent/artifact_uploader.go Line 145 in c7abbce
Maybe what would help is some background on why that was done, because it is so confusing! My best guess is the previous performance downsides of Thanks again. |
Perhaps I've been a little too polite here. Not allowing directories in the steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- ${PREFIX}/bin/*
command: make PREFIX="${PREFIX}" Then one day we decide to add a steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- ${PREFIX}/bin/*
+ - ${PREFIX}/lib/*
command: make PREFIX="${PREFIX}" But wait! This can be simplified! steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- - ${PREFIX}/bin/*
- - ${PREFIX}/lib/*
+ - ${PREFIX}/*
command: make PREFIX="${PREFIX}" ... woops, now our job has this in the logs: Skipping directory bin
Skipping directory lib
steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- - ${PREFIX}/*
+ - ${PREFIX}/**/*
command: make PREFIX="${PREFIX}" Sweet! We can even add Well, suppose then we want to add a README.md to the root of steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- ${PREFIX}/**/*
+ - ${PREFIX}/*
command: make PREFIX="${PREFIX}" But we still have these annoying steps:
- env:
- PREFIX=the/place/to/build
artifact_paths:
- ${PREFIX}
command: make PREFIX="${PREFIX}" Which, as I noted in the original description, can be shorted to this (in the simplest cases): steps:
artifact_paths:
- the/place/to/build
command: make PREFIX="${BUILDKITE_ARTIFACT_PATHS}" Which is what this PR enables. |
03f86bd
to
e9620e6
Compare
e9620e6
to
5af6007
Compare
Closes #1570
Previously, only glob patterns were allowed in
artifact_paths
. The name was confusing and also made certain patterns impossible. In particular, in a common bash script, it is impossible to unambiguously "unglob" a string. For instance, in a shell script with$BUILDKITE_ARTIFACT_PATHS=foo/**/*
and$PWD
in an empty directory, the following would create a directory named literallyfoo/**/*
(assuming no globbing shell options were set):mkdir $BUILDKITE_ARTIFACT_PATHS
Even if you were to try one of the globbing related shell options, there are none that would accomplish the desired effect.
Of course, the workaround would be to "just"
mkdir foo
in the shell script. However this breaks the decoupling between buildkite and the script as the buildkite configuration must "know" about the shell script and vice versa.This change also makes new things possible.
One example is the following in a shell script
command
step:make prefix=$BUILDKITE_ARTIFACT_PATHS
Which is a common idiom in many build systems.