Support directories in artifact_paths

[jsoo1]

Suppose I want to upload all contents of a directory foo/bar. I would love to specify artifact paths like this:

artifact_paths:
  - foo/bar

I would expect this to be treated the same as foo/bar/*. But actually no artifacts are uploaded, even if the directory exists and is non-empty. This is further complicated by the name of the variable/key. The problem is thatartifact_paths are not paths, they are patterns. It is even more confusing that they do not support directories which are paths!

Suppose I have a command step that wants to create one directory from $BUILDKITE_ARTIFACT_PATHS if it doesn't exist. How would I go about it without first-class directory support?

Well, let's try the common shell script and assume we have the following glob pattern configured:

artifact_paths:
  - foo/**/*

In our command:

# Try to create directory foo
mkdir $BUILDKITE_ARTIFACTS_PATHS

... oops, now we created a directory called foo/**/*, what I really wanted was to create a directory called foo and put all my artifacts in that directory.

This is even more confusing as a user because of the existence of nullglob, dotglob and the other variety of shell options. Maybe we should all stop using shell scripts (I don't disagree). But I think for now, it would be better to support directories in artifact_paths.

Describe the solution you'd like
If a path foo is specified in artifact_paths and it is a directory that exists, it should be treated like foo/* during the upload step.

Edit: I think the directory should be treated like foo/**/*