test: Use the bash container and a matrix strategy for bash versions

[lens0021]

There are some change between versions of bash. (See #592 (comment))
So makes a matrix strategy.

[lens0021]

Note: A test requires the test run by a non-root user.

[Mte90]

maybe it should detect if running inside docker so it will works always

[lens0021]

I've inspected it, but I am sorry that it does not make sense for me. Let me explain this.

The test I mentioned is:

amber/src/tests/stdlib/is_root.ab

Lines 1 to 6 in cba1671

	import { is_root } from "std/env"
	main {
	if not is_root() {
	echo "Succeeded"
	}
	}

and the implementation of is_root() is:

amber/src/std/env.ab

Lines 100 to 107 in cba1671

	/// Checks if the script is running with a user with root permission.
	pub fun is_root(): Bool {
	if trust $ id -u $ == "0" {
	return true
	}
	return false
	}

I think the best way to know whether the test is run as root should be just the implement of is_root() function. So If we use the best way, the test will be:

if trust $ id -u $ == "0" and is_root() {
    echo "Succeeded"
}

And it seems it tests nothing, because it tests just redundant conditions.

I think the test should be moved out of an amber script, so the best result should be like:

$ amber is_not_root.ab
I am a not root user!
$ sudo amber is_root.ab
I am a root user!

And it is out of the current test process of stdlibs. I am not sure what should I do, so I will leave --user 405 as it.

[Mte90]

Seems that fix also #422

[Mte90]

it seems also that a test fails for busybox

[lens0021]

I've excluded the busybox test now because #617 is not so simple to resolve.

[hdwalters]

I don't see anything glaring, but just questions for now. Also, bear in mind that I know very little about GitHub or CI workflows; please take opinions on the YAML file with a pinch of salt.

[Mte90]

@hdwalters it isn't better that this code there is only on a debug build like you did for another PR?

[hdwalters]

I don't like debug and release builds having different functionality. I reverted this behaviour in your documentation usage code when I did the revamped CLI; it's now enabled with a new command line option amber docs --usage.

[hdwalters]

Although in this case, it may make sense. I suggest asking for opinions on the "general" Discord group, and flagging @lens0021.

[lens0021]

It will be helpful if there is a CLI option to choose other bash executable or kind of entry point. Some other environment, for example Termux(an Android terminal emulator) has no /usr/bin/env so always fails to run amber eval. (amber build is ok)

[hdwalters]

Perhaps, but that's out of scope for this PR. AIUI, @Mte90's point was that since your GITHUB_ACTIONS_BASH_CONTAINER change was intended to support cargo test in a Docker container, it should probably not be enabled for release builds. If we agree on this, we could do something like:

#[cfg(not(windows))]
#[cfg(debug)]
fn find_bash() -> Option<Command> {
    // Use docker or /usr/bin/env...
}

#[cfg(not(windows))]
#[cfg(not(debug))]
fn find_bash() -> Option<Command> {
    // Use /usr/bin/env...
}

[Mte90]

Yep that's my idea.

[b1ek]

a bunch of things are wrong with this

1. dependencies are supposed added into a docker container in the build stage (dockerfile), not during runtime
2. there is no reason to expose the container to the host network
3. i suppose it will write as root to the PWD, which is usually not a very good idea since it will cause dubious ownership. the correct way is to run it as the current uid&gid inside docker
   3.1. (maybe) use docker volumes instead. its a usually cleaner way to mount something in a container, but it depends on a specific case and im not sure what exactly is happening there
4. you have set sleep inf as entrypoint, which is a wrong understanding of how docker works. i suppose you want to cargo test --all-targets --all-features inside docker, so you should set that as the entrypoint

[lens0021]

1. If other maintainers agree, I would create a Dockerfile after they're choosing a registry to store the image (Docker Hub or GitHub Cotainer registory(ghcr)). But maintaining an image requires another sort of maintaining resource and this building contains just two command, I used the inline commands. If your concern is the speed of testing, I will try caching the image with docker write and actions/cache.
2. That was because of the test

   amber/src/tests/extra.rs

   Lines 65 to 68 in 82817af

   	let server = std::thread::spawn(http_server);
   	let code = fs::read_to_string("src/tests/stdlib/no_output/download.ab")
   	.expect("Failed to open stdlib/no_output/download.ab test file");

   which starts a web server and tries to download a file from it. I didn't want to i) install rust in a bash image or ii) install bash in the rust image. If a cleaner solution is needed, we could set up a web server instead of rust code, maybe using docker-compose, and share their network, then do the test. I wanted the shortest answer, so used --network=host.
3. I agree. I saw that junk files are created in $PWD on my laptop. I will fix, thank you.
4. (I will respond as a reply of the below comment)

[Mte90]

I am against a Docker image for our needs to maintain etc

[lens0021]

I saw that junk files are created in $PWD on my laptop.

I've realized I didn't need to mount $PWD at all, so I've removed it.

[b1ek]

1. I would create a Dockerfile after they're choosing a registry to store the image

there's no need for that, you can have the dockerfile in the repo and just build from it when you have to.

$ cat Dockerfile
FROM bash:4.5
RUN apk add something
$ docker build -t testbash . # will use $PWD/Dockerfile
$ docker run --name tests --remove testbash cargo test

[Mte90]

I don't think that there is a majority that want a dockerfile for the Amber project right now, so I would ignore that suggestion right now.

[Ph0enixKM]

At first I was like what the heck 😆!? Why do we run docker image instead of running it from the CI itself. But then I realized that we just want to run the minimal bash docker with the generated script only if we're in the CI environment. This should be better documented (at least a comment should be left that clarifies what's going on before the developer has a "wtf" moment reading this snippet).

I like this approach, because it's pretty smart and non-invasive.

Do we skip this possibility that we may want to run the tests in CI workflow without any docker container? And also... if we ever wanted to move CI pipeline to anywhere else than GitHub.... or maybe I want to test the compiler locally with the docker container... I'd at the very least create our own environment variable like AMBER_TEST_STRATEGY="docker".

Also... the args shouldn't be hardcoded. At the very least we should let the developer change the arguments like container name and other arguments. Can we create additional environment variable that would interpolate those arguments? Like: AMBER_TEST_DOCKER_ARGS="exec --workdir /root ..."

And then:

Suggested change

	if env::var("GITHUB_ACTIONS_BASH_CONTAINER").is_ok() {
	let mut command = Command::new("docker");
	command.args(["exec", "--workdir", "/root", "--user", "405", "test_container", "bash"]);
	Some(command)
	} else {
	let mut command = Command::new("/usr/bin/env");
	command.arg("bash");
	Some(command)
	}
	if env::var("AMBER_TEST_STRATEGY").is_ok_and(|value| value == "docker") {
	let mut command = Command::new("docker");
	let args_string = env::var("AMBER_TEST_ARGS").expect("Please pass docker arguments in AMBER_TEST_ARGS environment variable.");
	let args: Vec<&str> = args_string.split_whitespace().collect();
	command.args(args);
	Some(command)
	} else {
	let mut command = Command::new("/usr/bin/env");
	command.arg("bash");
	Some(command)
	}

[Mte90]

@lens0021 about this?

[lens0021]

This should be better documented

I've added a function documentation. "Return bash command. In some situations, mainly for testing purposes, this can return a command, for example, containerized execution which is not bash but behaves like bash." 7c610c9

(Change suggestion)

Committed as 4b96959. Thanks

Do we skip this possibility that we may want to run the tests in CI workflow without any docker container?

I do not understand this sentence. Should I remove the docker approach from this PR?

I want to test the compiler locally with the docker container.

I will write an amber script for this... I am trying

[Mte90]

@Ph0enixKM

[lens0021]

I've written the first version of the script: https://github.com/lens0021/amber/blob/38d093d/run-various-bash-test.ab

You can try it on this branch: (of course, only if you are okay to download an arbitrary script using curl)

$ curl -sL https://raw.githubusercontent.com/lens0021/amber/38d093d/run-various-bash-test.sh | bash -s -- --help
bash [-h|--help] [--sleep NUM] [VERSION(S)]

Examples:
bash                 # Run test on bash 5.1. (default)
bash 3.2-5.2         # Run test on bash 3.2 to 5.2.
bash 4.4 5.2         # Run test on bash 4.4 and 5.2.
bash 3.2-4.1 5.2     # Mixed range is not supported.
bash --sleep 300 5.2 # Launch a container which lives 300 seconds. If tests are not done in the lifetime, bad things happen.

$ curl -sL https://raw.githubusercontent.com/lens0021/amber/38d093d/run-various-bash-test.sh | bash -s -- 5.1 5.2
...

(The examples show a wrong executable, anyway)

I have not decided where to place this. Feel free to copy, modify or locate in the GPL 3 manner please.

[lens0021]

• docker save and cache the image

I will do this on another PR later.

[Mte90]

let's see if now the tests works, I would give to this PR the priority.

[Mte90]

nevermind we need #634

[lens0021]

Rebased and re-enabled 3.2, 4.0 and 4.1 tests.

[Ph0enixKM]

I'll give it another look tomorrow. Thanks for your help @lens0021