Tar errors on cache restore after toolchain installation #424

matthewhughes-uw · 2023-09-15T08:51:32Z

Description:

When the version of Go in the go directive of go.mod is different from the version in the toolchain directive there is a cache conflict which causes errors on cache extraction.

Note: I think this is related to, but different from #403 since that appears to be covering the situation where some other tool is also working on the cache, but this issue is purely for the action running on a hosted runner without any additional changes to the environment etc.

Action version:

actions/setup-go@v4

Platform:

Ubuntu
macOS (untested)
Windows (untested)

Runner type:

Hosted
Self-hosted

Tools version:

go 1.21.0
toolchain go1.21.1

Repro steps:

See repo https://github.com/matthewhughes-uw/setup-go-test/, very basic repo with a single workflow that just runs this action (https://github.com/matthewhughes-uw/setup-go-test/blob/main/.github/workflows/go.yaml):

First run of the action https://github.com/matthewhughes-uw/setup-go-test/actions/runs/6195831453/job/16821285537

Installs Go 1.21.0 (version detected from go-version-file
Go detects toolchain and performs a toolchain install of Go 1.21.1 (see output go: downloading go1.21.1 (linux/amd64))
This places some files under ../../../go/pkg/mod/golang.org/[email protected]
Finds no existing cache, doesn't try to extract anything
Caches /home/runner/go/pkg/mod (which contains the files above) and /home/runner/.cache/go-build

Second run of the action https://github.com/matthewhughes-uw/setup-go-test/actions/runs/6195844389/job/16821321968

First two steps as above
Finds an existing cache, attempts to extract it, but because of the toolchain installation step files under ../../../go/pkg/mod/golang.org/[email protected] are already present on the machine, causing errors from tar

Expected behavior:

Caching of the toolchain is handled gracefully, no errors between runs that don't contain any extra caching logic outside of what the action provides by default

Actual behavior:

Errors on cache extraction (see repro steps)

The text was updated successfully, but these errors were encountered:

dsame · 2023-09-15T11:46:17Z

Hello @matthewhughes-uw , thank you for the input. We are starting to investigate the problem.

thall · 2024-01-08T09:47:09Z

Hello @matthewhughes-uw , thank you for the input. We are starting to investigate the problem.

do you have any updates around this to share?

HaraldNordgren · 2024-01-17T12:50:23Z

@matthewhughes-uw Any updates on this? After updating to Go version:

--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,8 @@
 module github.com/company/repo

-go 1.20
+go 1.21.1
+
+toolchain go1.21.6

 require (

We start having these warnings in the build:

/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected]/lib/time/mkzip.go: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected]/lib/time/zoneinfo.zip: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected]/lib/time/README: Cannot open: File exists
...

HaraldNordgren · 2024-01-17T13:23:39Z

The issue seems to be caused by the setup-go being confused about the difference in "Go" version and the newly introduced "Toolchain" version.

It first downloads and caches 1.21.1
Then tries to download 1.21.6 as toolchain version
But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache
And boom... it 1.21.1 already exists in the cache from the first step! 🧨

15s
Run actions/setup-go@v4
  with:
    go-version-file: go.mod
    check-latest: true
    token: ***
    cache: true
Setup go version spec [1](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:1).[2](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:2)1.1
Attempting to resolve the latest version from the manifest...
matching 1.21.1...
Resolved as '1.21.1'
Attempting to download 1.21.1...
matching 1.21.1...
Acquiring 1.21.1 from https://github.com/actions/go-versions/releases/download/1.21.1-6107260229/go-1.21.1-linux-x64.tar.gz
Extracting Go...
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/0b61aba1-9a59-4690-b4c0-4f21d65b8487 -f /home/runner/work/_temp/b0a91240-cd7a-42cc-bc74-0757c51b5e9b
Successfully extracted go to /home/runner/work/_temp/0b61aba1-9a59-4690-b4c0-4f21d65b8487
Adding to the cache ...
Successfully cached go to /opt/hostedtoolcache/go/1.21.1/x64
Added go to the path
Successfully set up Go version 1.21.1
go: downloading go1.21.6 (linux/amd64)
/opt/hostedtoolcache/go/1.21.1/x64/bin/go env GOMODCACHE
/opt/hostedtoolcache/go/1.21.1/x64/bin/go env GOCACHE
/home/runner/go/pkg/mod
/home/runner/.cache/go-build
Received 151688858 of 151688858 (100.0%), 166.5 MBs/sec
Cache Size: ~145 MB (151688858 B)
/usr/bin/tar -xf /home/runner/work/_temp/22489b85-a7d8-49[3](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:3)7-8083-e821[4](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:4)04[5](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:5)3fc7/cache.tzst -P -C /home/runner/work/hive/hive --use-compress-program unzstd
/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected].[6](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:6).linux-amd64/lib/time/mkzip.go: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected].[21](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:22).6.linux-amd64/lib/time/zoneinfo.zip: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected]/lib/time/README: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected]/lib/time/update.bash: Cannot open: File exists
...

matthewhughes-uw · 2024-01-18T10:05:31Z

The issue seems to be caused by the setup-go being confused about the difference in "Go" version and the newly introduced "Toolchain" version.
* It first downloads and caches 1.21.1
* Then tries to download 1.21.6 as toolchain version
* But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache
* And boom... it 1.21.1 already exists in the cache from the first step! 🧨

Note: the toolchain download is not done by setup-go, it's done by Go itself, e.g.

module example.com/my-mod

go 1.21.0

toolchain go1.21.1

Then if you try to run a go command using a version that doesn't match the toolchain the toolchain will be downloaded (creating new caches to ensure we download the new toolchain, and not just silently use an existing one):

$ mkdir gocache gomodcache
$ GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 env
go: downloading go1.21.1 (linux/amd64)

So unless setup-go is configured to use the go version (or perhaps, any newer version) specified in the toolchain directive then there will be this download. setup-go then continues to use the version it was configured with to run any future go commands (which may just invoke the toolchain version in the background)

$ GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 version
go version go1.21.1 linux/amd64

EDIT: I'm not sure if setup-go could just set the GOTOOLCHAIN env var to ensure it always use the configured version (docs https://go.dev/doc/toolchain)

The GOTOOLCHAIN environment setting can force a specific Go version, overriding the go and toolchain lines (and possibly just use the version from the toolchain directive by default)

$ GOTOOLCHAIN=go1.21.0 GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 version 
go version go1.21.0 linux/amd64

(from a quick test this skipped any toolchain download)

HaraldNordgren · 2024-01-18T16:03:57Z

@matthewhughes-uw That makes sense. What I don't understand is why Go on the second download defaults back from downloading 1.21.6 to instead getting 1.21.1 again.

Because if this didn't happen, then there would be no cache problem.

How I solved it for now is to upgrade using go get go, upgrading to 1.21.6 locally, which then removes the toolchain line altogether, since Go and Toolchain versions are now the same.

Then the caching issue does not appear.

matthewhughes-uw · 2024-01-22T10:39:06Z

What I don't understand is why Go on the second download defaults back from downloading 1.21.6 to instead getting 1.21.1 again.

There's a mix of Go and setup-go downloading bits. What I understand from your previous point:

It first downloads and caches 1.21.1: This is setup-go, and ends with the line Successfully set up Go version 1.21.1
Then tries to download 1.21.6 as toolchain version: setup-go then invokes go (specifically /opt/hostedtoolcache/go/1.21.1/x64/bin/go) which triggers a download of the toolchain, this is the line go: downloading go1.21.6 (linux/amd64)
But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache for go1.21.1: This is setup-go downloading the requested cache (not go doing anything)
And boom... it 1.21.1 already exists in the cache from the first step! 🧨: the issue is [email protected]/ exists in the cache (the toolchain downloaded from the 2nd step)

So the issue is we populate some of go1.21.1's GOMODCACHE and then restore the cache, which then complains because there are already some files there

You could remove the toolchain directive from your go.mod (unless you do have a requirement for a specific version of Go greater than what the go directive specifies when working in the module, see https://go.dev/blog/toolchain#toolchain-management)

Set the same Go version as specified in the `go.mod` because of the issue below: - actions/setup-go#424 Signed-off-by: Tiago Natel <[email protected]>

Set the same Go version as specified in the `go.mod` because of the issue below: - actions/setup-go#424 Additionally, the rand.Seed() was updated to use rand.New(rand.NewSource(seed)) because the former was removed from Go1.20. Signed-off-by: Tiago Natel <[email protected]>

Set the same Go version as specified in the `go.mod` because of many similar issues related to the setup-go cache unable to work when go.mod specifies a different version. - actions/setup-go#424 Additionally, the rand.Seed() was updated to use rand.New(rand.NewSource(seed)) because the former was removed from Go1.20. Signed-off-by: Tiago Natel <[email protected]>

spencerschrock · 2024-02-07T21:27:05Z

One way of avoiding this could be if actions/setup-go had a way of reading the toolchain directive instead of the go directive. Although it's still useful to test against the minimum version specified by the go directive, so I don't think changing the behavior of go-version-file is a good solution.

You could do your own toolchain version parsing in an earlier step, and set an env var or something:

uses: actions/setup-go@v5
with:
    go-version: ${{ env.GO_VERSION }}

`go version` is run before downloading the cache, but if this is run with a version of `go` that triggers a Toolchain download[1], e.g. if the installed Go is 1.20.0 but `go.mod` has a toolchain directive `go1.20.1` then a toolchain is downloaded to e.g. `$GOMODCACHE/golang.org/[email protected]`, if this file already exists in the cache (e.g. this is the second run of this action we not cache invalidation) then the cache download will try and overwrite these files resulting in noisy errors like: /usr/bin/tar: ../../../go/pkg/mod/golang.org/[email protected]/lib/time/mkzip.go: Cannot open: File exists Instead, force `go` to use the local toolchain (i.e. the one the one that shipped with the go command being run) via setting the `GOTOOLCHAIN` environment variable[1]: > When GOTOOLCHAIN is set to local, the go command always runs the bundled Go toolchain. This addresses actions#424 [1] https://go.dev/doc/toolchain#select

matthewhughes-uw · 2024-02-08T09:07:58Z

I think this can be avoided by just telling Go to use the local toolchain when running go version before downloading the cache, see: #456

I think there's another can of worms/separate issue around which version of Go should this action use when a toolchain directive is present, e.g. at the moment if you specify go-version but have a go.mod specifying a newer-version of Go I expect any go command will run using the toolchain version, which may be unexpected

## What this PR does / why we need it: Set the same Go version as specified in the `go.mod` because of many similar issues related to the setup-go cache unable to work when go.mod specifies a different version. - actions/setup-go#424 Additionally, the rand.Seed() was updated to use rand.New(rand.NewSource(seed)) because the former was removed from Go1.20. ## Which issue(s) this PR fixes: no ## Special notes for your reviewer: ## Does this PR introduce a user-facing change? ``` no ```

lukeed · 2024-04-07T15:17:25Z

For those finding this issue, the current workaround is to just ensure that your with/go-version value exactly matches the version specified in go.mod

// go.mod
go 1.22.2

# workflows/ci.yml
- uses: actions/setup-go@v5
  with:
    go-version: 1.22.2

matthewhughes-uw · 2024-04-09T08:38:25Z

For those finding this issue, the current workaround is to just ensure that your with/go-version value exactly matches the version specified in go.mod

You could also set the GOTOOLCHAIN env var to local:

    - name: Configure env for toolchain ignore
      shell: bash
      run: echo "GOTOOLCHAIN=local" >> "$GITHUB_ENV"
    - name: Setup Go
      uses: actions/setup-go@v5

see #460 for more details

timandy · 2024-08-15T07:02:28Z

The same issue for me. Only go1.23 has the problem.

https://github.com/timandy/gohack/actions/runs/10399737890

I found that disable cache can solve the problem.

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
          cache: false

matthewhughes-uw added bug Something isn't working needs triage labels Sep 15, 2023

matthewhughes-uw changed the title ~~Tar errors on cache restore with differing go and toolchain directives~~ Tar errors on cache restore after toolchain installation Sep 15, 2023

dsame removed the needs triage label Sep 15, 2023

dsame self-assigned this Oct 16, 2023

dsame mentioned this issue Oct 24, 2023

Remove toolchain directories from the cache. #437

Closed

2 tasks

dsame linked a pull request Oct 24, 2023 that will close this issue

Remove toolchain directories from the cache. #437

Closed

2 tasks

dsame linked a pull request Oct 24, 2023 that will close this issue

Remove toolchain directories from the cache #438

Open

2 tasks

dsame removed a link to a pull request Oct 25, 2023

Remove toolchain directories from the cache. #437

Closed

2 tasks

zalimeni mentioned this issue Jan 2, 2024

[NET-5619] ci: Single source of truth for Go version in CI and Dockerfile hashicorp/consul#20058

Merged

4 tasks

i4ki mentioned this issue Feb 2, 2024

chore(gha): fix macos action/setup-go@v4 cache miss. terramate-io/terramate#1417

Merged

matthewhughes-uw mentioned this issue Feb 8, 2024

Avoid Toolchain download before cache download #456

Closed

2 tasks

matthewhughes934 mentioned this issue Feb 14, 2024

More specific handling/detection of Go toolchain versions #457

Open

vincenthsh mentioned this issue Apr 24, 2024

ci: Fix go 1.22 caching vincenthsh/fogg#289

Merged

jedevc mentioned this issue Jun 17, 2024

Improve Releasing during v0.11.7 dagger/dagger#7616

Merged

rdmitr mentioned this issue Jun 27, 2024

fix: use Go version from go.mod in Github actions getAlby/nostr-wallet-connect-next#526

Merged

matthewhughes934 mentioned this issue Aug 21, 2024

Something about the go and toolchain directive in the go.mod file cli/cli#9489

Closed

aparnajyothi-y unassigned dsame Oct 3, 2024

priyagupta108 self-assigned this Oct 3, 2024

priyagupta108 removed their assignment Nov 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tar errors on cache restore after toolchain installation #424

Tar errors on cache restore after toolchain installation #424

matthewhughes-uw commented Sep 15, 2023 •

edited

Loading

dsame commented Sep 15, 2023

thall commented Jan 8, 2024

HaraldNordgren commented Jan 17, 2024

HaraldNordgren commented Jan 17, 2024

matthewhughes-uw commented Jan 18, 2024 •

edited

Loading

HaraldNordgren commented Jan 18, 2024 •

edited

Loading

matthewhughes-uw commented Jan 22, 2024 •

edited

Loading

spencerschrock commented Feb 7, 2024

matthewhughes-uw commented Feb 8, 2024

lukeed commented Apr 7, 2024

matthewhughes-uw commented Apr 9, 2024 •

edited

Loading

timandy commented Aug 15, 2024 •

edited

Loading

Tar errors on cache restore after toolchain installation #424

Tar errors on cache restore after toolchain installation #424

Comments

matthewhughes-uw commented Sep 15, 2023 • edited Loading

dsame commented Sep 15, 2023

thall commented Jan 8, 2024

HaraldNordgren commented Jan 17, 2024

HaraldNordgren commented Jan 17, 2024

matthewhughes-uw commented Jan 18, 2024 • edited Loading

HaraldNordgren commented Jan 18, 2024 • edited Loading

matthewhughes-uw commented Jan 22, 2024 • edited Loading

spencerschrock commented Feb 7, 2024

matthewhughes-uw commented Feb 8, 2024

lukeed commented Apr 7, 2024

matthewhughes-uw commented Apr 9, 2024 • edited Loading

timandy commented Aug 15, 2024 • edited Loading

matthewhughes-uw commented Sep 15, 2023 •

edited

Loading

matthewhughes-uw commented Jan 18, 2024 •

edited

Loading

HaraldNordgren commented Jan 18, 2024 •

edited

Loading

matthewhughes-uw commented Jan 22, 2024 •

edited

Loading

matthewhughes-uw commented Apr 9, 2024 •

edited

Loading

timandy commented Aug 15, 2024 •

edited

Loading