Hercules agent appears to delete the .git folder

[jappeace]

Description

At supercede we're trying to make the docs and ghc timings report not build for every master merge, and do this perhaps once per week.
Both these products are simply not that interesting to be kept up to date all the time, but they take up to 30% of CI running times.

This is problematic considering we sometimes need to do "hot" fixes, and having to wait an additional 20 minutes on some build product you don't care about isn't great.

So we wanted to pin this to some previous version of the product.
And have this pin updated once a week by a cron job.

To Reproduce

The most obvious thing we tried was creating a nix file labeled nix/old-supercede.nix and then:

builtins.fetchGit {
                     url = ../.;
                     rev = "47d62c85ab6ab32a0f08310a1575a659e750d7d2";
}

This fails because there is no .git folder.
(although this particular command won't tell, the nixpkgs fetchGit will).

Expected behavior

I'd expect the git folder to be there

Logs

this is from fetch git:

but a more eleborate scheme where I ran all git commands myself resulted in:

Platform / Version

[roberth]

I'm glad you've described both problems, because since this week, you can solve the asynchronous build directly with the new onSchedule job feature.

While adding the .git may make your solution work, I don't think even evaluating the slow building attribute is a nice experience.

The onSchedule feature builds on the herculesCI attribute, which I think needs more documentation for the non-flakes use case.
Reference documentation for onSchedule is available here, but no guide specifically on how to move a build into a scheduled job yet, especially not for ci.nix, which I believe is what you're using, ie no flakes.

As a general outline, to use this feature the steps are

1. Update the agent to 0.9.8
2. Put an herculesCI attribute in ci.nix
   • This will replace the ci/hercules/derivations job. This needs an herculesCI.onPush.default, which is somewhat non-trival without flakes, but something I will take care of.
3. Remove the reference to the slow build from the herculesCI.onPush.default.outputs
4. Re-add in a herculesCI.onSchedule.foo.outputs.bar

Click to open draft guide...

Here's a draft of the ci.nix migration.

TODO (@roberth)

• write fixupRecurseForDerivations
• test
• turn into proper prose

Starting point

ci.nix

{
  foo = mkDerivation ...;
  bar = barPackages.bar;
  slow = runCommand "slow" {} "";
}

Add herculesCI

ci.nix

let ciNix = {
  foo = mkDerivation ...;
  bar = barPackages.bar;
  slow = runCommand "slow" {} "";
  herculesCI = { ... }: {
    onPush.default.outputs = fixupRecurseForDerivations (removeAttrs ["herculesCI"] ciNix);
  };
}

Update the branch protection settings on GitHub to look for the new status instead.

Add onSchedule and move the slow attr over.

ci.nix

let ciNix = {
  foo = mkDerivation ...;
  bar = barPackages.bar;
  slow = runCommand "slow" {} "";
  herculesCI = { ... }: {
    onPush.default.outputs = fixupRecurseForDerivations (removeAttrs ["herculesCI"] ciNix) // { slow = null; };
    onSchedule.slow = {
      outputs = { inherit (ciNix) slow; };
      when = {
        # every day at 2:xx UTC
        # see [docs](https://docs.hercules-ci.com/hercules-ci-agent/evaluation/#attributes-herculesCI.onSchedule-when)
        hour = [ 2 ];
      };
    };
  };
}

[jappeace]

This solves 90% of our problem, however we also want to deploy a version of the haddock and timings report with the rest of our application in a http authenticated 'dev' environment.
Is it possible to use scheduled builds like that? How do we know which artificat was build? (eg the store path)

[roberth]

Where it says herculesCI = { ... }: {, you can ask for the git revision, herculesCI = { rev, ... }: {, and add that to the report by copying the pure report from ci.nix, or using .overrideAttrs.

You could deploy the haddocks and timings report to a separate profile; for example /nix/var/nix/profiles/dev-report. You could then configure that as the content location in your web server or app.
This way you have two independent deployments: one being your main application including its infrastructure, and the other just the dev content.

Alternatively, I now realize that your fetchGit doesn't have to be local. If you use the remote url, you can stick closer to your original plan, and Nix will cache the repo for you, so the performance should be equal.

I can generalize the update effect in hercules-ci-effects next week, so that you don't have to reinvent the wheel for your cronjob.

[roberth]

A workaround for this, when using a ci.nix, is to start the file with:

args@{
  src ? { ref = null; },
  submodulesFetched ? builtins.pathExists ./some/path/inside/submodule,
  ...
}:

# Fetch submodules if needed, on CI
if src?remoteHttpUrl && ! submodulesFetched
then
  let source =
    builtins.fetchGit {
      url = src.remoteHttpUrl;
      rev = src.rev;
      ref = src.ref;
      submodules = true;
    };
  in import (source + "/ci.nix") (args // { submodulesFetched = true; })
else