Provide builtins.currentSystem as a input to flakes or make system a parameter

[Mic92]

Is your feature request related to a problem? Please describe.

Right now one has to explicitly define system in flake outputs.
The nix flake itself already comes with boiler code like this:

outputs = { self } : let
   # ...
   forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system);
{
  defaultPackage = forAllSystems (system: self.packages.${system}.nix);;  
};

I think having to use that much code hurts portability of the ecosystem because people
will only specify the minimum and there seems no easy way of overriding it without changing the
flake itself.

Describe the solution you'd like

outputs = { self, system } : {
  defaultPackage.${system} = self.packages.${system};  
};

Describe alternatives you've considered

Use boilercode in every project or rely on external libraries like https://github.com/numtide/flake-utils

If someone needs to explicitly specify platforms i.e. to build packages for different architectures with hydra this should be still possible.

[Mic92]

cc @zimbatm who is involved in flake-utils

[edolstra]

Not passing in system (or any other arguments) is intentional:

• It breaks hermetic evaluation: it makes the evaluation result depend on something external.
• It makes it impossible to enumerate the contents of the flake, since Nix doesn't know what the permissible values of the arguments are.
• The packages output should provide packages that work, not packages that might work.
• Evaluation caching becomes harder (not a huge problem for system, but would be a problem for arbitrary arguments).
• It would mean that fully-qualified flake output attributes no longer uniquely determine their evaluation result. I.e. an output attribute foo.bar will produce a different result on every system. Having a single name to specify flake outputs (rather than a tuple (attrname, system type, args)) is very useful, e.g. for Hydra job names (you can count on hydraJobs.foo to produce the same result for everybody).
• With a top-level system argument, you cannot use flake outputs for different systems in the same evaluation. (E.g. on x86_64-linux, you might want to use some i686-linux derivations like wine; and a Hydra aggregate job needs to be able to depend on jobs for different systems.)

[zimbatm]

There are multiple usability issues with the current design:

The list of supported systems is currently internal to the flake. As a flake author, it is useful to be able to specify which systems are supported. As a flake user, it means that it is not possible to change that list without forking the repository. Especially once the flake ecosystem starts to grow, it will quickly become prohibitive to introduce new platforms.

The amount of boilerplate that is needed to specify all the permutations of systems for all the derivations. That's why I wrote flake-utils, to make that usage a bit less cumbersome. The outputs are mixing both system-dependent and independent keys so it's not as easy as passing the system as an argument.

Typing x86_64-linux is error-prone. When a typo happens, the feedback system is not able to point to the typo directly. The user will see that their system is not supported, and then have to make the mental connection to look for a typo.

Each flake ends-up re-initializing nixpkgs. This adds to the boilerplate and evaluation time.

---

One possibility would be to introduce a new systems attribute to the top-level that lists all the supported systems by the flake. This has the advantage of making that list discoverable with tooling. And introduce typo checks for common archs. Then split the outputs to generic and system-dependent:

{
  description = "Usability flakes";
  systems = [ "x86_64-linux" "x86_64-darwin"];
  inputs = {}; # nothing changes here

  # system-independent outputs
  exports = inputs: {
    nixosConfiguration = {};
    overlay = final: prev: {};
    lib = {};
  };
  
  # system-dependent outputs
  outputs = system: { self, nixpkgs, ... }@inputs:
  let
    # I don't have a good idea to avoid that boilerplate
    pkgs = import nixpkgs {
      # Notice that the overlay is accessible via `self`
      overlays = [ self.overlay ];
      inherit system;
    };
  in
  {
    packages.hello = pkgs.hello;
    apps = {};
  };
}

One of the use-case that this design prevents is combining the derivations of multiple systems. This is for example if you want to create a release tarball that includes multiple architectures. In practice, that kind of construct is quite difficult to build outside of CI systems which have all the archs attached as remote builders.

[zimbatm]

Another possibility is that systems could inherit the list from the nixpkgs flake if left unspecified. That way, if nixpkgs gain a new architecture it will be easier to upgrade all of the flakes.

[7c6f434c]

Could we recognise the usefulness of ternary logic known-good/known-broken/unsure? The less overhead there is in checking whether something never-tried works due to upstream efforts, the better information we will have available.

[DavHau]

The original RFC states:

Flakes provide an attribute set of values, such as packages, Nixpkgs overlays, NixOS modules, library functions, Hydra jobs, nix-shell definitions, etc.

Is my assumption correct, that the idea of exposinglibrary functions has been abandoned? Since it's not possible to pass any arguments, it will never be possible to use a function of a flake?

I'm looking for exactly that use case. I'd like to expose a function that produces a python derivation, given a set of constraints.
Currently I have to use a really weird hack where I generate an infinitely deep attribute set, to allow the user to select packages:

nix run github:davhau/mach-nix#with.requests.tensorflow.aiohttp

With this trick, it's already possible to build constructs which allow passing arguments (kind of). Why not support it properly?

[edolstra]

No, you can definitely have library functions as flake outputs. You just can't call them from the command line.

Why not support it properly?

See my comment above.

Having said that, the configs branch does add experimental support for setting Nix module options from the command line (dc4a280).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-with-unfree-licenses/9405/1

[MagicRB]

I have similar problem as @DavHau, I'd like to give users of my flake the ability to compile firmware from the command line. The ideal UX is that one would write a config file, describing the build for the build system of the original project and then a derivation would be built, which would contain just the resulting .bin. This is currently impossible with flakes as I can't pass arguments into my flake from the command line.

What's the alternative?

[zimbatm]

What format is the config file in? nix run github:user/repo#compile --config ./myconfig.toml would work fine for example. nix run and nix develop use the current system to find the attribute. If the user wants to compose Nix files then they need another flake.

[MagicRB]

it's Kconfig, the kernel like build config

[domenkozar]

I wrote a proposal without knowing about this issue.

@Mic92 pointed me to it later on and it's interesting to observe that I've come to the same conclusions.

https://www.notion.so/Flakes-system-as-an-input-instead-of-an-output-2d2cdef8eac2434a833d6faae15b35c0

[thufschmitt]

I strongly support the idea of making the system handling simpler − esp. given that 80% of the time people just don’t care about it.

I think the current design has a really good thing though, in that it’s conceptually very simple (the only part of the Nix codebase that has to know about the notion of a flake “system” is the cli shortcut that expands foo#bar into foo#packages.x86_64-linux.bar). So although it’s painful and verbose, it’s explicit and doesn’t have any hidden magic.

(commenting here on @domenkozar's doc because I can't comment inline on it)

Flake would still pass the system implicitly by default (as it already does today when selecting the output attribute) but it would be explicit in flake.nix inputs.
There's already --system XXX argument to nix command line that can be used to override the default value matching host.

I guess you'd need a way to override the system arguments of the input flakes too (and separately from --system) for building multi-arch systems (Edit: I see this is handled by still having a .{system} attribute in the flake outputs, but that makes things slightly weird as it means that the system becomes both an input and an output)

We would also need to add top-level attribute supportedSystems = [ ... ] that would allow building hydraJobs and checks for all systems.

Then that leaves the problem that “The list of supported systems is currently internal to the flake” (Edit: you use inputs.nixpkgs.supportedSystems = [ ... ]; in a code snippet, which seems to imply that this could be overriden. In that case it's worth pointing out explicitely).

It is hermetic, the outputs would depend on the system input. Flakes caching can take the system into the account when calculating the key cache.

Maybe it’s the case (need to think it a bit more), but this only holds as long as Nix can generate a system-independent lockfile (because it would be pretty bad if the lockfile couldn’t be shared between two different systems)

There could be --local-system argument to flake commands to restrict evaluation to only one system.

Alternatively, that could be the default with --all-system providing the current default.

That bit isn't really clear to me: If system is an input, then evaluating the flake will only evaluate it for the system that's given as input. Or do you mean that a flake would actually evaluate to something like map (system: outputs (inputs // { inherit system; }) supportedSystems?

You'll have to evaluate each input with a different system. That can now be done in parallel.

I don’t think it’s really easier to parallelize this than the current version. The blocker in both cases is that the evaluator isn’t reentrant.

[thufschmitt]

Maybe a (relatively) small change to solve the problem of systems being hardcoded in a flake (which might also solve other issues) would be to allow raw Nix values as input for a flake (this shouldn’t mess-up caching as long as these values end-up in the lockfile). Then we could have a convention that flakes accept a supportedSystems input − in which case it could be overriden without having to fork the flake.

That does in a way add yet-more-convention and doesn’t do anything for reducing the boilerplate so I’m not sure it’s a good idea as-it-is, but there might be something to build on top of that.

[domenkozar]

I strongly support the idea of making the system handling simpler − esp. given that 80% of the time people just don’t care about it.

Thanks for putting it so succinctly!

I think the current design has a really good thing though, in that it’s conceptually very simple (the only part of the Nix codebase that has to know about the notion of a flake “system” is the cli shortcut that expands foo#bar into foo#packages.x86_64-linux.bar). So although it’s painful and verbose, it’s explicit and doesn’t have any hidden magic.

That's because it pushes most of the questions to the writer of the flake. I consider that a con, because things have to be implemented over and over again.

I guess you'd need a way to override the system arguments of the input flakes too (and separately from --system) for building multi-arch systems (Edit: I see this is handled by still having a .{system} attribute in the flake outputs, but that makes things slightly weird as it means that the system becomes both an input and an output)

It wouldn't be an output, but you'd declare it as a setting for the input itself - evaluating the same input for different systems. I'll update the document to clarify this.

Then that leaves the problem that “The list of supported systems is currently internal to the flake”

It gives the space for warning the user that the system isn't supported by the flake, instead of getting an attribute error. That seems quite an improvement for the experience.

Maybe it’s the case (need to think it a bit more), but this only holds as long as Nix can generate a system-independent lockfile (because it would be pretty bad if the lockfile couldn’t be shared between two different systems)

Looking at the lockfiles (there's, unfortunately, no documentation about them), what does narHash consist of?

That bit isn't really clear to me: If system is an input, then evaluating the flake will only evaluate it for the system that's given as input. Or do you mean that a flake would actually evaluate to something like map (system: outputs (inputs // { inherit system; }) supportedSystems?

It would mean you can evaluate the flake for all given systems, for example on a CI to make sure everything evaluates.

I don’t think it’s really easier to parallelize this than the current version. The blocker in both cases is that the evaluator isn’t reentrant.

It's parallelizable because one needs to pass system as an input, so when evaluating for more than one system, that's one evaluation per system.

Maybe a (relatively) small change to solve the problem of systems being hardcoded in a flake (which might also solve other issues) would be to allow raw Nix values as input for a flake (this shouldn’t mess-up caching as long as these values end-up in the lockfile). Then we could have a convention that flakes accept a supportedSystems input − in which case it could be overriden without having to fork the flake.

That will relative flakes would really go a long way with modularity. I'd still make supportedSystem a special meaning for warnings, etc as mentioned above.

That does in a way add yet-more-convention and doesn’t do anything for reducing the boilerplate so I’m not sure it’s a good idea as-it-is, but there might be something to build on top of that.

Yes, this design change is "convention over configuration" applied to dealing with system. Why do you think more conventions is bad?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/improving-flakes/12831/24

[Ericson2314]

I agree with the gist of this change. The larger issue though is Flakes is constantly lagging behind idioms in Nixpkgs that have been polished over the years. For example:

1. We don't just pass one system, we pass stdenv.{build,host,target}Platform which provided much more detailed information and supports cross compilation.
2. lib.platforms used to be not a list of systems (again, too crude) but an arbitrary predicate on the host platform, which allowed intent to be conveyed much more precisely. This was reverted because it broke some (IMO layer-violating) feature of Hydra or nix-env.

We could incorporate all this stuff in Nix for Flakes, but that leaves Flakes quite brittle when we can and do refine this stuff from time to time in Nixpkgs. Or maybe we should implement more of Flakes in Nix. Factoring out lib/ and such things to make a standard library of sorts so we still have some agility. But that that point, what are flakes? Just something to manage inputs / lockfile without much extra structure?

[edolstra]

The larger issue though is Flakes is constantly lagging behind idioms in Nixpkgs that have been polished over the years.

Maybe the nix UI, but not flakes, because...

Just something to manage inputs / lockfile without much extra structure?

That's basically correct. The flake file format doesn't impose a lot of requirements on what the output attributes of a flake are. Thus it can accommodate whatever idioms regarding systems/platforms we may come up with in the future. (This is also why we shouldn't have a system input attribute, because that would lock us into a particular idiom.)

OTOH, various tools can define "well-known" flake output attributes, e.g. nix build looks for packages.<name>.<system> by default, as a convenience. These can of course evolve, e.g. we could have a packages_v2 output that uses a different encoding for system types.

This was reverted because it broke some (IMO layer-violating) feature of Hydra or nix-env.

The semantics of those attributes are determined by the tools that operate on them. You can't just change the meaning of platforms and then complain that nix-env barfs :-)

[edolstra]

Looking at the lockfiles (there's, unfortunately, no documentation about them), what does narHash consist of?

Lock files are documented here: https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-flake.html#lock-files

It's parallelizable because one needs to pass system as an input, so when evaluating for more than one system, that's one evaluation per system.

The current hydra-evaluator is already parallel. Having to deal with a system input type would just make it more complex since it then has two axes of parallelism to deal with: the system type and the jobs for each system type.

[gilligan]

That's basically correct. The flake file format doesn't impose a lot of requirements on what the output attributes of a flake are. Thus it can accommodate whatever idioms regarding systems/platforms we may come up with in the future. (This is also why we shouldn't have a system input attribute, because that would lock us into a particular idiom.)

I understand what you are arguing for, but I am confused because to me it seems like this might at odds with the goals of nix flakes as declared in the abstract of the RFC

Abstract: This RFC proposes a mechanism to package Nix expressions into composable entities called "flakes". Flakes allow hermetic, reproducible evaluation of multi-repository Nix projects; impose a discoverable, standard structure on Nix projects; and replace previous mechanisms such as Nix channels and the Nix search path.

So "standard structure" is indeed only to be understood as "something to manage inputs / lockfile" as @Ericson2314 put it above?

[tmillr]

I'm sort of running into this. I'm new to nix, and flakes especially, but I'm just trying to create a simple flake which just downloads, aggregates, and manages all of my editor plugins (i.e. vim/neovim plugins that are each merely git repositories of interpreted vim/lua scripts).

Using a flake sounds nice because I can: list/specify all of my plugins declaratively within a single/simple flake.nix file, automatically track versions in a lockfile, and then also update all of them easily in one go. These plugins are mostly written in lua or vimscript and don't really have a notion of platform or system, nor a proper build stage/step. You essentially just clone the repository and you're good to go. I don't really want to hardcode the system in the output attrs because I want it to work on any system, and I can't use currentSystem (nor lib.systems.doubles.all?) without specifying --impure on the command line every time. I'm not sure if there's a better way to go about this or something I'm missing (I am aware of the vim utils in nixpkgs, but there are several such files to consider and documentation is sort of lacking, so I decided to go with a more direct and custom solution that will hopefully be quicker/simpler and maybe teach me a few things about nix/flakes in the process). Perhaps it's best to just map my single flake output/drv (which is merely a dir of all my plugins) to each of the systems supported by my editor (Neovim)?

So that's my use-case, although take it with a grain of salt as I only started using and migrating my packages to nix last week.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-as-a-unified-format-for-profiles/29476/1

[samueldr]

• It breaks hermetic evaluation: it makes the evaluation result depend on something external.

Should I open an issue for those?

• builtins.langVersion
• builtins.nixVersion

And somewhat this one too?

• builtins.storeDir

---

• The packages output should provide packages that work, not packages that might work.

In practice this is what happens already.

• https://github.com/numtide/flake-utils/blob/main/allSystems.nix

I'm pretty sure that not everything relying on flake-utils makes sure that every package is correctly working all the time for every system they expose.

While the goal is noble, either this means developers have to be too narrow and ignore plausibly working situations (“I haven't built this tool for anything else than x86_64-linux, so I'm just exporting x86_64-linux”) or end-up doing the undesirable option here.

---

I've talked about it with multiple people that know flakes better than I do, and no one was able to answer me: “What does $system in packages.$system in the flakes interface really is meant to represent?”

Is it the system that the thing should be built on? If so, I'm not sure the semantics here are correct.

Is it the system the thing should be running on? If so, would a U-Boot build for e.g. an AArch64 system only be in packages.aarch64-????.uBootSomethingAArch64. How would end-users be expected to cross-compile it? (Relatedly, what should the correct system name be for "non-OS" artifacts like a platform firmware?)

Here I'm not even talking about the "middle steps" of cross-compilation. Or about unconventional outputs like WASM, HTML pages, fonts, images, PDFs, GBA ROMs.

Even the manual section for the format doesn't really say:

• https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html?highlight=flake.nix#flake-format

The value returned by the outputs function must be an attribute set. The attributes can have arbitrary values; however, various nix subcommands require specific attributes to have a specific value (e.g. packages.x86_64-linux must be an attribute set of derivations built for the x86_64-linux platform).

[B]uilt for is the best keywords we have for this.

I might have missed something, but assuming I want to use Flakes to provide a U-Boot, I have to either lie about what the thing is built for, or somehow have something like this:

packages = {
  "aarch64-????-but-built-on-x86_64-linux" = ...;
  "aarch64-????-but-built-on-aarch64-linux" = ...;
};

This ends-up meaning the default output cannot be used, and I have to provide clunky instructions that vary depending on the system the user happens to be using. Instead when builtins.currentSystem can be used, it can intrinsically provide a better UX for the end-user, as it will provide them only the attributes they want (the target board to build for) and the commands are the same for the same "moral" builds whichever system you end-up being evaluating on.

So in practice, the current "arbitrary" values for packages, and similar friends, ends-up being builtins.currentSystem with extra steps and less flexibility. It's pushing the duty of tracking an input (the current system) from the tool onto the end-user of the build.

[emilazy]

In practice this is what happens already.

• https://github.com/numtide/flake-utils/blob/main/allSystems.nix

FWIW people seem to usually use eachDefaultSystem, which is a considerably more reasonable/conservative set: https://github.com/nix-systems/default. But I agree with you: flake outputs as guarantee of expected-to-work are dead on arrival, and anyway promises of being reasonably tested shouldn't be tied to it being horribly painful or even impossible to try to run on a given system; metadata for that is reasonable, having it determine whether you can even try without poking at overlays or even source files is not. This already hurt a lot during the aarch64-darwin transition.

Is it the system that the thing should be built on? If so, I'm not sure the semantics here are correct.

In practice I believe that this is implicitly the semantics because of e.g. Nixpkgs and pkgsCross. Admittedly, that's legacyPackages, so you could argue it's exempt from the rules. But I don't see how anything other than "the Nix derivation build system" really makes sense here, especially in the presence of, like you say, outputs that are not executable code. So I'd expect packages.x86_64-linux.crossAArch64.uboot or whatever. I think this is arguably kind of bad though, and I don't get the sense that the flakes design was really constructed with cross compilation in mind at all.

[samueldr]

FWIW people seem to usually use eachDefaultSystem

I debated linking to that or the eachDefaultSystem, but the crux of the message is what mattered, either you're too narrow, or too wide here.

So I'd expect packages.x86_64-linux.crossAArch64.uboot or whatever

I believe the packages.* attrsets have to be flat. [citation needed]

And anyway still comes to the same conclusion: you're pushing tracking an ambiant automatically discoverable detail (the current system) to the end-user.

(Yes, I believe we agree here, I simply wanted to make it clear to others.)

[roberth]

hermetic evaluation

Should I open an issue for those?

Apologies for my literal interpretation. We have

• In pure mode, set builtins.nixVersion to a constant #5971
• langVersion does not have one, and I think we should keep it as contingency, but perhaps weakened similar to what I've proposed for nixVersion: a comparison function returning bool.

Note that these are not the kind of impurity that would prevent caching as they are still controlled by Nix itself, and such a bool is unlikely to end up in a .drv by itself, and would be unlikely to be used for something other than workarounds, where they are indispensable.

---

But I don't see how anything other than "the Nix derivation build system" really makes sense here, especially in the presence of, like you say, outputs that are not executable code.

That's for nix build perhaps.
nix run would require the "system" attribute to match hostPlatform while buildPlatform could be anything for which you have a local or remote build capability.

[LunNova]

I really like the perSystem function roberth suggested last year, and have been using that approach in my own flakes successfully for a few months now. It feels very clean, and you can easily be backwards compatible with the standard flake schema by calling perSystem to create it.

Has anyone tried to work cross-compiles into this? I guess an attrset passed to perSystem with the host/target rather than just a string system?

[tobiasBora]

Just a tiny message to remind a comment above saying that it would be great to provide meta information to say if a program compiles for a given plateform, does not compiled (e.g. binary not available), or **not tested** (it may or may not work), to avoid unecessary errors.

[roberth]

@tobiasBora The proposed Nixpkgs "problem" infrastructure could perhaps be used for this purpose. It wasn't specifically design for this, but it seems that it could be made to work well outside Nixpkgs as well. It's currently in development at:

• check-meta.nix: Add package warnings nixpkgs#177272

[roberth]

I've renamed this issue to hint at an alternative that is less contradicting to the requirements suggested in the first response.

Suggestions have been made to solve this by

• Adding a function such as perSystem
• Making the collections of system based attrs behave more like a function
  • Using __functor, or
  • Proxy like abstraction for an attrset #8187 with the side note that this either stretches the definition of a valid attrset, or it is an insufficient solution that would still require all valid systems to be enumerated by the user

We can change this back to a discussion about a single solution if you prefer, but otherwise I think this issue is representative of roughly the requirements

• Users shouldn't have to specify a list of systems. Some flakes are just portable, even if they do have derivations
• Users should be able to build a flake for a niche or not so niche system that wasn't specified in a flake they depend on. (Similar to the previous requirement, yet different)
• Cross compilation should be possible in a standardized way

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/experimental-does-not-mean-unstable-detsyss-perspective-on-nix-flakes/32703/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/experimental-does-not-mean-unstable-detsyss-perspective-on-nix-flakes/32703/25

[alper]

esp. given that 80% of the time people just don’t care about it

I would love never having to lookup the correct string for my system ever again (an absurd bit of busywork). Whatever you need to do to make this happen, go ahead.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-25-nix-team-meeting-minutes-89/33489/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-design-of-system-os/36654/4

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-design-of-system-os/36654/7