Unify pkg.override and drv.overrideAttrs

[roberth]

Issue description

Noting that

• each overriding method adds overhead, both in UX and in complexity in the code that implements the fixpoints (e.g. hacks such as adding back .override after .overrideAttrs)
• each overriding method is separated from the next overriding method by a function, such as the package definition functions
• a function definition + application can be similar to a set of options
• usually the function is only called once, thereby resembling module composition more than functional programming

we can reduce the package definition to a single fixpoint and single overriding mechanism.

Goal

This issue is done when .override has become unnecessary.

Example

A package definition currently looks as follows.

{ pkg1, pkg2, stdenv }:

stdenv.mkDerivation (self: {
  buildInputs = [ pkg1 pkg2 ];
})

Instead it could look like

pkgs:

pkgs.stdenv.mkDerivation (self: {
  # deps here is a "local" variable, not passed to builtins.mkDerivation
  deps = {
    inherit (pkgs) pkg1 pkg2;
  };
  buildInputs = [ self.deps.pkg1 self.deps.pkg2 ];
})

which can be reduced with function reflection "syntax sugar" to:

pkgs:

pkgs.stdenv.mkDerivation ({ pkg1, pkg2 }: self: {
  buildInputs = [ pkg1 pkg2 ];
})

and have its boilerplate DRY-ed into a function that is used similarly to callPackage today to become

{ pkg1, pkg2 }: self: {
  buildInputs = [ pkg1 pkg2 ];
}

Alternatives

Another way to approach this is to merge callPackage and mkPackage into one function, and possibly reduce mkDerivation to a function we generally don't override.

Notes

Note that this also resolves the problem of dependency choices having to be specified in all-packages.nix. Refs #295042

mkDerivation does not support local variables yet. This touches on the idea of a mkPackage function that leaves the actual derivation stuff up to its caller. Refs #295044

Open question: how does cross compilation splicing fit into this picture?

[aameen-tulip]

This was one I've been working on a lot.
This is my prototype, certainly not perfect but I've been refining it a lot over the last month.
It's comprised of two sorts of recursive attrsets, one for an "entry" ( single package or element ) and entry-sets ( like a package-set or scope ). They share a common base ExtInfo that is an extensible attrset full of functors.
ExtInfo: https://github.com/aameen-tulip/at-node-nix/blob/292ba02b88e3251e3828c257c1c1ce5dd231fe90/lib/meta.nix#L148
metaEnt: https://github.com/aameen-tulip/at-node-nix/blob/292ba02b88e3251e3828c257c1c1ce5dd231fe90/lib/meta.nix#L390
metaSet: https://github.com/aameen-tulip/at-node-nix/blob/292ba02b88e3251e3828c257c1c1ce5dd231fe90/lib/meta.nix#L478

These are used for my Node.js routines that dynamically generate derivations from various types of Node.js metadata ( package[-lock].json, manifest.json, etc ).
https://github.com/aameen-tulip/at-node-nix/blob/224bdf7de6556e36f58b351699062726bf6298bf/pkgs/node-pkg-set.nix#L151 ( this is a branch with a slightly older definition of the "meta" constructs; but it illustrates the usage ).

EDIT: I just realized I meant to post this in the other issue: "inconsistent override mechanisms". I can't delete this comment for whatever reason though.

[infinisil]

Very related to #295046

[roberth]

@DavHau and I have found an interesting solution to implement this in the experimental drv-parts

Here you can see it in action: the deps option is exposed to the global sources of packages (ie pkgs) by means of a type coercion. Although technically this type coercion does not bring truly new information into the package module, as it is powered by a module argument in the end, but it establishes a pattern that establishes effectively the same indirection that was created by the callPackage arguments to the package function.

This avoids the function indirection, which would be hard to override, and it allows everything about the package, including the default dependencies, to be declared in the same file.

[roberth]

PoC alternate simple module implementation (as this issue usually comes up in modular package discussion):

• https://gist.github.com/roberth/940dff88ca5f5f95949dc309dbe60a65

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-30-nixpkgs-architecture-team-meeting-30/25973/1

[ghost]

Open question: how does cross compilation splicing fit into this picture?

I think there may be some problems, but perhaps some of them will cease to exist before they are encountered.

Enough people seem to be getting frustrated with splicing that there is talk of providing a much simpler alternative and seeing if people are willing to switch. Some of those alternatives fit very nicely with the "let .overrideAttrs do .override's job"¹ solution described at the top of this issue.

Footnotes

1. Credit where due: apparently RFC 3 was the first to propose this, but in the context of much much larger changes to nixpkgs (including new Nix syntax). ↩

[roberth]

dream2nix has a deps pattern that achieves this unification without loss.
In #273815 I propose to use that idea as well.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/override-derivation-with-overrideattrs-any-way-to-handle-let-in/47307/2