[RFC 0075] Declarative wrappers

[doronbehar]

Rendered.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/declarative-wrappers-another-idea/6661/7

[FRidh]

In my opinion we need some kind of passthru "tag" system (where the tags can have values) that is used in conjuction with a buildEnv that takes a list of functions that use these tags to decide how to perform the composition. Then, there will be functions for the various Python interpreters (although they're essentially all the same).

Anyway, we need to start somewhere and see what issues we run into. As I commented elsewhere, nobody here has the entire overview.

[FRidh]

Limiting ourselves to environment variables is at times too limiting. Also, environment variables should be the implementation detail, although the variable names could be used as identifier.

Python example.

If you have a closure of packages, and there happens to be a Python application with Python libraries as dependencies, then that group of packages needs to be composed. If there is another Python application depending on other libraries, then that needs to be composed separately. At least, they are not allowed to share the same *PYTHONPATH. (Note one also should not use PYTHONPATH but that's a different discussion.)

[doronbehar]

If there is another Python application depending on other libraries, then that needs to be composed separately. At least, they are not allowed to share the same *PYTHONPATH

I see this is all about ABI compatibility, and I understand the issue while I remember you mentioned this in NixOS/nixpkgs#85103 . I'm still unsure whether it's possible it'd happen, assuming our libraries are composed OK. @FRidh could you perhaps give a situation (with perhaps some rough code examples) to a situation where say Python2's and Python3's libraries will share the same *PYTHONPATH? It's important to note that the current design counts upon buildInputs and propagatedBuildInputs only.

[FRidh]

Different Python version is one case, the other is simply applications requiring different versions of a dependency.

[timokau]

This can happen for example when you have a python3 application that shells out to an python2 application.

[FRidh]

Currently we prevent some of these effects by unsetting NIX_PYTHONPATH when a Python program starts, thereby avoiding leakage. This problem with wrappers and environment variables should also be noted clearly in the RFC.

[doronbehar]

OK I think I start to understand better the concerns. But still, this is not a "Drawback" of the idea, I mean, we still need to workaround such edge cases specifically in our current methods right? I agree though that it might be more complicated if someone is obliged to use wrapGeneric and not a wrap hook.

As an implementation detail, it may be worth noting that it'd be nice to make wrapGeneric able to "catch" such incompatible values in an environment variable.

[jtojnar]

Both of the issues related to gdk-pixbuf only happened for non-wrapped derivations and current wrapGAppsHook would already fix them.

The issue was, IIRC, that gdk-pixbuf NixOS module sets GDK_PIXBUF_MODULE_FILE environment variable, which was enabling binary gdk-pixbuf modules with ABI incompatible with the gdk-pixbuf the programs linked against. In one case it was caused by a bug that slipped through the ABI stability guarantees, in the other by running a program for i686 on x86_64.

Either way, it is somewhat orthogonal to wrappers. Only that we might want to always wrap graphical applications with librsvg gdk-pixbuf module.

[jtojnar]

Though there is one concern around GDK_PIXBUF_MODULE_FILE – the environment variable supports only a single path so we need to build the cache containing all the modules using gdk-pixbuf-query-loaders – the current hook actually does not actually do that! but it is required for some image formats to work in e.g. eog. We will need to take inspiration in the module.

[michaelpj]

In my opinion we need some kind of passthru "tag" system (where the tags can have values) that is used in conjuction with a buildEnv that takes a list of functions that use these tags to decide how to perform the composition.

This reminds me of @nbp's old proposal which took this to the extreme of delaying any construction of derivations until the last minute. This has the advantage of giving you access to all the information that goes into a derivation when composing things, like shoving everything in passthru.

Obviously this is a radical change (and the motivation in the previous proposal was quite different), but I think it would help here too.

[timokau]

This reminds me of @nbp's old proposal which took this to the extreme of delaying any construction of derivations until the last minute.

Which in turn sounds a lot like the Nix-lang 2.0 proposal. We shouldn't extend the scope of this RFC too much, but its at least somewhat related.

[FRidh]

Many derivations nowadays have multiple outputs. Will this RFC consider multiple outputs? If so, let's say we have a multiple-output derivation, and we only needed to wrap bin/ which happens to be in the "bin" output, but there are also other outputs, and one of those outputs is in extraOutputsToInstall. Do we expect that, through our wrapper derivation, that output is picked up and included as expected?

[doronbehar]

Many derivations nowadays have multiple outputs. Will this RFC consider multiple outputs? If so, let's say we have a multiple-output derivation, and we only needed to wrap bin/ which happens to be in the "bin" output, but there are also other outputs, and one of those outputs is in extraOutputsToInstall. Do we expect that, through our wrapper derivation, that output is picked up and included as expected?

Since extraOutputsToInstall is a NixOS option, wrapped derivations should be handled by a NixOS module. Here's the vision I had for the far future regarding this:

Once declarative wrappers will get mature, we should make environment.systemPackages detect derivations that are wrapped with wrapGeneric. Then, if needed, make the module call wrapGeneric differently then how it is called in all-packages.nix while considering extraOutputsToInstall.

In general, wrapGeneric should symlink to a single derivation both meta.outputsToInstall and what's in extraOutputsToInstall.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-module-system-for-wrappers-in-nixpkgs/42281/1